* fix(security+ops): #82 #85 #87 — auth hardening, API validation, deploy posture Security and operational hardening across three issue groups: - M23: docker-compose.override.yml → docker-compose.dev.yml (BREAKING, prod foot-gun) - C13: Container runs as non-root user 'agnes' (USER directive in Dockerfile) - M21: Docker resource limits (mem_limit, cpus) on app + scheduler - M22: Caddyfile security headers (X-Frame-Options, X-Content-Type-Options, Referrer-Policy, -Server) - M17: /api/health split into minimal (unauth) + /api/health/detailed (auth) (BREAKING) - M26: release.yml restricts build-and-push to main + workflow_dispatch; paths-ignore for docs - C2: table_id traversal validation on /api/data/{table_id}/download - M4: Upload streaming (chunk-read + temp file) instead of full-buffer; /local-md hashed filename - C5: reset_token removed from POST /api/users/{id}/reset-password response - C8: Startup WARNING when no user has password_hash (bootstrap window visible) - M9: Audit log on failed web form login (mirrors /auth/token endpoint) - M10: Atomic magic-link consume via compare-and-swap (CONSUMED: marker + DuckDB conflict catch) Also: SSRF protection on /api/admin/configure (#46), memory stats SQL aggregation (#90) Generated with [Devin](https://cli.devin.ai/docs) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(review): SSRF 169.254.x.x + IPv6 multicast; M10 marker cleanup safety Review fixes: - Add 169.254.0.0/16 (link-local, cloud metadata) to SSRF regex — was missing, allowing requests to AWS/GCP/Azure metadata endpoints - Add ff[0-9a-f]{2}: (IPv6 multicast) to SSRF regex - M10: wrap Step 3 (CONSUMED marker cleanup) in try-except with warning log — prevents unhandled exception if DB write fails after successful token consumption - Add test for 169.254.169.254 SSRF rejection Generated with [Devin](https://cli.devin.ai/docs) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(review): SSRF IPv6 bypass, CLI health endpoint, upload FD leak Address Devin Review findings on PR #104: 1. SSRF IPv6 bypass: Replace hostname regex with DNS resolution + ipaddress module checks. The old regex patterns like `fe80:` only matched up to the first colon, missing real IPv6 addresses like `fe80::1`, `fc00::1`, `ff02::1`. The new approach resolves the hostname via getaddrinfo and checks each resulting IP against ipaddress.is_private/is_loopback/is_link_local/is_reserved/is_multicast. 2. CLI commands broken: `da setup test-connection`, `da setup verify`, `da diagnose`, `da status` all called /api/health expecting the old format (status=="healthy", services dict). Now they call /api/health/detailed for service-level checks (with graceful fallback to the minimal endpoint when auth is not configured). 3. Temp file handle leak: _stream_to_temp returns an open NamedTemporaryFile; callers now close it before shutil.move() to prevent FD leaks until GC. Also adds IPv6 SSRF test cases (loopback, link-local, unique-local, multicast) with mocked DNS resolution for test environment independence. Generated with [Devin](https://cli.devin.ai/docs) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(review): download regex blocks hyphenated IDs; document health split Address Devin Review round-3 findings on PR #104: 1. _SAFE_IDENTIFIER regex blocked hyphenated table IDs: The download endpoint used the strict SQL-identifier regex which does not allow dots or hyphens, but Keboola table IDs like in.c-crm.orders contain both. Switched to _SAFE_QUOTED_IDENTIFIER which allows dots and hyphens while still blocking path-traversal chars (/, .., \) and quote/control characters. Added test for hyphenated/dotted IDs. 2. Documented health endpoint split in DEPLOYMENT.md: Added Health checks & external monitoring section explaining both endpoints (minimal unauth /api/health vs authenticated /api/health/detailed) and how to wire external monitoring tools to the detailed endpoint with a PAT. Generated with [Devin](https://cli.devin.ai/docs) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * release(0.12.1): cut hotfix for snapshot integrity + #82/#85/#87 hardening * fix(security): apply CAS pattern to password reset confirm (#82/M10 follow-up) Devin review on the rebased PR flagged the asymmetry: magic-link verify got the atomic compare-and-swap pattern in the original M10 fix, but password reset confirm at /auth/password/reset/confirm was still using read-validate-clear. Two concurrent POSTs with the same valid reset token could both succeed in setting different new passwords (last-write- wins). Lower severity than the magic-link race because the attacker would need the reset token AND to race the legitimate user, but the asymmetry was a polish gap. Mirrors app/auth/providers/email.py::_consume_token CAS exactly: write unique CONSUMED:<random> marker via UPDATE...WHERE token=old_token, then SELECT to verify our marker won, then proceed. Only the winner clears the marker and applies the password change. New regression test_concurrent_reset_only_one_wins in tests/test_password_flows.py::TestResetConfirm pins the contract: two ThreadPoolExecutor workers + Barrier hit /reset/confirm with the same token; exactly one gets 302 (password applied), the other gets 200 with 'Invalid or expired'. Sanity-checked against the pre-CAS code — both POSTs got 302 (race confirmed). --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
205 lines
7.9 KiB
YAML
205 lines
7.9 KiB
YAML
name: Release
|
|
|
|
on:
|
|
push:
|
|
branches:
|
|
- main
|
|
- "**" # build :dev-<slug> image for any branch push (e.g. feature/x, zs/edit, fix/y)
|
|
paths-ignore:
|
|
- "docs/**"
|
|
- "*.md"
|
|
- "LICENSE"
|
|
workflow_dispatch: # manual trigger for explicit dev-<slug> builds
|
|
|
|
permissions:
|
|
contents: write
|
|
packages: write
|
|
|
|
jobs:
|
|
test:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v5
|
|
|
|
- uses: actions/setup-python@v6
|
|
with:
|
|
python-version: "3.13"
|
|
|
|
- name: Install uv
|
|
uses: astral-sh/setup-uv@v7
|
|
|
|
- name: Install dependencies
|
|
run: uv pip install --system ".[dev]"
|
|
|
|
- name: Run tests
|
|
run: pytest tests/ -v --tb=short
|
|
env:
|
|
TESTING: "1"
|
|
|
|
build-and-push:
|
|
needs: test
|
|
# Only publish images from main pushes or manual triggers.
|
|
# Non-main branch pushes run tests only; use workflow_dispatch
|
|
# for explicit dev-<slug> image builds when needed.
|
|
if: github.ref == 'refs/heads/main' || github.event_name == 'workflow_dispatch'
|
|
runs-on: ubuntu-latest
|
|
outputs:
|
|
image_tag: ${{ steps.meta.outputs.versioned_tag }}
|
|
version: ${{ steps.meta.outputs.version }}
|
|
channel: ${{ steps.meta.outputs.channel }}
|
|
steps:
|
|
- uses: actions/checkout@v5
|
|
with:
|
|
fetch-depth: 0
|
|
fetch-tags: true
|
|
|
|
- name: Claim version tag (with retry to avoid race conditions)
|
|
id: meta
|
|
run: |
|
|
git config user.name "github-actions[bot]"
|
|
git config user.email "github-actions[bot]@users.noreply.github.com"
|
|
|
|
YEAR_MONTH=$(date +%Y.%m)
|
|
if [[ "${{ github.ref }}" == "refs/heads/main" ]]; then
|
|
CHANNEL="stable"
|
|
else
|
|
CHANNEL="dev"
|
|
fi
|
|
SHORT_SHA=$(echo "${{ github.sha }}" | cut -c1-7)
|
|
|
|
# Claim a unique version by pushing a git tag BEFORE building.
|
|
# Retry up to 5 times if another CI run took our N.
|
|
TAG_CLAIMED=false
|
|
for ATTEMPT in 1 2 3 4 5; do
|
|
git fetch --tags --force
|
|
# Use max(N) not count — safe even if tags are deleted
|
|
MAX_N=$(git tag -l "*-${YEAR_MONTH}.*" | sed 's/.*\.//' | sort -n | tail -1)
|
|
N=$(( ${MAX_N:-0} + 1 ))
|
|
VERSION="${YEAR_MONTH}.${N}"
|
|
TAG="${CHANNEL}-${VERSION}"
|
|
|
|
git tag -a "$TAG" -m "Release $TAG"
|
|
if git push origin "$TAG" 2>/dev/null; then
|
|
echo "Claimed tag $TAG (attempt $ATTEMPT)"
|
|
TAG_CLAIMED=true
|
|
break
|
|
else
|
|
echo "Tag $TAG already exists, retrying... (attempt $ATTEMPT)"
|
|
git tag -d "$TAG"
|
|
sleep 2
|
|
fi
|
|
done
|
|
|
|
if [ "$TAG_CLAIMED" != "true" ]; then
|
|
echo "::error::Failed to claim a unique version tag after 5 attempts"
|
|
exit 1
|
|
fi
|
|
|
|
echo "channel=${CHANNEL}" >> "$GITHUB_OUTPUT"
|
|
echo "version=${VERSION}" >> "$GITHUB_OUTPUT"
|
|
echo "versioned_tag=${TAG}" >> "$GITHUB_OUTPUT"
|
|
echo "short_sha=${SHORT_SHA}" >> "$GITHUB_OUTPUT"
|
|
|
|
# Per-branch slug for dev builds (enables branch-aware dev VMs)
|
|
if [[ "${{ github.ref }}" != "refs/heads/main" ]]; then
|
|
BRANCH_NAME="${GITHUB_REF#refs/heads/}"
|
|
BRANCH_SLUG=$(echo "$BRANCH_NAME" | sed 's|^feature/||' | sed 's|[^a-zA-Z0-9-]|-|g' | tr '[:upper:]' '[:lower:]' | cut -c1-50)
|
|
echo "branch_slug=${BRANCH_SLUG}" >> "$GITHUB_OUTPUT"
|
|
echo "Branch slug: ${BRANCH_SLUG}"
|
|
|
|
# User prefix for <prefix>/<whatever> branches — powers the
|
|
# dev-<prefix>-latest alias tag so each developer's personal VM
|
|
# can pin to their prefix and auto-pull the latest push. Common
|
|
# Git Flow prefixes are skipped so `feature/x`, `fix/y` etc.
|
|
# don't create noisy -latest tags.
|
|
if [[ "$BRANCH_NAME" == *"/"* ]]; then
|
|
USER_PREFIX=$(echo "$BRANCH_NAME" | cut -d/ -f1 | sed 's|[^a-zA-Z0-9-]|-|g' | tr '[:upper:]' '[:lower:]')
|
|
case "$USER_PREFIX" in
|
|
feature|fix|hotfix|bugfix|docs|chore|test|ci|ops|refactor|perf|style|build)
|
|
echo "Branch prefix '$USER_PREFIX' is a Git Flow convention — skipping dev-*-latest alias"
|
|
;;
|
|
*)
|
|
echo "user_prefix=${USER_PREFIX}" >> "$GITHUB_OUTPUT"
|
|
echo "User prefix: ${USER_PREFIX} (will push dev-${USER_PREFIX}-latest alias)"
|
|
;;
|
|
esac
|
|
fi
|
|
fi
|
|
|
|
echo "Channel: ${CHANNEL}"
|
|
echo "Version: ${VERSION}"
|
|
echo "Versioned tag: ${TAG}"
|
|
|
|
- name: Extract package version from pyproject.toml
|
|
id: pkgver
|
|
run: |
|
|
# Single source of truth for the product version: the
|
|
# pyproject.toml [project] table. The CalVer "${YEAR_MONTH}.${N}"
|
|
# claimed above stays as the git / image tag (release identity),
|
|
# but AGNES_VERSION — what /api/version, /cli/latest, and `da
|
|
# --version` all expose — tracks the package version.
|
|
VERSION=$(grep '^version' pyproject.toml | head -1 | sed -E 's/^version\s*=\s*"([^"]+)".*/\1/')
|
|
if [ -z "$VERSION" ]; then
|
|
echo "::error::Could not extract version from pyproject.toml"
|
|
exit 1
|
|
fi
|
|
echo "version=${VERSION}" >> "$GITHUB_OUTPUT"
|
|
echo "Package version: ${VERSION}"
|
|
|
|
- name: Log in to GHCR
|
|
uses: docker/login-action@v4
|
|
with:
|
|
registry: ghcr.io
|
|
username: ${{ github.actor }}
|
|
password: ${{ secrets.GITHUB_TOKEN }}
|
|
|
|
- name: Build and push
|
|
uses: docker/build-push-action@v7
|
|
with:
|
|
push: true
|
|
build-args: |
|
|
AGNES_VERSION=${{ steps.pkgver.outputs.version }}
|
|
RELEASE_CHANNEL=${{ steps.meta.outputs.channel }}
|
|
AGNES_COMMIT_SHA=${{ github.sha }}
|
|
AGNES_TAG=${{ steps.meta.outputs.versioned_tag }}
|
|
tags: |
|
|
ghcr.io/${{ github.repository }}:${{ steps.meta.outputs.channel }}
|
|
ghcr.io/${{ github.repository }}:${{ steps.meta.outputs.versioned_tag }}
|
|
ghcr.io/${{ github.repository }}:sha-${{ steps.meta.outputs.short_sha }}
|
|
${{ steps.meta.outputs.channel == 'dev' && format('ghcr.io/{0}:dev-{1}', github.repository, steps.meta.outputs.branch_slug) || '' }}
|
|
${{ steps.meta.outputs.channel == 'dev' && steps.meta.outputs.user_prefix != '' && format('ghcr.io/{0}:dev-{1}-latest', github.repository, steps.meta.outputs.user_prefix) || '' }}
|
|
|
|
smoke-test:
|
|
needs: build-and-push
|
|
if: github.ref == 'refs/heads/main'
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v5
|
|
|
|
- name: Start Agnes from built image
|
|
run: |
|
|
# Create empty .env (docker-compose.yml requires env_file: .env, gitignored)
|
|
touch .env
|
|
# Use prod compose (GHCR images) + CI overlay (test secrets)
|
|
export AGNES_TAG="${{ needs.build-and-push.outputs.image_tag }}"
|
|
docker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.ci.yml up -d app
|
|
# Wait for healthy (max 60s)
|
|
timeout 60 bash -c 'until curl -sf http://localhost:8000/api/health | python3 -c "import sys,json; d=json.load(sys.stdin); sys.exit(0 if d[\"status\"]!=\"unhealthy\" else 1)"; do sleep 3; done'
|
|
|
|
- name: Run smoke tests
|
|
run: bash scripts/smoke-test.sh http://localhost:8000
|
|
|
|
- name: Collect logs on failure
|
|
if: failure()
|
|
run: docker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.ci.yml logs > smoke-test-logs.txt
|
|
|
|
- name: Upload logs
|
|
if: failure()
|
|
uses: actions/upload-artifact@v4
|
|
with:
|
|
name: smoke-test-logs
|
|
path: smoke-test-logs.txt
|
|
|
|
- name: Teardown
|
|
if: always()
|
|
run: docker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.ci.yml down -v
|