agnes-the-ai-analyst/docker-compose.yml
ZdenekSrotyr 1be997f6d4 feat(caddy): file_server for parquet downloads — bypass uvicorn
A single analyst's multi-GB `agnes pull` held the only uvicorn worker
for the duration of the stream, starving UI / /api/health / every other
API endpoint. Container flipped to `unhealthy`. Triggered while a
6.8 GB `order_economics` pull was in-flight on prod 2026-05-05.

Caddy now intercepts `GET /api/data/{table_id}/download` and serves
the parquet directly via sendfile from the data volume (mounted r-o
at /srv inside the caddy container). RBAC enforced by `forward_auth`
to a new lightweight `GET /api/data/{table_id}/check-access` endpoint
(returns 204 / 403) — the bulk transfer never reaches uvicorn.

Path discovery via `try_files` over the known extract.duckdb v2 source
subdirs. Anything not at a static path falls through to the existing
app handler so legacy `src_data/parquet` and future connectors still
work without a Caddyfile change. Non-Caddy deployments are unchanged.

Stage 1 (multi-worker uvicorn) was considered but blocked by the
single-writer DuckDB lock on system.duckdb — workers > 1 would crash
at startup on "Could not set lock on file", the same race that pushed
the scheduler from in-process writes to HTTP-via-app. Multi-reader
workers + single-writer coordination is out of scope for this PR.
2026-05-05 16:41:33 +02:00

138 lines
4.2 KiB
YAML

services:
app:
build: .
# --proxy-headers + --forwarded-allow-ips make uvicorn honor the
# X-Forwarded-Proto / X-Forwarded-Host headers any reverse proxy (Caddy,
# nginx, Cloudflare Tunnel) sets. Without it, request.url_for() emits
# http://localhost:8000/... even when the user is on https://, which
# breaks OAuth callbacks (redirect_uri_mismatch). Belt-and-suspenders —
# FORWARDED_ALLOW_IPS=* in .env does the same via env var.
command: uvicorn app.main:app --host 0.0.0.0 --port 8000 --proxy-headers --forwarded-allow-ips='*'
ports:
- "8000:8000"
volumes:
- data:/data
- ./config:/app/config:ro
# - ./custom-connectors:/app/connectors/custom:ro # Tier A: AI-generated connectors
env_file: .env
environment:
- DATA_DIR=/data
healthcheck:
test: ["CMD", "curl", "-sf", "http://localhost:8000/api/health"]
interval: 30s
timeout: 5s
retries: 3
restart: unless-stopped
mem_limit: 4g
mem_reservation: 1g
cpus: 2.0
# One-shot: run extractor then rebuild orchestrator views
extract:
build: .
command: >
sh -c "python -m connectors.keboola.extractor &&
python -c 'from src.orchestrator import SyncOrchestrator; print(SyncOrchestrator().rebuild())'"
volumes:
- data:/data
- ./config:/app/config:ro
env_file: .env
environment:
- DATA_DIR=/data
profiles:
- extract
scheduler:
build: .
command: python -m services.scheduler
volumes:
- data:/data
- ./config:/app/config:ro
env_file: .env
environment:
- DATA_DIR=/data
- API_URL=http://app:8000
- SEED_ADMIN_EMAIL=${SEED_ADMIN_EMAIL:-}
depends_on:
app:
condition: service_healthy
restart: unless-stopped
mem_limit: 2g
cpus: 1.0
telegram-bot:
build: .
command: python -m services.telegram_bot
volumes:
- data:/data
env_file: .env
environment:
- DATA_DIR=/data
depends_on:
- app
profiles:
- full
restart: unless-stopped
ws-gateway:
build: .
command: python -m services.ws_gateway
volumes:
- data:/data
env_file: .env
environment:
- DATA_DIR=/data
depends_on:
- app
profiles:
- full
restart: unless-stopped
# NOTE: corporate-memory + session-collector previously ran here as
# tight `restart: unless-stopped` boot loops behind `profiles: [full]`.
# As of #176 the scheduler container drives both through admin HTTP
# endpoints (/api/admin/run-corporate-memory,
# /api/admin/run-session-collector). The verification-detector job
# was never in compose; it now ships the same way. The app remains
# the sole writer to system.duckdb. Operators previously running
# COMPOSE_PROFILES=full need to drop those service stanzas from any
# custom Compose overrides.
# TLS reverse proxy. Corporate-CA certs mounted from /data/state/certs
# (managed by scripts/ops/agnes-tls-rotate.sh on the VM). For local
# development without certs, run without --profile tls and hit :8000
# directly.
caddy:
image: caddy:2-alpine
ports:
- "80:80"
- "443:443"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- /data/state/certs:/certs:ro
- caddy_data:/data
- caddy_config:/config
# Read-only mount of the agnes data dir so Caddy's file_server can
# serve parquets directly (sendfile/zero-copy) and bypass the app's
# uvicorn workers — see Caddyfile's @download handler. Mounted at
# /srv (not /data) because /data is already the caddy_data volume.
- data:/srv:ro
environment:
- DOMAIN=${DOMAIN:-localhost}
# Passes through whatever the operator set in .env. Caddyfile uses
# {$CADDY_TLS:tls /certs/fullchain.pem /certs/privkey.pem} so:
# - unset → cert-file mode (corp PKI rotated by tls-rotate.sh)
# - "tls <email>" → Let's Encrypt auto-issue
# - "tls internal" → Caddy-managed self-signed
- CADDY_TLS
depends_on:
app:
condition: service_healthy
restart: unless-stopped
profiles:
- tls
volumes:
data:
caddy_data:
caddy_config: