agnes-the-ai-analyst/docs/state-dir.md

State directory layout

Agnes splits its persistent data into two tiers:

Tier	Path	Contents	Backup posture
data	/data	analytics workspace, extracts, DuckDB caches	regenerable
state	${STATE_DIR}	system.duckdb, .session_secret, .jwt_secret, certs/*	irreplaceable

STATE_DIR is an environment variable that selects the host path the state tier is mounted from. Two layouts are supported:

Layout A — nested (legacy default)

sdb at /data
sdc at /data/state    (nested inside the data mount)

STATE_DIR=/data/state (or unset — that's the default). Used by the original deployment topology.

Pros: single bind mount per service (/data:/data recursive). Single env var defaults work.

Cons:

• Bind-mount propagation matters. Non-recursive bind silently shadows the nested sdc mount, causing the app to write to an invisible subdirectory of sdb. Recovery requires docker volume rm + manual data migration.
• Two writers (host's tls-rotate.timer running as root; container app running as uid 999) share /data/state with different mount-namespace views → ownership conflicts.
• Resizing sdb requires unmounting sdc first.

The 2026-05-05 incident in the Groupon FoundryAI deployment was a manifestation of the propagation gotcha. See PRs in this repo and the deployer's infra repo for full root-cause notes.

Layout B — flat

sdb at /data         (analytics, regenerable)
sdc at /data-state   (state, irreplaceable — parallel to /data, not nested)

STATE_DIR=/data-state. Two parallel host binds per service: /data:/data and /data-state:/data-state. Use the docker-compose.flat-mount.yml overlay.

Pros:

• No nested-mount propagation class. Each disk is its own bind.
• Single writer per disk (host scripts → certs on sdc, container app → DuckDB on sdc; both at the same path).
• sdb resize doesn't touch sdc.
• Direct service binds default to recursive in modern Docker — no driver_opts immutability footgun.

Cons:

• One-time per-VM migration: tear down /data/state mount, mount sdc at /data-state instead, copy state contents.
• Two binds per service (slightly more compose YAML).

Choosing

Situation	Recommendation
Existing deployment, no plans to expand	stay on layout A
New deployment	layout B (cleaner, no shadow class)
Existing deployment hit by 2026-05-05 shadow class	migrate to layout B
CI / local dev	neither (use ephemeral compose volumes)

Migration A → B

Steps to move an existing VM from nested to flat:

# 1. Stop containers
sudo docker compose --env-file /opt/agnes/.env \
  -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.host-mount.yml \
  --profile tls down

# 2. Snapshot the existing state
sudo cp -a /data/state /tmp/state-backup-$(date -u +%Y%m%dT%H%M%SZ)

# 3. Unmount sdc from /data/state (its current nested location)
sudo umount /data/state
sudo rmdir /data/state  # remove the now-empty mount point on sdb

# 4. Create the new flat mount point and remount sdc there
sudo mkdir /data-state
echo "LABEL=agnes-state /data-state ext4 defaults,nofail 0 2" | sudo tee -a /etc/fstab
# (also remove the old /data/state line from fstab)
sudo mount /data-state

# 5. Restore state from the backup
sudo cp -a /tmp/state-backup-*/. /data-state/

# 6. Set STATE_DIR in /opt/agnes/.env
echo "STATE_DIR=/data-state" | sudo tee -a /opt/agnes/.env

# 7. Bring the stack back up with the flat overlay
cd /opt/agnes
sudo docker compose --env-file /opt/agnes/.env \
  -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.flat-mount.yml \
  --profile tls up -d

Verify: sudo docker exec agnes-app-1 ls /data-state should show system.duckdb etc.

What reads STATE_DIR

App code:

• src/db.py::_get_state_dir() — the canonical helper. Used by get_system_db() and the schema migration snapshot.
• app/secrets.py::_state_dir() — for .session_secret, .jwt_secret. Mirrors the helper since app/ shouldn't import from src/.
• app/main.py — for the .env_overlay startup file (loaded at process start).

Host scripts:

• scripts/ops/agnes-auto-upgrade.sh — mount-sanity check + cert detection.
• scripts/ops/agnes-tls-rotate.sh — CERT_DIR=$STATE_DIR/certs.

Both scripts source /opt/agnes/.env with set -a, so adding STATE_DIR=/data-state to that file propagates everywhere.

Caddy cert mount

Caddy mounts the cert directory from the host at /certs:ro. The host-side path follows STATE_DIR/certs:

• Layout A: /data/state/certs (in docker-compose.yml directly).
• Layout B: /data-state/certs (overridden in docker-compose.flat-mount.yml).

Compose-time env substitution happens at compose up, not at runtime, so the overlay must be selected at deploy time — there's no single compose YAML that switches based on STATE_DIR.