agnes-the-ai-analyst/docs/testing/e2e_clean_analyst_bootstrap.md

E2E Verification: clean-analyst-bootstrap (PR #173)

End-to-end verification of the clean-analyst-bootstrap rewrite on a deployed VM. Designed for parallel sub-agent dispatch — Phase 0 prerequisites run sequentially, then 10 parallel slices in a single Claude Code message, then sequential hook test, then aggregation.

Estimated runtime: ~15-20 min total (5 min Phase 0 + 5-10 min Phase 1 parallel + 3 min Phase 2 + 1 min aggregation).

How to use: open Claude Code on the VM in any cwd; paste this entire document into the conversation. Claude Code will execute Phase 0, then dispatch the 10 slices in parallel via Agent tool calls in a single message, then guide you through Phase 2.

Prerequisites — fill these in before starting

• Server URL: https://<your-agnes-host>
• Active user: confirm via web login (Google OAuth or password works)
• Test PAT: mint via web /setup?role=analyst → click Generate prompt → copy clipboard → extract the PAT. Save as $TEST_PAT.
• Picked test tables:
  • LOCAL_TABLE: any query_mode='local' table from agnes catalog --json
  • REMOTE_TABLE: any query_mode='remote' BigQuery table (preferably small or partitioned)
  • REMOTE_WHERE: a simple WHERE clause that filters REMOTE_TABLE to a small subset (e.g., event_date = DATE '2026-01-01')
  • REMOTE_SELECT: 1-2 columns to select from REMOTE_TABLE

---

Phase 0 — Prerequisites (sequential, ~5 min)

# 1. Confirm server reachable + on the new build
curl -fsS "$SERVER_URL/api/health" | head -3
curl -fsS "$SERVER_URL/cli/version" 2>/dev/null || true   # should mention 0.32.0+ post-rename

# 2. Confirm /setup?role=analyst renders the role tiles
curl -fsS "$SERVER_URL/setup?role=analyst" | grep -E "role-tiles|Analyst workspace|const ROLE" | head -5
# Expect: at least 3 matches (CSS class, tile heading, JS const)

# 3. Mint analyst PAT (see Prerequisites above) and export it
export SERVER_URL="https://<your-agnes-host>"
export TEST_PAT="agnes_pat_..."

# 4. Bootstrap the BASE workspace that read-only slices share
mkdir -p /tmp/agnes-e2e-base
cd /tmp/agnes-e2e-base
# Paste the install prompt verbatim from /setup?role=analyst.
# After it finishes:
tree -aL 2 /tmp/agnes-e2e-base | head -30

Gate: if the base workspace doesn't have the expected shape (CLAUDE.md, AGNES_WORKSPACE.md, .claude/settings.json, user/duckdb/analytics.duckdb), STOP and report the failure mode — the rest of the plan assumes a working base workspace.

---

Phase 1 — Parallel slices (dispatch ALL 10 in ONE message, ~5-10 min)

For each slice, dispatch a general-purpose Agent. Send all 10 Agent tool calls in a single response message so they run concurrently. Pass SERVER_URL and TEST_PAT from Phase 0 into each prompt.

Slice 1 — Web UI + role tiles + paste-prompt content

Verify the /setup?role=analyst and /setup?role=admin pages on $SERVER_URL.

Required checks:
- /setup?role=analyst renders 2 role tiles (Analyst workspace + Admin CLI);
  Analyst tile has is-active class, Admin tile is inactive.
- Page contains `const ROLE = "analyst"` (JSON-escaped form, with quotes).
- The JS array `SETUP_INSTRUCTIONS_TEMPLATE` contains `agnes init` and
  `agnes catalog`, does NOT contain `claude plugin marketplace add`,
  `agnes auth import-token`, or `agnes diagnose`.
- /setup?role=admin: admin tile active, prompt contains `agnes auth import-token`.
- /install (no query) returns 302 to /setup.

Tools: curl + grep/python regex.
Report: PASS/FAIL per check + 5 lines of the rendered prompt for both roles.

Slice 2 — agnes init workspace inventory

In /tmp/agnes-e2e-init (NEW empty folder):
  agnes init --server-url $SERVER_URL --token $TEST_PAT --workspace .

Verify EXACTLY this file set is present:
- CLAUDE.md (non-empty, contains "agnes pull")
- AGNES_WORKSPACE.md (does NOT contain $TEST_PAT, no "{placeholder}" leaks,
  contains $SERVER_URL, contains the absolute workspace path,
  has 6 H2 sections matching ^## )
- .claude/settings.json (hooks.SessionStart with `agnes pull --quiet`,
  hooks.SessionEnd with `agnes push --quiet`, has `model` and `permissions.allow`)
- .claude/CLAUDE.local.md (stub content "# My Notes")
- user/duckdb/analytics.duckdb (file exists, non-zero size)

Verify NONE of these exist:
  data/parquet, data/duckdb, data/metadata, user/artifacts, .agnes

Conditional dirs (per lazy-mkdir contract):
- if server/parquet/ exists, must contain ≥1 .parquet
- if .claude/rules/ exists, must contain ≥1 km_*.md

Run: tree -aL 3 /tmp/agnes-e2e-init
Report: full file inventory + PASS/FAIL per assertion.

Slice 3 — Reader smoke matrix (read-only against base workspace)

cd /tmp/agnes-e2e-base
For each command, capture exit code + first stderr line.
Forbidden: any "Traceback" in stderr.

  agnes catalog
  agnes catalog --json
  agnes catalog --metrics
  agnes schema $LOCAL_TABLE
  agnes describe $LOCAL_TABLE -n 5
  agnes status
  agnes status --json
  agnes diagnose
  agnes diagnose system
  agnes disk-info
  agnes auth whoami
  agnes skills list
  agnes skills show agnes-data-querying

Bad-input no-crash:
  agnes schema __nonexistent__
  agnes describe __nonexistent__
  agnes explore __nonexistent__

PASS = all rc ∈ {0, 1}, no Traceback.
Report: command-by-command rc + traceback-Y/N + first stderr line.

Slice 4 — Local + remote query paths

cd /tmp/agnes-e2e-base

Local (parquet/DuckDB) path:
  agnes query "SELECT count(*) FROM $LOCAL_TABLE LIMIT 1"
  agnes query "SELECT * FROM $LOCAL_TABLE LIMIT 3"
  agnes query --json "SELECT count(*) FROM $LOCAL_TABLE"
  agnes explore $LOCAL_TABLE   # friendly even in non-TTY

Remote (BigQuery passthrough) path:
  agnes query --remote "SELECT count(*) FROM $REMOTE_TABLE LIMIT 1"
  agnes query --remote "SELECT count(*) FROM $REMOTE_TABLE" --limit 1

Report: per-query rc, first 5 lines of stdout, traceback-Y/N.
Flag any 502 / cost-guardrail / typed-error envelopes.

Slice 5 — Snapshot lifecycle

cd /tmp/agnes-e2e-base

  agnes snapshot list                                            # baseline
  agnes snapshot create $REMOTE_TABLE --select $REMOTE_SELECT --where '$REMOTE_WHERE' --as e2e_test_snap --estimate
  agnes snapshot create $REMOTE_TABLE --select $REMOTE_SELECT --where '$REMOTE_WHERE' --as e2e_test_snap
  agnes snapshot list                                            # should show e2e_test_snap
  agnes query "SELECT count(*) FROM e2e_test_snap"
  agnes snapshot refresh e2e_test_snap
  agnes snapshot drop e2e_test_snap
  agnes snapshot list                                            # back to baseline

PASS = clean lifecycle, snapshot file present after create + absent after drop.
Report: per-step rc + first stdout line.

Slice 6 — Force / protection scenarios

Setup:
  cp -r /tmp/agnes-e2e-base /tmp/agnes-e2e-force
  cd /tmp/agnes-e2e-force
  echo "# my private edit" > .claude/CLAUDE.local.md

Test 1: re-init without --force should refuse
  agnes init --server-url $SERVER_URL --token $TEST_PAT --workspace .
  Expect rc=1, stderr contains "already initialized" or "partial_state",
  no Traceback.

Test 2: --force regenerates CLAUDE.md but PRESERVES CLAUDE.local.md
  agnes init --server-url $SERVER_URL --token $TEST_PAT --workspace . --force
  Expect rc=0
  Then: cat .claude/CLAUDE.local.md  → must still contain "# my private edit"

Report: PASS/FAIL per test + grep result for the private edit marker.

Slice 7 — Pre-init reader smoke (no-traceback contract)

mkdir -p /tmp/agnes-e2e-pre && cd /tmp/agnes-e2e-pre

For each, capture rc + stderr:
  agnes query "SELECT 1"
  agnes snapshot create __nope__ --as x --estimate
  agnes explore foo
  agnes snapshot list
  agnes status
  agnes catalog
  agnes disk-info

Forbidden: any "Traceback" in stderr.
Expected: rc=1 with friendly hint mentioning "agnes init" or "agnes pull".

Report: per-command rc + traceback-Y/N + did-hint-mention-init-or-pull.

Slice 8 — Auth + token lifecycle

agnes auth whoami     # should print email + role

# Token CRUD
agnes auth token list
TID=$(agnes auth token create e2e-test --expires-in-days 1 | grep -oE 'tok_[a-zA-Z0-9_-]+' | head -1)
agnes auth token list   # new token visible
agnes auth token revoke "$TID"
agnes auth token list   # revoked_at populated for $TID

# Bad token → friendly 401
AGNES_TOKEN=fake-pat agnes catalog 2>&1 | tail -5
# Expect friendly hint, no Traceback

Report: PASS/FAIL per step + the captured TID.

Slice 9 — Admin metrics + catalog --metrics

# Read paths (any analyst)
agnes catalog --metrics
agnes catalog --metrics --show revenue/mrr   # adjust to a real metric path

# Write paths (admin only — skip if the test user isn't admin)
agnes admin metrics --help    # surface check (import/export/validate listed)

If admin:
  agnes admin metrics validate
  agnes admin metrics export /tmp/metrics-backup
  ls /tmp/metrics-backup/ | head

Report: per-command rc + first stdout line.

Slice 10 — AGNES_WORKSPACE.md content quality

cd /tmp/agnes-e2e-base
cat AGNES_WORKSPACE.md | head -100

Programmatic checks:
- Contains "Created:" header line with ISO timestamp
- Contains "Server:" line with $SERVER_URL
- Contains "Workspace:" line with absolute path
- $TEST_PAT does NOT appear anywhere in the file
- No literal "{created_at}", "{server_url}", "{workspace_path}" substrings
- Has exactly 6 H2 sections (^## )
- Cheat sheet section mentions: agnes catalog, agnes query, agnes pull,
  agnes snapshot create, agnes status
- Uninstall section mentions: uv tool uninstall, ~/.config/agnes, ~/.agnes
- For each path in the "Globally installed" table that's a real file path
  (~/.local/bin/agnes, ~/.config/agnes/{config.yaml,token.json}):
  test -e and report exists/missing.

Report: each assertion PASS/FAIL.

---

Phase 2 — Hook behavior (sequential, ~3 min)

Sub-agents can't open Claude Code sessions, so this part is manual.

cd /tmp/agnes-e2e-base
claude                    # opens Claude Code; SessionStart hook fires `agnes pull --quiet`

# Inside Claude Code, ask: "show me 5 rows of $LOCAL_TABLE"  → should work without errors

/exit                     # SessionEnd hook fires `agnes push --quiet`

ls /tmp/agnes-e2e-base/user/sessions/    # should be non-empty (transcript captured)

# Re-enter to verify SessionStart fires again
claude
/exit

Verify in server audit log: 2× agnes pull GETs (one per session start)

• 2× agnes push POSTs (one per session end). Tail the server audit endpoint or DB table to confirm.

---

Phase 3 — Aggregation

Compile a single PASS/FAIL table:

Slice	Status	Notes
1 — Web UI role tiles	…	…
2 — agnes init inventory	…	…
3 — Reader smoke matrix	…	…
4 — Query paths	…	…
5 — Snapshot lifecycle	…	…
6 — Force / protection	…	…
7 — Pre-init no-traceback	…	…
8 — Auth + tokens	…	…
9 — Admin metrics	…	…
10 — AGNES_WORKSPACE.md	…	…
Hooks (Phase 2)	…	…

For any FAIL: preserve the failing folder/output, report the exact command

• first traceback / first stderr line. Don't fix in place — flag for follow-up.

---

Cleanup after testing

rm -rf /tmp/agnes-e2e-base /tmp/agnes-e2e-init /tmp/agnes-e2e-force /tmp/agnes-e2e-pre /tmp/metrics-backup
agnes auth token revoke <e2e-test PAT id>

---

Slice priority

If time is constrained, run these load-bearing slices first:

1. Slice 1 (Web UI role tiles) — proves the web entry point works.
2. Slice 2 (agnes init inventory) — proves the bootstrap creates the exact expected file set with no dead dirs.
3. Slice 7 (pre-init no-traceback) — proves the reader contract holds.
4. Slice 10 (AGNES_WORKSPACE.md content) — proves the human-facing docs render correctly with no PAT leak.

Slices 3-6, 8-9 are breadth coverage — important but lower-priority.

---

Coverage honesty — what this plan reveals (and what it doesn't)

This plan ≠ exhaustive. It catches contract-level bugs — file inventory, no-traceback reader contract, JS ternary direction, lazy-mkdir compliance. It does NOT catch a meaningful slice of real-world failure modes.

What the plan reveals (✅)

• Workspace inventory bug — missing/extra file after agnes init
• No-Python-traceback contract for every reader command
• PAT leak into AGNES_WORKSPACE.md
• JS ternary direction bug — analyst tile mints admin PAT
• Lazy-mkdir violation — pre-allocated empty directories
• Force vs no-force — regenerates CLAUDE.md but preserves CLAUDE.local.md
• Snapshot lifecycle — estimate → fetch → query → drop
• Auth flow — token CRUD, 401 friendly errors
• Help text / flag surface — every CLI command's --help lists expected flags
• Hook command shape — SessionStart→agnes pull, SessionEnd→agnes push

What the plan does NOT reveal (❌)

Gap	Why this plan misses it
Cross-platform (Windows / Linux / macOS)	VM is one OS. setup_instructions.py has 200+ lines of platform-specific TLS logic that never runs
Private CA bootstrap	If VM uses public TLS, the entire step-0 trust block (cert install, OS keychain, ~/.agnes/ca-bundle.pem) stays unexercised
Migration from old da workspace	Greenfield rewrite — but existing analysts have old data. Does agnes init behave well in a folder where da analyst setup previously ran? Unknown
Network failures — slow / packet loss / mid-pull 5xx	Local tests don't mock flaky network
Concurrent users — 2 analysts paste-prompt simultaneously	Single-user happy path only
Disk full during pull	Not simulated
PAT expiry mid-session	1 h TTL → hooks 401 after expiry. What happens to the next session? Untested
Unicode / locale in workspace paths	--workspace /tmp/path with spaces and ěščřž/
Read-only HOME / non-standard permissions	agnes init writes to ~/.config/agnes/ — what if it can't?
Browser variance — paste-prompt clipboard	Different copy semantics across Safari/Chrome/Firefox/clipboards
Claude Code version drift	Spec assumes a specific .claude/settings.json hook schema
Admin CLAUDE.md override edge cases	Custom template without _INIT_MARKER substring — how does init behave?
Long-soak — 1+ day, hooks fire 50×	Only one-shot tests
Real BQ data scale — TB-scale tables, partition cost decisions	Test PATs/tables will be small

Recommended additional coverage layers

Tier 1 (before merge):

1. Run this E2E plan on the VM — ~15-20 min, catches ~70 % of typical bugs.
2. Manual smoke with the actual analyst use-case — the person who'll use this opens a fresh workspace, asks 5 real questions, watches what breaks. Best source of surprises.

Tier 2 (before broad rollout): 3. Soak test — one analyst uses it for a week. Hooks fire ~10×/day. Anything that accumulates (sessions, snapshots, log files) surfaces. 4. Migration test — find an existing da user, walk them through uv tool uninstall agnes-the-ai-analyst → reinstall → agnes auth import-token → agnes init --force in their existing folder. Watch for confusion / breakage.

Tier 3 (nice-to-have): 5. Cross-platform — if you have Windows/Linux users, repeat Phase 0+1 there. 6. Private CA — if you deploy with a private CA, run a separate VM with that configuration and exercise the full trust-bootstrap step. 7. Network chaos — toxiproxy / tc to introduce latency/loss between client and server, verify hooks don't hang sessions.

Realistic coverage estimate

Layer	Coverage of analyst-visible bugs
This plan alone	~70 %
Plan + Tier 1 manual smoke	~80 %
Plan + Tier 1 + Tier 2 soak/migration	~95 %
Plan + all Tiers	~98 %

The remaining 2-5 % surfaces only in real production use across diverse analyst workflows. Rule of thumb: ship after Tier 1 if the surface area is small (< 10 analysts), wait for Tier 2 before scaling to a wider audience.

Prerequisites — what you need on the VM to run this plan

Item	Why	Falls back to
Server on the new build	All slices need the new /setup?role= endpoint + /api/welcome + PAT scope/TTL fields	Auto-upgrade cron picks up :dev/:keboola-deploy-latest within ~5 min of the merge tag landing
Web access (browser)	Phase 0 step 3 — mint a PAT via /setup?role=analyst "Generate prompt" button	Or manual curl -u admin:pw -X POST /auth/tokens if you can't open a browser
Account with grants	test_pat needs resource_grants for ≥ 1 query_mode='local' table + ≥ 1 query_mode='remote' BigQuery table (Slices 4 + 5). Admin role for Slice 9	Slice 4/5/9 skip cleanly if grants missing — other slices still cover the bootstrap path
BigQuery configured server-side	Slice 4 (agnes query --remote) and Slice 5 (snapshot create) hit BQ via the server	Slices skip with a 400/501 error if BQ isn't configured — won't false-fail the plan
agnes on $PATH after Phase 0 step 4	Sub-agent slices invoke agnes directly	If install fails, paste-prompt step itself catches it before any slices run
Network from VM to server	Slices use curl and agnes (which uses httpx)	If network is broken, Phase 0 step 1 detects it and stops

How to dispatch the slices in parallel

If you're running this from Claude Code on the VM:

1. Paste this entire document into the Claude Code conversation.
2. Claude Code will execute Phase 0 sequentially (no choice — each step gates the next).
3. After Phase 0 completes, Claude Code dispatches all 10 slices in a single response message (multiple Agent tool invocations in one block). They run concurrently.
4. Phase 2 (hook test) needs you in the loop — Claude Code will pause and walk you through the manual claude session opens.
5. Phase 3 — Claude Code aggregates and produces the final PASS/FAIL table.

If you're running it manually (no Claude Code agent dispatch), execute the slices serially in 10 terminal tabs / tmux panes. Same content per slice, just no automatic aggregation — you compile the table by hand.