AI-Cognitive-Leap/agnes-the-ai-analyst
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

User management + PAT + CLI distribution + HTML auth redirect (#9 #10 #11 #12) (#28)

Browse source 
* fix: redirect unauthenticated HTML routes to /login (#10)

* docs(plan): user mgmt + PAT + CLI distribution implementation plan (#9 #10 #11 #12)

* build(docker): produce wheel artifact for /cli/download (#9)

* feat(db): schema v5 — users.active + deactivated_at/by (#11)

* feat(api): /cli/download wheel + /cli/install.sh with baked server URL (#9)

* feat(users): repository supports active flag + count_admins (#11)

* feat(ui): /install page with per-deployment install instructions (#9)

* feat(api): user PATCH/reset-password/set-password/activate/deactivate (#11)

* fix(cli): da login prompts for password and sends it in body (#9)

* test(api): safeguard tests for self-deactivate and last admin (#11)

* feat(auth): reject requests from deactivated users (#11)

* fixup(#10): propagate next through /login buttons + lock down sanitizer tests

* feat(cli): da admin set-role/activate/deactivate/reset-password/set-password (#11)

* feat(ui): /admin/users management page (#11)

* feat(db): schema v6 — personal_access_tokens (#12)

* feat(users): access_tokens repository (#12)

* feat(auth): JWT carries typ (session|pat) and explicit jti (#12)

* feat(auth): reject revoked/expired PATs; update last_used_at (#12)

* feat(api): /auth/tokens CRUD + admin revoke; session-only guard (#12)

* feat(cli): da auth token create/list/revoke (#12)

* feat(ui): /profile page with PAT create/list/revoke (#12)

* docs: PAT usage and session/PAT TTL clarification (#12)

* feat(auth): PAT first-use-from-new-IP audit + last_used_ip (schema v7) (#12)

Closes remaining acceptance gap from issue #12: audit_log entry on first use
of a PAT from an IP that differs from the recorded last_used_ip.

- schema v7: personal_access_tokens.last_used_ip column
- AccessTokenRepository.mark_used now stores the client IP
- get_current_user extracts client IP (X-Forwarded-For first hop, fallback
  to request.client.host) and emits a token.first_use_new_ip audit when the
  IP changes on a subsequent use (not the very first use)
- tests: new-ip audit, same-ip no-op, first-ever-use no-op, schema v7 column

* fix: address Devin review findings on PR #28

- app/main.py: exclude /auth/* from HTML redirect handler so JSON
  endpoints under /auth/ (PAT CRUD used by `da auth token` CLI) keep
  their 401 JSON contract (Devin #1, bug)
- app/api/tokens.py: reject expires_in_days <= 0 explicitly; use
  `is not None` so 0 no longer silently creates a non-expiring token
  (Devin #2)
- app/api/users.py: validate role against Role enum in create_user
  to match update_user and prevent 500 on role-protected requests
  later (Devin #3)
- app/web/templates/admin_users.html: escape user-supplied strings
  before innerHTML; move onclick handlers to addEventListener via
  data attributes so emails with quotes / HTML no longer break the UI
  or enable stored XSS (Devin #4)
- app/auth/router.py, app/auth/providers/{password,google}.py:
  reject deactivated users at login instead of issuing a JWT that
  would then fail on the next request — removes the confusing
  redirect loop (Devin #5)
- CLAUDE.md: document schema v7 instead of stale v4 (Devin #6)
- tests/test_web_ui.py: regression test for the /auth/* JSON 401

* feat(web): add /profile and /admin/users links to dashboard nav

* feat(web): point setup banner at /install page

* chore(web): drop unused setup_instructions context

* fix: address Devin review round 2 on PR #28

- app/api/tokens.py: when expires_in_days is None (the "never" option),
  use a ~100-year JWT expiry so the token doesn't silently die in 24h
  via the session-default fallback in create_access_token. The real
  expiry enforcement stays in verify_token's DB-level check (Devin 🔴)
- app/web/templates/profile.html: escape t.name and other user-supplied
  strings via esc() helper before innerHTML, same pattern as
  admin_users.html. Move revoke onclick to data-attribute +
  addEventListener (Devin 🟡)
- app/api/cli_artifacts.py: use `mktemp -d` with X's at end of template
  for GNU/BSD portability, place wheel inside the temp dir and
  clean up with rm -rf (Devin 🚩)

* feat(web): redesign /install page; make curl one-liner primary, collapse manual

Rebuild the public /install page using the dashboard visual language
(shared header, card layout, gradient hero, design tokens from
style-custom.css). The page is now anchored on the one-liner install
path: curl -fsSL <server>/cli/install.sh | bash is rendered as the
primary, prominent step 1, while the old manual wheel-download flow
is tucked behind a closed-by-default <details> block for users in
restricted/offline environments.

Information architecture:
  hero (server URL + version)
  -> step 1: quick install (one-liner, big Copy button)
  -> step 2: create PAT on /profile + export DA_TOKEN / da auth whoami
  -> step 3: Claude Code / MCP via ~/.config/da/token.json
  -> collapsed "Manual install" details for download-wheel flow
  -> footer link to docs/HEADLESS_USAGE.md

Every shell snippet has a vanilla-JS "Copy" button that confirms
visually ("Copied!" for 1.5s) and falls back to textarea+execCommand
on non-secure contexts. No new dependencies, no bundler.

The route now also pulls an optional user so the header shows the
same nav (Dashboard / Profile / Logout) as dashboard.html when a
session exists, while staying fully public when signed out.

* fix(cli): use real wheel filename in install.sh (broken pip/uv install)

The installer wrote the downloaded wheel as agnes_cli.whl, which lacks a
PEP-427 version component — both pip and uv tool install reject it and
abort the one-liner.

Use curl -OJ so Content-Disposition determines the on-disk filename, then
resolve it via glob. Install an EXIT trap to remove the tmpdir even when
install fails.

* fix(web): correct manual install wheel glob and add PEP 668 / PATH hints

- Wheel glob is agnes_the_ai_analyst-*.whl (not agnes-*.whl) — the old
  pattern never matched the real artefact name from the build.
- Add — or — separator between uv tool install and pip install.
- Warn that pip install --user is blocked on macOS Homebrew / modern
  Debian (PEP 668) and recommend uv tool install as the default path.
- Both flows now show the ~/.local/bin PATH hint so a fresh shell can
  find the da binary after install.

* fix(web): consistent session.user reference in install header

The avatar-letter fallback inside {% if session.user %} was reading
user.name / user.email directly, but the route dependency can pass
user=None — those references resolved to an empty FlexDict and produced
an empty avatar circle. Read everything through session.user to match
the guard and the dashboard pattern.

* fix(web): point headless usage link at GitHub source

/docs/HEADLESS_USAGE.md 404s — no static route serves repo docs. Point
the footer link at the rendered markdown on GitHub instead of adding a
dedicated docs serving route just for one file.

* feat(web): /install hero size, anon sign-in banner, step 2 copy polish

- Bump hero h1 from 26px to 30px to match dashboard primary scale.
- Anonymous visitors see a small sign-in banner above Step 2 (creating
  a token requires auth; without the banner the flow appears stuck).
- Add an 'After generating your token' section label inside Step 2 so
  the /profile CTA button no longer looks wedged mid-sentence between
  adjacent paragraphs.

* chore(web): /install a11y + version pill polish

- aria-live='polite' on copy buttons so screen readers announce the
  'Copied!' state change.
- Replace redundant INSTANCE_NAME eyebrow (already in the header logo)
  with 'Getting started'.
- Hide the version pill when AGNES_VERSION is unset/'dev' — avoids the
  misleading 'vdev' label in local/unbuilt runs.
- Manual summary focus-visible outline-offset +2px (was -2px which
  clipped inside the card), and mark the chevron as decorative.

* fix(web): use session.user in dashboard avatar fallback

Inside {% if session.user %} guard, the avatar fallback referenced
(user.name or user.email). If user is None the block crashes when
the profile picture is absent. Align with the guard variable.

* fix: address Devin review round 3 on PR #28

- app/api/users.py: stop auto-sending email from reset_password. The
  magic-link sender would deliver a "Login Link" that — when clicked —
  consumes the reset_token via verify_magic_link and logs the user in
  WITHOUT prompting for a new password. Admins now share the raw
  reset_token from the API response manually, or use set-password
  directly. email_sent is always False. Documented inline. (Devin 🟡)
- app/api/cli_artifacts.py: harden /cli/install.sh generation against
  shell injection via Host header or AGNES_VERSION. base_url is
  validated against a strict scheme+host+port regex; version against
  an alnum + dot/dash/underscore allowlist. Both values are also
  piped through shlex.quote() as defense in depth. (Devin 🟡)

The shared users.reset_token column between magic-link and password-
reset flows (Devin 🚩) remains an architectural gap; splitting into
separate columns needs schema v8 and is tracked for a follow-up PR.

* docs, chore(grpn): manual-deploy helpers + hackathon deploy learnings

Adds scripts/grpn/ — Makefile + agnes-auto-upgrade.sh + README for
operating Agnes on GRPN's existing foundryai-development VM when the
full Terraform flow is blocked by org policies:

- iam.disableServiceAccountKeyCreation (org constraint) forbids SA
  JSON keys, so GCP_SA_KEY-based CI is unavailable
- No projectIamAdmin delegation → bootstrap-gcp.sh can't grant roles
- Secret Manager IAM bindings require setIamPolicy which editor lacks

Helper targets: deploy, deploy-tag, recreate, restart, stop, start,
status, version, logs, ps, env, ssh, tunnel, open, bootstrap-admin,
set-data-source, install-cron, uninstall-cron.

docs/superpowers/plans/2026-04-22-grpn-deploy-learnings.md — running
log of all org-policy constraints hit during the hackathon deploy,
with workarounds and derived follow-ups (WIF support, external_ip
variable, customer onboarding IAM checklist).

Not a replacement for the TF flow — stopgap until WIF lands.

* fix(web): make header logos clickable links to home

* feat(web): one-click "Setup a new Claude Code" button

Adds a single-button flow on the dashboard and /install page that
generates a fresh personal access token via POST /auth/tokens and
copies a complete, paste-ready setup script (server URL, token,
install/verify commands) to the clipboard. Falls back to a modal
textarea when the clipboard is blocked; redirects to /login on 401;
surfaces backend errors inline.

- dashboard.html: replaces the top "Set up your local environment"
  anchor with a real button wired to setupNewClaude(). Removes the
  duplicate bottom setup banner to keep a single entry point.
- install.html: for signed-in users, Step 1 leads with the one-click
  button and demotes the curl one-liner into a collapsible "Or run
  manually" aside. Anonymous visitors still see the curl flow plus a
  sign-in hint.
- No new deps. Vanilla JS. Token lives in memory/clipboard only —
  never rendered into persistent DOM.

* feat(cli): add "da auth import-token" for non-interactive PAT login

Writes a provided JWT into ~/.config/da/token.json using the canonical
{access_token, email, role} shape expected by save_token(). Decodes the
token locally to pull email/role claims, verifies it against the server
via GET /api/catalog/tables, and refuses to overwrite an existing token
file if the server returns 401. --email / --role overrides exist for
tokens missing those claims; --skip-verify bypasses the server round-trip
for offline / CI scenarios.

* test(cli): cover da auth import-token success + 401 + claim-fallback paths

Three new tests in TestAuthImportToken:
- valid JWT + 200 -> canonical token.json written
- 401 from /api/catalog/tables -> exit 1, existing token file untouched
- JWT without email/role claims -> refused without overrides, accepted
  with --email / --role flags

* feat(web): update one-click Claude setup instructions — explicit uv install, import-token, skills question

Replaces the fragile `cat > token.json <<EOF` clipboard payload with an
explicit, auditable sequence:

  1. `curl -fsSL /cli/download` + `uv tool install --force` (no opaque
     `curl | bash`).
  2. `da auth import-token --token ...` instead of hand-written JSON.
  3. Explicit PATH persistence for zsh/bash.
  4. A required question to the user about whether to copy the bundled
     skills into ~/.claude/skills/agnes/ or pull them on-demand via
     `da skills show`.
  5. A final confirmation step with whoami + version output.

Factored both pages to include a shared partial
(app/web/templates/_claude_setup_instructions.jinja) so dashboard.html
and install.html can never drift apart again. {server_url} and {token}
stay as runtime placeholders substituted by renderSetupInstructions().

* feat(ui): modernize /admin/users + unify header nav across pages

- New shared partial app/web/templates/_app_header.html — single source
  of truth for the top navigation. Used by base.html and dashboard.html
  (which doesn't extend base.html). Active page highlighted via
  request.url.path. Admin "Users" link gated by session.user.role.
- style-custom.css: add .app-header / .app-nav-link / .app-btn-logout /
  .app-avatar styles (mirrors dashboard's previous inline copy under
  app-* prefix). Mobile-friendly fallback at <720px.
- base.html: include the new partial so every page extending base
  (admin_users, profile, login_email, error, …) gets the same chrome
  the dashboard has.
- dashboard.html: replace its inline <header class="header"> markup
  with the shared partial. Inline .header CSS left in place as
  harmless dead code (separate cleanup PR).
- admin_users.html: rewritten with avatars, role pills (color-coded
  per role), toggle switch for active, search/filter input, toast
  notifications, modal dialogs replacing alert/confirm/prompt,
  one-click copy for the reset token, empty / loading states.
  All XSS-safe via the existing esc() helper + data-attribute
  event delegation.
- tests/test_web_ui.py: smoke test that /admin/users renders the new
  shared header chrome and the modernized markup.

* feat(api): serve CLI wheel at /cli/agnes.whl for direct uv install

uv tool install inspects the URL path suffix to recognise a wheel, so
/cli/download (which has no .whl suffix) cannot be installed directly.
Expose a stable /cli/agnes.whl alias over the same wheel lookup so users
can run: uv tool install --force https://<server>/cli/agnes.whl

* test(cli): cover da auth import-token --server persisting to config.yaml

The server persistence was already implemented in the import-token command
(save_config({server}) call) but not covered by tests. Add an explicit test
so the one-step setup contract — single import-token call writes both token
and server — cannot regress.

* feat(web): simpler Claude setup — single uv install URL, single import-token call

User feedback: the prior clipboard payload repeated the server URL and
token across multiple steps (curl + tmpfile + install + rm + separate
seed-config + import-token). Collapse to:

 1. uv tool install --force {server_url}/cli/agnes.whl  (single URL, direct)
 2. da auth import-token --token ... --server ...        (one call, persists both)
 3. da auth whoami
 4. skills (ask user first)
 5. confirm

uv accepts HTTPS URLs that end in .whl and installs them directly, so
the tmpfile dance is unnecessary. import-token --server already persists
the server to config.yaml, so no separate printf > config.yaml step.

* fix(tests): update admin users heading assertion after template rename

The admin_users.html template now uses <h2 class="users-title">Users</h2>
instead of <h2>User management</h2>. Update the assertion to match.

* feat(ui): unify header across remaining 7 standalone pages

These 7 pages render their own full <html> and don't extend base.html,
so the previous unification commit only covered base + dashboard. Each
had its own ad-hoc <header> markup with inconsistent classes
(.top-header / .header / .page-header), inconsistent nav-link sets,
and inconsistent avatar/email styling.

Replace each inline <header>...</header> block with the shared
{% include '_app_header.html' %} so /activity-center, /admin/permissions,
/admin/tables, /catalog, /corporate-memory, /corporate-memory/admin,
and /install all show the same chrome (Dashboard / Install CLI /
Profile / Users / email + avatar / Logout) with the active page
highlighted via request.url.path.

Old inline header CSS (.header, .top-header, .page-header, .nav-link,
etc.) is left in place as harmless dead code; it can be cleaned up in
a follow-up sweep.

* feat(web): add readable preview of Claude setup payload on dashboard + /install

Move the line-by-line setup instructions into app/web/setup_instructions.py
as the single source of truth, then render them in two modes from the
existing _claude_setup_instructions.jinja partial:

- preview_mode=True  → visible, read-only <pre><code> block with the real
  server URL and a clearly-styled placeholder token (never a real one).
- preview_mode=False → the JS SETUP_INSTRUCTIONS_TEMPLATE used by the
  one-click flow (unchanged behaviour).

Both /dashboard (env-setup-cta card) and /install (Step 1 card) now show
the preview directly under the 'Setup a new Claude Code' button so users
can see exactly what will land in their clipboard before they click.

* feat(web): update setup instructions — `da diagnose` step, explicit section titles

Rework the Claude Code setup payload to:

- Give every numbered step an unambiguous verb header ("1) Install the CLI",
  "2) Log in", "3) Verify the login", "4) Run diagnostics", "5) Skills (ask
  the user first)", "6) Confirm").
- Add step 4 `da diagnose` as the post-login health check. The CLI already
  ships this command (cli/commands/diagnose.py); it prints "Overall:
  healthy" and a list of green checks that map cleanly to next actions.
- Ask the skills copy-vs-on-demand question verbatim so Claude Code always
  prompts the user the same way.
- Replace the terse "Confirm" line with a 4-bullet summary (version,
  whoami, skills choice, diagnose status) so the return message is
  structured and comparable across setups.

* chore(web): remove stale MCP card from /install (no MCP server today)

The 'Use with Claude Code / MCP' card (Step 3 on /install) referenced an
MCP integration Agnes does not ship. Remove the whole card. The one-click
'Setup a new Claude Code' flow in Step 1 already covers the long-lived
client use case and is less confusing than dangling persistence tips for
a non-existent integration.

* feat(api): include user_email + last_used_ip + user_id in admin tokens list response

Adds AdminTokenItem response model (superset of TokenListItem) and
AccessTokenRepository.list_all_with_user() joining personal_access_tokens
with users to denormalize user_email. Needed for /admin/tokens UI where
admins triage tokens across all users.

* feat(web): /admin/tokens page — list, filter, search, revoke across all users

Adds a new admin-only page with client-side filtering (status, user email,
last-used window), column sorting, counts bar (active/revoked/expired),
and an inline revoke action. Mirrors the /admin/users visual language.

* feat(web): add Tokens nav link for admins + deep-link from admin/users row

Admin-only nav entry to /admin/tokens, and a per-row Tokens button on
/admin/users that prefills the token page's user filter via ?user=<email>.

* test(admin): cover /admin/tokens rendering, filter state, non-admin denial, revoke

Verifies admin can render the page (title + JS hooks present), a non-admin
is blocked, unauthenticated users are redirected, the admin list response
includes user_email / user_id / last_used_ip, and admin can revoke another
user's token.

* feat(web): modern redesign of /admin/tokens — hero, stat strip, refined table, responsive cards, a11y

* feat(web): ditch the table — /admin/tokens as a card stack, modern GitHub-style list

Replaces the table-based layout with a stack of self-contained token cards
inside a <ul role=list>. Each card is a flex row: avatar + name/meta on the
left, last-used block in the middle, status pill + outlined 'Revoke' button
on the right. Status and sort controls are pill-shaped toggle chips; user
email search has an inline search icon. No <table>/<tr>/<th>/<td> anywhere.
Responsive below 720px (card stacks vertically) and 480px (stat chips 2x2).
Preserves filter IDs (flt-status, flt-user, flt-last-used) and data-revoke
for existing tests.

* feat(web): add /tokens (role-aware) — single page for both user PAT CRUD and admin overview

- Rename admin_tokens.html -> tokens.html with a new is_admin context flag.
- New route GET /tokens: renders the same card-stack UI for everyone.
  * Admins: loads /auth/admin/tokens, shows owner column + stat strip, keeps
    the owner-email search box and sort-by-owner chip.
  * Non-admins: loads /auth/tokens (own tokens only), hides owner column +
    stat chips, adds a 'New token' CTA in the hero that opens a modal
    (name + expires_in_days) calling POST /auth/tokens. The raw token is
    revealed once in a dismissable banner and cleared from the DOM on Hide.
- GET /admin/tokens now 302-redirects to /tokens, preserving query string
  (so the /admin/users deep-link ?user=foo still works).

* feat(web): /tokens full-bleed layout to match dashboard width

The hero, toolbar, and card list used to sit inside base.html's .container
(max-width 800px). Break out with negative horizontal margins so the page
spans the viewport like /dashboard does, capped at 1440px for readability
on very wide screens with a 24px gutter on each side.

- No change to base.html itself. The override is scoped to .tokens-page.
- body { overflow-x: hidden; } guards against rare horizontal scrollbars.
- < 808px viewport: reset to natural flow (mobile already narrower).
- ≥ 1488px viewport: cap to 1440px and re-center.

* chore(web): remove /profile template + nav link (redirect /profile -> /tokens)

The old /profile PAT CRUD page is now redundant — the modern /tokens page
covers both user and admin flows. Delete the template; the router's
/profile handler already 302-redirects to /tokens.

Nav cleanup:
- Remove the 'Profile' link.
- Show a single 'Tokens' link to every signed-in user (previously only
  admins saw it).
- Active-state matches /tokens, /admin/tokens, and /profile so the
  highlight survives the redirect chain.

/install CTA now points at /tokens instead of /profile.

* test: cover /tokens for admin + non-admin flows, /profile redirect, nav update

tests/test_admin_tokens_ui.py
- Point admin rendering test at /tokens directly and tighten assertions
  (admin-only stat strip + owner search, non-admin CTA absent).
- Add test_non_admin_can_render_tokens_page: personal body, New-token CTA,
  create-modal, reveal banner; stat strip + owner search absent.
- Add test_admin_tokens_redirects_to_tokens: 302 to /tokens, query string
  (?user=...) preserved for the /admin/users deep-link.
- Add test_profile_redirects_to_tokens: 302 to /tokens.
- Add test_non_admin_can_create_pat_via_tokens_page_api: exercises the
  POST /auth/tokens call that the non-admin create-modal submits.

tests/test_pat.py
- test_profile_page_renders -> test_profile_page_redirects_to_tokens:
  assert the 302 + that /tokens lands on the unified non-admin body.

tests/test_web_ui.py
- admin_users nav assertion: 'Tokens' link present, 'Profile' link absent.
- Add test_nav_shows_tokens_link_for_non_admin: non-admins see the same
  'Tokens' link (previously only admins did).
- Add test_profile_redirects_to_tokens back-compat check.

* feat(web): collapse 'What Claude Code will receive' by default

The preview block on /dashboard and /install now uses <details>/<summary>
so it is hidden by default. Click the chevron/title to expand and review
the clipboard payload. Markup stays in the DOM so existing tests that
assert on content continue to pass.

* fix(web): /tokens width — override .container to 1280px like dashboard

The negative-margin full-bleed trick was fragile and pushed content past
the right edge on deployed viewports. Replace with a simple max-width
override of base.html's .container on this page only, matching
/dashboard's 1280px center-column layout.

* feat(web): split role-aware /tokens into my_tokens.html + admin_tokens.html

* feat(web): router — separate handlers for /tokens (own) and /admin/tokens (all)

* feat(web): nav — show Tokens for all, add All tokens for admins

* test: cover split token pages (own vs all) + admin access gating

* feat(web): move 'My tokens' into a user dropdown menu

Replaces the separate Tokens/email/Logout nav trio with a rounded
avatar trigger that opens a dropdown containing the user's email,
role, a 'My tokens' link, and Logout. Admin-only 'All tokens' stays
as a top-level nav item since it's an admin function, not a personal
one. Click-outside and Escape close the panel; chevron rotates on
open.

* fix(api): allow PATs to list/get/revoke their own tokens (CLI flow)

The documented 'da auth token list/revoke' CLI flow in
docs/HEADLESS_USAGE.md uses a PAT, but the previous dependency
(require_session_token) returned 403. Only create_token must be
session-only to prevent PAT-spawning-PAT chains; listing and
revoking your own tokens is safe with a PAT.

* fix(api): cap expires_in_days at 3650 to avoid datetime overflow (500 to 400)

Values above ~11 million days overflowed datetime.max in
datetime.now(utc) + timedelta(days=...) and surfaced as an
unhandled OverflowError → 500. Cap at 10 years with a clear
400 instead; the no-expiry code path is unaffected.

* fix(api): relax _SAFE_URL_RE to allow path prefixes, underscores, and IPv6

The previous regex rejected legitimate reverse-proxy base_url values
(https://host/agnes/), underscores in Docker Compose hostnames, and
IPv6 literals (http://[::1]:8000). Widen the charset and allow an
optional trailing path. shlex.quote continues to provide
defense-in-depth against any metacharacter that slips through.

* fix(web): /login/email and Google OAuth propagate next_path

Previously, /login/email silently dropped the ?next=<path> query
param so the hidden form field rendered empty and login always
landed on /dashboard. Google's button was hard-coded to
/auth/google/login, ignoring next entirely.

- /login page now appends ?next to the Google button URL
- /login/email reads + sanitizes next, passes as template context
- google_login stashes sanitized next_path in session['login_next']
- google_callback pops + re-sanitizes and redirects there

Sanitization factored into app/auth/_common.safe_next_path.

* fix(auth): differentiate argon2 VerifyMismatchError from internal errors in web login

The previous except (VerifyMismatchError, Exception) collapsed both
cases into the generic 'invalid credentials' redirect, silently
hiding corrupted-hash / library errors from ops. Split the two:
bad password still gets ?error=invalid; anything else logs via
logger.exception and redirects with ?err=auth_internal so ops have
a visible signal and users don't retry forever against a broken
password_hash column.

* docs: correct CLAUDE.md table name (personal_access_tokens)

v7 note referenced 'access_tokens.last_used_ip' but the real table
is personal_access_tokens (as mentioned two tokens earlier in the
same bullet). Same-file consistency fix.

* chore(web): clarify admin user-reset UI — encourage Set password over the unused reset_token

POST /api/users/{id}/reset-password stores and returns a token
but no endpoint consumes it — the magic-link sender would log the
user in without prompting for a new password, defeating the reset.
- Drop the 'Reset' row action from admin_users so admins aren't
  pointed at a dead end.
- Rewrite the reveal-modal copy to tell admins to use Set password
  and explicitly note that the magic-link flow isn't available
  for reset tokens in this build.
The API endpoint stays for API-level future use.

* test: cover PAT CLI flow, expires_in_days overflow, proxy base_url, next propagation

- tests/test_pat.py: PAT can list own tokens (200, was 403);
  PAT can revoke own tokens (204); create_token returns 400 for
  expires_in_days > 3650 (was 500 via datetime overflow).
- tests/test_cli_artifacts.py: _SAFE_URL_RE accepts reverse-proxy
  path prefixes, underscores, and IPv6 literals; end-to-end check
  of cli_install_script with a stubbed base_url that includes
  a path prefix (Agnes behind /agnes/).
- tests/test_web_ui.py: /login propagates ?next to the Google
  button URL; /login/email renders next in the hidden form field
  and strips hostile values; unit coverage of safe_next_path.

* fix(security): use \Z instead of $ in URL/version allowlists (trailing-\n bypass)

Python regex `$` also matches just before a trailing newline, so a Host
header or AGNES_VERSION value like "good.example.com\n$(rm -rf /)"
would slip past the allowlist. `\Z` anchors to strict end-of-string.

shlex.quote downstream remains as defense-in-depth, but the allowlist
is now the tight gate it claims to be.

* fix(auth): PAT with null expiry omits JWT exp claim (DB is the source of truth)

Previously a PAT created with `expires_in_days=null` (user-requested
"never expires") set the DB `expires_at` to NULL (correct) but still
baked a ~100y `exp` claim into the JWT. That is misleading: the PAT
silently did expire eventually, despite the UI and API promising
"no expiry".

`create_access_token` now accepts `omit_exp=True` to skip the `exp`
claim entirely. `app/api/tokens.py` passes that when `expires_in_days
is None`. The authoritative expiry check lives in
`app/auth/dependencies.py`, which reads `expires_at` from the DB row —
unchanged. PyJWT accepts claim-less JWTs indefinitely.

* test: cover trailing-newline regex bypass + no-exp JWT for unbounded PAT

- test_safe_url_re_rejects_trailing_newline_bypass: asserts both
  `_SAFE_URL_RE` and `_SAFE_VERSION_RE` reject values with a trailing
  `\n` (previously accepted because Python `$` matches before `\n`).
- test_pat_null_expiry_jwt_has_no_exp_claim: POST /auth/tokens with
  `expires_in_days=null`, decode the returned JWT, assert `exp` is
  absent while `typ=pat`, `sub`, and `jti` are still present.
- test_pat_with_null_expiry_is_accepted_by_verify_token: verify_token
  round-trips a claim-less JWT without ExpiredSignatureError.
- test_pat_null_expiry_end_to_end_allows_authenticated_request: use
  the null-expiry PAT against /auth/tokens and confirm it authenticates.

* docs(auth): document X-Forwarded-For trust model in _client_ip

Deployment runs behind Caddy which strips incoming X-Forwarded-For
and sets its own, so the leftmost hop is trustworthy. Clarify that
the stored last_used_ip is audit-only and never used for access
control — if the app is ever exposed directly, this value becomes
client-settable.

* docs: /profile → /tokens in install.sh next-steps, CLI error, HEADLESS_USAGE, security skill

After splitting PAT management to /tokens (with /profile as a back-compat
302), stale references remained in user-facing text. Update them to the
canonical /tokens URL so shell scripts, CLI error hints, docs, and the
bundled security skill are all consistent.
This commit is contained in:
ZdenekSrotyr 2026-04-22 14:24:28 +02:00 committed by GitHub
parent 963db420fe
commit d2c76cb221
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
61 changed files with 18344 additions and 269 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
CLAUDE.md
View file

@ -186,7 +186,7 @@ Auth providers in `app/auth/` (FastAPI-based):
## Key Implementation Details
### DuckDB Schema (src/db.py)
- Schema v4 with auto-migration from v1→v2→v3→v4
- Schema v7 with auto-migration from v1→v2→v3→v4→v5→v6→v7 (v5 adds `users.active`, v6 adds `personal_access_tokens`, v7 adds `personal_access_tokens.last_used_ip`)
- `table_registry`: id, name, source_type, bucket, source_table, query_mode, sync_schedule, etc.
- `sync_state`, `sync_history`: track extraction progress
- `users`, `dataset_permissions`, `audit_log`: auth + RBAC

7
Dockerfile
View file

@ -1,9 +1,7 @@
FROM python:3.13-slim
# Install curl for healthcheck
RUN apt-get update && apt-get install -y --no-install-recommends curl && rm -rf /var/lib/apt/lists/*
# Install uv for fast dependency management
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
ARG AGNES_VERSION=dev
@ -15,12 +13,13 @@ ENV AGNES_COMMIT_SHA=${AGNES_COMMIT_SHA}
WORKDIR /app
# Copy application code
COPY . .
# Build wheel artifact (served at /cli/download)
RUN uv build --wheel --out-dir /app/dist
# Install production dependencies from pyproject.toml
RUN uv pip install --system --no-cache .
# Default: run FastAPI server
EXPOSE 8000
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

140
app/api/cli_artifacts.py Normal file
View file

@ -0,0 +1,140 @@
"""CLI artifact download + install script endpoints (#9)."""
import os
import re
import shlex
from pathlib import Path
from fastapi import APIRouter, HTTPException, Request
from fastapi.responses import FileResponse, PlainTextResponse
# Strict allowlists for values interpolated into the generated install.sh.
# The endpoint is unauthenticated and users `curl | bash` it, so any shell
# metacharacter leaking through the Host header or AGNES_VERSION env var
# would become RCE. `shlex.quote` is applied on top for defense in depth.
#
# Host charset allows underscores (Docker Compose hostnames) and `[` `]` `:`
# so IPv6 literals like http://[::1]:8000 pass. Optional trailing path lets
# reverse-proxy deployments (request.base_url = "https://host/agnes/") work.
#
# `\Z` (not `$`) anchors strictly to end-of-string. Python's `$` also matches
# immediately before a trailing `\n`, which would let a crafted Host header
# like "good.example.com\n$(rm -rf /)" slip past the allowlist. `\Z` closes
# that bypass — shlex.quote downstream is still defense-in-depth.
_SAFE_URL_RE = re.compile(r"^https?://[A-Za-z0-9._\-\[\]:]+(:\d+)?(/[A-Za-z0-9._\-/]*)?\Z")
_SAFE_VERSION_RE = re.compile(r"^[A-Za-z0-9._\-]+\Z")
router = APIRouter(tags=["cli"])
def _dist_dir() -> Path:
return Path(os.environ.get("AGNES_CLI_DIST_DIR", "/app/dist"))
def _find_wheel() -> Path | None:
d = _dist_dir()
if not d.exists():
return None
wheels = sorted(d.glob("*.whl"))
return wheels[-1] if wheels else None
@router.get("/cli/download")
async def cli_download():
wheel = _find_wheel()
if not wheel:
raise HTTPException(
status_code=404,
detail=(
"CLI wheel not found in dist dir. Build it with `uv build --wheel` "
"or run the official docker image (which builds on image-build)."
),
)
return FileResponse(
path=str(wheel),
filename=wheel.name,
media_type="application/octet-stream",
headers={"Content-Disposition": f'attachment; filename="{wheel.name}"'},
)
@router.get("/cli/agnes.whl")
async def cli_wheel_stable():
"""Stable `.whl` URL alias so `uv tool install <server>/cli/agnes.whl` works.
`uv tool install` inspects the URL path to decide how to treat the resource
and only accepts it as a wheel when the path ends in `.whl`. The existing
`/cli/download` path does not, which forces users through a multi-step
curl + tmpfile + install + rm dance. This alias collapses that into a
single `uv tool install` invocation.
"""
wheel = _find_wheel()
if not wheel:
raise HTTPException(
status_code=404,
detail=(
"CLI wheel not found in dist dir. Build it with `uv build --wheel` "
"or run the official docker image (which builds on image-build)."
),
)
return FileResponse(
path=str(wheel),
filename=wheel.name,
media_type="application/octet-stream",
headers={"Content-Disposition": f'attachment; filename="{wheel.name}"'},
)
@router.get("/cli/install.sh", response_class=PlainTextResponse)
async def cli_install_script(request: Request):
"""Shell installer — bakes this server's URL into the generated config."""
base_url = str(request.base_url).rstrip("/")
if not _SAFE_URL_RE.match(base_url):
raise HTTPException(status_code=400, detail="Unexpected server URL format")
version = os.environ.get("AGNES_VERSION", "dev")
if not _SAFE_VERSION_RE.match(version):
version = "dev"
# shlex.quote hardens against anything that slipped past the regex
server_q = shlex.quote(base_url)
version_q = shlex.quote(version)
script = f"""#!/usr/bin/env bash
# Agnes CLI installer — server: {base_url}
set -euo pipefail
SERVER={server_q}
echo "Installing Agnes CLI from $SERVER (version: {version_q})"
# 1. Download the wheel
# Portable mktemp: X's must be at the end of the template on both GNU and BSD/macOS.
TMPDIR_WHEEL=$(mktemp -d -t agnes_cli.XXXXXX)
trap 'rm -rf "$TMPDIR_WHEEL"' EXIT
# Use -OJ so curl honours Content-Disposition and saves the wheel with its real
# PEP-427 filename (pip / uv tool install reject filenames without a version).
(cd "$TMPDIR_WHEEL" && curl -fsSL -OJ "$SERVER/cli/download")
WHEEL=$(ls "$TMPDIR_WHEEL"/*.whl 2>/dev/null | head -n1)
if [ -z "$WHEEL" ]; then
echo "error: wheel download failed (no .whl found in $TMPDIR_WHEEL)" >&2
exit 1
fi
# 2. Install via pip (prefer uv tool install if available)
if command -v uv >/dev/null 2>&1; then
uv tool install --force "$WHEEL"
else
python3 -m pip install --user --force-reinstall "$WHEEL"
fi
# 3. Seed the server URL in CLI config
CFG_DIR="${{DA_CONFIG_DIR:-$HOME/.config/da}}"
mkdir -p "$CFG_DIR"
cat > "$CFG_DIR/config.yaml" <<EOF
server: $SERVER
EOF
echo "Installed."
echo "Next steps:"
echo " 1. Sign in to $SERVER and create a personal access token at $SERVER/tokens"
echo " 2. Export it: export DA_TOKEN=<your-token>"
echo " 3. Verify: da auth whoami"
"""
return script

197
app/api/tokens.py Normal file
View file

@ -0,0 +1,197 @@
"""Personal access token endpoints (#12)."""
import hashlib
import secrets
import uuid
from datetime import datetime, timezone, timedelta
from typing import Optional, List
import duckdb
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from app.auth.dependencies import require_session_token, require_role, get_current_user, _get_db
from src.rbac import Role
from src.repositories.access_tokens import AccessTokenRepository
from src.repositories.audit import AuditRepository
from app.auth.jwt import create_access_token
router = APIRouter(prefix="/auth/tokens", tags=["tokens"])
admin_router = APIRouter(prefix="/auth/admin/tokens", tags=["tokens-admin"])
class CreateTokenRequest(BaseModel):
name: str
expires_in_days: Optional[int] = 90 # null = no expiry
class CreateTokenResponse(BaseModel):
id: str
name: str
prefix: str
token: str # raw token — returned exactly once
expires_at: Optional[str]
created_at: str
class TokenListItem(BaseModel):
id: str
name: str
prefix: str
created_at: str
expires_at: Optional[str]
last_used_at: Optional[str]
revoked_at: Optional[str]
class AdminTokenItem(TokenListItem):
"""Admin list row: adds owner identity + last IP for incident response."""
user_id: str
user_email: Optional[str] = None
last_used_ip: Optional[str] = None
def _audit(conn, actor: str, action: str, target: str, params=None):
try:
AuditRepository(conn).log(user_id=actor, action=action,
resource=f"token:{target}", params=params)
except Exception:
pass
def _row_to_item(row: dict) -> TokenListItem:
return TokenListItem(
id=row["id"], name=row["name"], prefix=row["prefix"],
created_at=str(row.get("created_at") or ""),
expires_at=str(row["expires_at"]) if row.get("expires_at") else None,
last_used_at=str(row["last_used_at"]) if row.get("last_used_at") else None,
revoked_at=str(row["revoked_at"]) if row.get("revoked_at") else None,
)
def _row_to_admin_item(row: dict) -> AdminTokenItem:
return AdminTokenItem(
id=row["id"], name=row["name"], prefix=row["prefix"],
created_at=str(row.get("created_at") or ""),
expires_at=str(row["expires_at"]) if row.get("expires_at") else None,
last_used_at=str(row["last_used_at"]) if row.get("last_used_at") else None,
revoked_at=str(row["revoked_at"]) if row.get("revoked_at") else None,
user_id=row.get("user_id") or "",
user_email=row.get("user_email"),
last_used_ip=row.get("last_used_ip"),
)
@router.post("", response_model=CreateTokenResponse, status_code=201)
async def create_token(
payload: CreateTokenRequest,
user: dict = Depends(require_session_token),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
if not payload.name.strip():
raise HTTPException(status_code=400, detail="name is required")
if payload.expires_in_days is not None and payload.expires_in_days <= 0:
raise HTTPException(status_code=400, detail="expires_in_days must be a positive integer")
# Cap at 10 years — larger values overflow datetime.max during the
# `datetime.now(utc) + timedelta(days=...)` addition and surface as an
# unhandled OverflowError → 500. 10y is well past any legitimate PAT
# lifetime (the no-expiry path below uses ~100y and doesn't compute
# expires_at on the datetime object).
if payload.expires_in_days is not None and payload.expires_in_days > 3650:
raise HTTPException(status_code=400, detail="expires_in_days must not exceed 3650 (10 years)")
repo = AccessTokenRepository(conn)
token_id = str(uuid.uuid4())
expires_at: Optional[datetime] = None
expires_delta: Optional[timedelta] = None
omit_exp = payload.expires_in_days is None
if payload.expires_in_days is not None:
expires_delta = timedelta(days=payload.expires_in_days)
expires_at = datetime.now(timezone.utc) + expires_delta
# else: "no expiry" — DB stores expires_at=NULL and the JWT carries no
# `exp` claim. The authoritative expiry check lives in
# app/auth/dependencies.py (via the DB row).
# Build the JWT that embeds jti=token_id and typ=pat
jwt_token = create_access_token(
user_id=user["id"], email=user["email"], role=user["role"],
token_id=token_id, typ="pat",
expires_delta=expires_delta, omit_exp=omit_exp,
)
# Prefix: first 8 chars of the jti (UUID) — uniquely identifies the token in UI
# without exposing JWT headers (which all start with "eyJhbGci…" and are useless
# for identification). The JWT itself is returned ONCE in the response body.
prefix = token_id.replace("-", "")[:8]
# token_hash = sha256(raw JWT). Used in verify_token as defense-in-depth.
token_hash = hashlib.sha256(jwt_token.encode()).hexdigest()
repo.create(
id=token_id, user_id=user["id"], name=payload.name.strip(),
token_hash=token_hash, prefix=prefix, expires_at=expires_at,
)
_audit(conn, user["id"], "token.create", token_id, {"name": payload.name})
return CreateTokenResponse(
id=token_id, name=payload.name.strip(), prefix=prefix,
token=jwt_token, # returned EXACTLY ONCE; never retrievable again
expires_at=str(expires_at) if expires_at else None,
created_at=str(datetime.now(timezone.utc)),
)
@router.get("", response_model=List[TokenListItem])
async def list_tokens(
user: dict = Depends(get_current_user),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
# PATs may list their owner's own tokens — required by the documented
# `da auth token list` CLI flow (HEADLESS_USAGE.md). Only `create_token`
# is session-only (to block PAT-spawning-PAT chains).
rows = AccessTokenRepository(conn).list_for_user(user["id"])
return [_row_to_item(r) for r in rows]
@router.get("/{token_id}", response_model=TokenListItem)
async def get_token(
token_id: str,
user: dict = Depends(get_current_user),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
row = AccessTokenRepository(conn).get_by_id(token_id)
if not row or row["user_id"] != user["id"]:
raise HTTPException(status_code=404, detail="Token not found")
return _row_to_item(row)
@router.delete("/{token_id}", status_code=204)
async def revoke_token(
token_id: str,
user: dict = Depends(get_current_user),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
repo = AccessTokenRepository(conn)
row = repo.get_by_id(token_id)
if not row or row["user_id"] != user["id"]:
raise HTTPException(status_code=404, detail="Token not found")
repo.revoke(token_id)
_audit(conn, user["id"], "token.revoke", token_id)
# Admin — list & revoke tokens across users (for incident response)
@admin_router.get("", response_model=List[AdminTokenItem])
async def admin_list_tokens(
user: dict = Depends(require_role(Role.ADMIN)),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
return [_row_to_admin_item(r) for r in AccessTokenRepository(conn).list_all_with_user()]
@admin_router.delete("/{token_id}", status_code=204)
async def admin_revoke_token(
token_id: str,
user: dict = Depends(require_role(Role.ADMIN)),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
repo = AccessTokenRepository(conn)
row = repo.get_by_id(token_id)
if not row:
raise HTTPException(status_code=404, detail="Token not found")
repo.revoke(token_id)
_audit(conn, user["id"], "token.admin_revoke", token_id, {"owner_id": row["user_id"]})

216
app/api/users.py
View file

@ -1,19 +1,42 @@
"""User management endpoints."""
"""User management endpoints (#11)."""
import uuid
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from datetime import datetime, timezone
from typing import Optional, List
import duckdb
from fastapi import APIRouter, Depends, HTTPException, Request
from pydantic import BaseModel
from argon2 import PasswordHasher
from app.auth.dependencies import require_role, Role, _get_db
from src.repositories.users import UserRepository
from src.repositories.audit import AuditRepository
router = APIRouter(prefix="/api/users", tags=["users"])
def _audit(conn: duckdb.DuckDBPyConnection, actor_id: str, action: str, target_id: str, params: Optional[dict] = None) -> None:
try:
# Convert non-JSON-serializable values (datetime) to strings first
safe_params = None
if params:
safe_params = {}
for k, v in params.items():
if isinstance(v, datetime):
safe_params[k] = v.isoformat()
else:
safe_params[k] = v
AuditRepository(conn).log(
user_id=actor_id,
action=action,
resource=f"user:{target_id}",
params=safe_params,
)
except Exception:
pass # never block the endpoint on audit failure
class CreateUserRequest(BaseModel):
email: str
name: str
@ -23,6 +46,11 @@ class CreateUserRequest(BaseModel):
class UpdateUserRequest(BaseModel):
name: Optional[str] = None
role: Optional[str] = None
active: Optional[bool] = None
class SetPasswordRequest(BaseModel):
password: str
class UserResponse(BaseModel):
@ -30,7 +58,21 @@ class UserResponse(BaseModel):
email: str
name: Optional[str]
role: str
active: bool = True
created_at: Optional[str]
deactivated_at: Optional[str] = None
def _to_response(u: dict) -> UserResponse:
return UserResponse(
id=u["id"],
email=u["email"],
name=u.get("name"),
role=u["role"],
active=bool(u.get("active", True)),
created_at=str(u.get("created_at", "")),
deactivated_at=str(u["deactivated_at"]) if u.get("deactivated_at") else None,
)
@router.get("", response_model=List[UserResponse])
@ -38,37 +80,177 @@ async def list_users(
user: dict = Depends(require_role(Role.ADMIN)),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
repo = UserRepository(conn)
users = repo.list_all()
return [
UserResponse(
id=u["id"], email=u["email"], name=u.get("name"),
role=u["role"], created_at=str(u.get("created_at", "")),
) for u in users
]
return [_to_response(u) for u in UserRepository(conn).list_all()]
@router.post("", response_model=UserResponse, status_code=201)
async def create_user(
request: CreateUserRequest,
payload: CreateUserRequest,
request: Request,
user: dict = Depends(require_role(Role.ADMIN)),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
repo = UserRepository(conn)
if repo.get_by_email(request.email):
if repo.get_by_email(payload.email):
raise HTTPException(status_code=409, detail="User with this email already exists")
try:
Role(payload.role)
except ValueError:
raise HTTPException(status_code=400, detail=f"Unknown role: {payload.role}")
user_id = str(uuid.uuid4())
repo.create(id=user_id, email=request.email, name=request.name, role=request.role)
return UserResponse(id=user_id, email=request.email, name=request.name, role=request.role, created_at=None)
repo.create(id=user_id, email=payload.email, name=payload.name, role=payload.role)
_audit(conn, user["id"], "user.create", user_id, {"email": payload.email, "role": payload.role})
created = repo.get_by_id(user_id)
return _to_response(created)
@router.patch("/{user_id}", response_model=UserResponse)
async def update_user(
user_id: str,
payload: UpdateUserRequest,
request: Request,
user: dict = Depends(require_role(Role.ADMIN)),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
repo = UserRepository(conn)
target = repo.get_by_id(user_id)
if not target:
raise HTTPException(status_code=404, detail="User not found")
updates: dict = {}
if payload.name is not None:
updates["name"] = payload.name
if payload.role is not None:
# Validate role is a known value
try:
Role(payload.role)
except ValueError:
raise HTTPException(status_code=400, detail=f"Unknown role: {payload.role}")
# Protect: don't let admin demote themselves if they are the last admin
if (
target["id"] == user["id"]
and target["role"] == "admin"
and payload.role != "admin"
and repo.count_admins(active_only=True) <= 1
):
raise HTTPException(status_code=409, detail="Cannot demote the last active admin")
updates["role"] = payload.role
if payload.active is not None:
# Protect: cannot self-deactivate
if target["id"] == user["id"] and payload.active is False:
raise HTTPException(status_code=409, detail="Cannot deactivate yourself")
# Protect: cannot deactivate the last active admin
if (
target.get("role") == "admin"
and payload.active is False
and repo.count_admins(active_only=True) <= 1
):
raise HTTPException(status_code=409, detail="Cannot deactivate the last active admin")
updates["active"] = payload.active
if payload.active is False:
updates["deactivated_at"] = datetime.now(timezone.utc)
updates["deactivated_by"] = user["id"]
else:
updates["deactivated_at"] = None
updates["deactivated_by"] = None
if updates:
repo.update(id=user_id, **updates)
_audit(conn, user["id"], "user.update", user_id, {k: v for k, v in updates.items() if k != "deactivated_at"})
return _to_response(repo.get_by_id(user_id))
@router.delete("/{user_id}", status_code=204)
async def delete_user(
user_id: str,
request: Request,
user: dict = Depends(require_role(Role.ADMIN)),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
repo = UserRepository(conn)
if not repo.get_by_id(user_id):
target = repo.get_by_id(user_id)
if not target:
raise HTTPException(status_code=404, detail="User not found")
if target["id"] == user["id"]:
raise HTTPException(status_code=409, detail="Cannot delete yourself")
if target.get("role") == "admin" and repo.count_admins(active_only=True) <= 1:
raise HTTPException(status_code=409, detail="Cannot delete the last active admin")
repo.delete(user_id)
_audit(conn, user["id"], "user.delete", user_id, {"email": target["email"]})
@router.post("/{user_id}/reset-password")
async def reset_password(
user_id: str,
request: Request,
user: dict = Depends(require_role(Role.ADMIN)),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
"""Generate a reset token and (best-effort) email it to the user."""
import secrets
repo = UserRepository(conn)
target = repo.get_by_id(user_id)
if not target:
raise HTTPException(status_code=404, detail="User not found")
token = secrets.token_urlsafe(32)
repo.update(
id=user_id,
reset_token=token,
reset_token_created=datetime.now(timezone.utc),
)
_audit(conn, user["id"], "user.reset_password", user_id, {"email": target["email"]})
# Intentionally do NOT auto-send an email. The magic-link sender
# (`app/auth/providers/email.py:_send_email`) would deliver a "Login Link"
# that — when clicked — consumes the reset_token via verify_magic_link and
# logs the user in WITHOUT prompting for a new password, defeating the
# reset. Until a dedicated password-reset email flow with its own token
# column exists, admins share the `reset_token` below manually (or use the
# `set-password` endpoint directly).
return {"reset_token": token, "email_sent": False}
@router.post("/{user_id}/set-password", status_code=204)
async def set_password(
user_id: str,
payload: SetPasswordRequest,
request: Request,
user: dict = Depends(require_role(Role.ADMIN)),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
if not payload.password or len(payload.password) < 8:
raise HTTPException(status_code=400, detail="Password must be at least 8 characters")
repo = UserRepository(conn)
target = repo.get_by_id(user_id)
if not target:
raise HTTPException(status_code=404, detail="User not found")
ph = PasswordHasher()
repo.update(id=user_id, password_hash=ph.hash(payload.password))
_audit(conn, user["id"], "user.set_password", user_id, {"email": target["email"]})
@router.post("/{user_id}/deactivate", response_model=UserResponse)
async def deactivate_user(
user_id: str,
request: Request,
user: dict = Depends(require_role(Role.ADMIN)),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
return await update_user(
user_id=user_id,
payload=UpdateUserRequest(active=False),
request=request, user=user, conn=conn,
)
@router.post("/{user_id}/activate", response_model=UserResponse)
async def activate_user(
user_id: str,
request: Request,
user: dict = Depends(require_role(Role.ADMIN)),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
return await update_user(
user_id=user_id,
payload=UpdateUserRequest(active=True),
request=request, user=user, conn=conn,
)

24
app/auth/_common.py Normal file
View file

@ -0,0 +1,24 @@
"""Shared helpers for auth providers (Google OAuth, password, email link).
Kept out of `dependencies.py` so it doesn't pull FastAPI auth machinery into
thin provider modules that only need the sanitizer.
"""
from typing import Optional
def safe_next_path(candidate: Optional[str], default: str = "/dashboard") -> str:
"""Return `candidate` if it's a same-origin absolute path, else `default`.
Open-redirect guard: must start with a single `/` and must NOT start with
`//` (which browsers treat as protocol-relative, i.e. cross-origin).
Accepts plain paths like `/catalog` or `/foo?bar=baz`. Rejects
`javascript:...`, `http://...`, `//evil/`, bare `dashboard`, empty/None, etc.
"""
if not candidate or not isinstance(candidate, str):
return default
if not candidate.startswith("/"):
return default
if candidate.startswith("//"):
return default
return candidate

103
app/auth/dependencies.py
View file

@ -19,6 +19,26 @@ def _get_db():
conn.close()
def _client_ip(request: Optional[Request]) -> Optional[str]:
"""Return the request's client IP, preferring the first hop of X-Forwarded-For.
Trust model: this deployment runs behind Caddy (see repo Caddyfile), which
strips incoming X-Forwarded-For and sets its own. The leftmost hop is
therefore trustworthy. If the app is ever exposed directly to the internet
without a proxy, this value becomes client-settable and should only be
relied on for audit/diagnostics, never access control. Value is stored in
personal_access_tokens.last_used_ip and audit_log entries — informational
only, never authorization.
"""
if request is None:
return None
xff = request.headers.get("x-forwarded-for")
if xff:
return xff.split(",", 1)[0].strip() or None
client = getattr(request, "client", None)
return getattr(client, "host", None) if client else None
async def get_current_user(
request: Request = None,
authorization: Optional[str] = Header(None),
@ -54,6 +74,69 @@ async def get_current_user(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="User not found",
)
if not bool(user.get("active", True)):
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED,
detail="Account deactivated",
)
# PAT validation: check it's not revoked / expired / unknown in DB.
if payload.get("typ") == "pat":
from datetime import datetime, timezone
import hashlib
from src.repositories.access_tokens import AccessTokenRepository
def _fail(detail: str) -> None:
raise HTTPException(
status_code=status.HTTP_401_UNAUTHORIZED, detail=detail
)
tokens_repo = AccessTokenRepository(conn)
record = tokens_repo.get_by_id(payload.get("jti", ""))
if not record:
_fail("Token unknown")
if record.get("revoked_at") is not None:
_fail("Token revoked")
exp_at = record.get("expires_at")
if exp_at is not None:
if isinstance(exp_at, str):
exp_at = datetime.fromisoformat(exp_at)
if exp_at.tzinfo is None:
exp_at = exp_at.replace(tzinfo=timezone.utc)
if datetime.now(timezone.utc) > exp_at:
_fail("Token expired")
# Defense-in-depth: stored token_hash must match sha256(bearer JWT).
# Protects against a forged-but-unrevoked JWT using a stolen key.
stored_hash = record.get("token_hash")
if stored_hash:
actual = hashlib.sha256(token.encode()).hexdigest()
if actual != stored_hash:
_fail("Token mismatch")
# First-use-from-new-IP audit entry (#12 acceptance criterion).
# Only emit when the IP changes on a *subsequent* use — the very
# first use of a token is not surprising and doesn't need an entry.
current_ip = _client_ip(request)
previous_ip = record.get("last_used_ip")
already_used = record.get("last_used_at") is not None
if already_used and current_ip and current_ip != previous_ip:
try:
from src.repositories.audit import AuditRepository
AuditRepository(conn).log(
user_id=user["id"],
action="token.first_use_new_ip",
resource=f"token:{payload['jti']}",
params={"ip": current_ip, "previous_ip": previous_ip},
)
except Exception:
pass # audit failure must not block auth
# Record last_used_at / last_used_ip synchronously — acceptable cost; can batch later.
try:
tokens_repo.mark_used(payload["jti"], ip=current_ip)
except Exception:
pass
return user
@ -90,3 +173,23 @@ async def require_admin(user: dict = Depends(get_current_user)) -> dict:
detail="Admin access required",
)
return user
async def require_session_token(request: Request, user: dict = Depends(get_current_user)) -> dict:
"""Like get_current_user but rejects PAT — for endpoints that must not
be callable via a long-lived CI token (e.g. creating new tokens, changing password)."""
auth = request.headers.get("authorization", "")
token = None
if auth.startswith("Bearer "):
token = auth.removeprefix("Bearer ")
if not token and request:
token = request.cookies.get("access_token")
if token:
from app.auth.jwt import verify_token
payload = verify_token(token) or {}
if payload.get("typ") == "pat":
raise HTTPException(
status_code=status.HTTP_403_FORBIDDEN,
detail="This endpoint requires an interactive session, not a PAT",
)
return user

22
app/auth/jwt.py
View file

@ -48,18 +48,30 @@ def create_access_token(
email: str,
role: str = "analyst",
expires_delta: Optional[timedelta] = None,
token_id: Optional[str] = None,
typ: str = "session",
omit_exp: bool = False,
) -> str:
expire = datetime.now(timezone.utc) + (
expires_delta or timedelta(hours=ACCESS_TOKEN_EXPIRE_HOURS)
)
"""Create a JWT. `typ` is "session" (interactive login) or "pat" (long-lived).
If `omit_exp=True`, no `exp` claim is embedded. This is used by PATs with
"no expiry" — the authoritative expiry check is the DB row in
`personal_access_tokens.expires_at`, and a claim-less JWT avoids the
misleading ~100y horizon that previously pretended to be "never".
"""
payload = {
"sub": user_id,
"email": email,
"role": role,
"exp": expire,
"typ": typ,
"iat": datetime.now(timezone.utc),
"jti": uuid.uuid4().hex,
"jti": token_id or uuid.uuid4().hex,
}
if not omit_exp:
expire = datetime.now(timezone.utc) + (
expires_delta or timedelta(hours=ACCESS_TOKEN_EXPIRE_HOURS)
)
payload["exp"] = expire
return jwt.encode(payload, _get_cached_secret_key(), algorithm=ALGORITHM)

27
app/auth/providers/google.py
View file

@ -9,6 +9,7 @@ from fastapi.responses import RedirectResponse
from starlette.config import Config as StarletteConfig
from app.auth.jwt import create_access_token
from app.auth._common import safe_next_path
from app.instance_config import get_allowed_domains
logger = logging.getLogger(__name__)
@ -42,9 +43,21 @@ _setup_oauth()
@router.get("/login")
async def google_login(request: Request):
"""Redirect to Google OAuth."""
"""Redirect to Google OAuth.
Honors `?next=<path>` by stashing the sanitized value in the session so the
callback can redirect there instead of the default /dashboard. The session
is the right stash — OAuth flow is stateful and the `state` param is
managed by Authlib.
"""
if not is_available():
return RedirectResponse(url="/login?error=google_not_configured")
next_path = safe_next_path(request.query_params.get("next"), default="")
if next_path:
request.session["login_next"] = next_path
else:
# Clear any stale value from an earlier aborted attempt.
request.session.pop("login_next", None)
redirect_uri = str(request.url_for("google_callback"))
return await oauth.google.authorize_redirect(request, redirect_uri)
@ -84,15 +97,23 @@ async def google_callback(request: Request):
user_id = str(uuid.uuid4())
repo.create(id=user_id, email=email, name=name, role="analyst")
user = repo.get_by_email(email)
if not bool(user.get("active", True)):
return RedirectResponse(url="/login?error=deactivated")
finally:
conn.close()
# Issue JWT
jwt_token = create_access_token(user["id"], user["email"], user["role"])
# Redirect to dashboard with token in cookie
# Redirect to the post-login target. Prefer the value stashed by
# google_login() — re-sanitize defensively in case of session tampering.
target = safe_next_path(
request.session.pop("login_next", None), default="/dashboard"
)
# Redirect to target with token in cookie
is_production = os.environ.get("TESTING", "").lower() not in ("1", "true")
response = RedirectResponse(url="/dashboard", status_code=302)
response = RedirectResponse(url=target, status_code=302)
response.set_cookie(
key="access_token", value=jwt_token,
httponly=True, max_age=86400, samesite="lax",

20
app/auth/providers/password.py
View file

@ -43,6 +43,8 @@ async def password_login(
user = repo.get_by_email(request.email)
if not user or not user.get("password_hash"):
raise HTTPException(status_code=401, detail="Invalid email or password")
if not bool(user.get("active", True)):
raise HTTPException(status_code=401, detail="Account deactivated")
# Verify password
try:
@ -62,25 +64,37 @@ async def password_login(
async def password_login_web(
email: str = Form(...),
password: str = Form(""),
next: str = Form(""),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
"""Web form login — sets cookie and redirects to dashboard."""
"""Web form login — sets cookie and redirects to `next` (or /dashboard)."""
repo = UserRepository(conn)
user = repo.get_by_email(email)
if not user or not user.get("password_hash"):
return RedirectResponse(url="/login/password?error=invalid", status_code=302)
if not bool(user.get("active", True)):
return RedirectResponse(url="/login/password?error=deactivated", status_code=302)
try:
ph = PasswordHasher()
ph.verify(user["password_hash"], password)
except (VerifyMismatchError, Exception):
except VerifyMismatchError:
# Genuinely wrong password → usual UX.
return RedirectResponse(url="/login/password?error=invalid", status_code=302)
except Exception:
# Corrupted hash / library error → surface a distinct error code so ops
# can tell broken-hash cases apart from bad-password cases. Log loudly.
logger.exception("Unexpected error during web password verification for %s", email)
return RedirectResponse(url="/login/password?err=auth_internal", status_code=302)
token = create_access_token(user["id"], user["email"], user["role"])
# Secure cookie only over HTTPS (detect via X-Forwarded-Proto or request scheme)
# For dev/staging on plain HTTP, secure=False so the cookie is actually sent
use_secure = os.environ.get("DOMAIN", "") != "" # DOMAIN set = production with TLS
response = RedirectResponse(url="/dashboard", status_code=302)
# Sanitize `next`: must start with `/` and must not start with `//` (open-redirect guard)
target = next if (next.startswith("/") and not next.startswith("//")) else "/dashboard"
response = RedirectResponse(url=target, status_code=302)
response.set_cookie(
key="access_token", value=token,
httponly=True, max_age=86400, samesite="lax",

3
app/auth/router.py
View file

@ -65,6 +65,9 @@ async def create_token(
user = repo.get_by_email(request.email)
if not user:
raise HTTPException(status_code=401, detail="User not found")
if not bool(user.get("active", True)):
_audit(user["id"], "login_failed", result="deactivated")
raise HTTPException(status_code=401, detail="Account deactivated")
# If user has password_hash, require and verify it
if user.get("password_hash"):

28
app/main.py
View file

@ -3,12 +3,15 @@
import logging
from contextlib import asynccontextmanager
from pathlib import Path
from urllib.parse import quote
import os
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
from fastapi.responses import RedirectResponse
from fastapi.staticfiles import StaticFiles
from starlette.exceptions import HTTPException as StarletteHTTPException
from starlette.middleware.sessions import SessionMiddleware
from app.auth.router import router as auth_router
@ -30,6 +33,8 @@ from app.api.jira_webhooks import router as jira_webhooks_router
from app.api.metrics import router as metrics_router
from app.api.metadata import router as metadata_router
from app.api.query_hybrid import router as query_hybrid_router
from app.api.cli_artifacts import router as cli_artifacts_router
from app.api.tokens import router as tokens_router, admin_router as tokens_admin_router
from app.web.router import router as web_router
logger = logging.getLogger(__name__)
@ -157,10 +162,33 @@ def create_app() -> FastAPI:
app.include_router(metrics_router)
app.include_router(metadata_router)
app.include_router(query_hybrid_router)
app.include_router(cli_artifacts_router)
app.include_router(tokens_router)
app.include_router(tokens_admin_router)
# Web UI router (must be last — has catch-all routes)
app.include_router(web_router)
@app.exception_handler(StarletteHTTPException)
async def _html_auth_redirect_handler(request, exc: StarletteHTTPException):
"""Redirect unauthenticated HTML page loads (GET) to /login.
Only GET requests outside `/api/` and `/auth/` are redirected — that
targets browser navigations to HTML pages. POSTs, API prefixes, and
non-401 errors fall through to Starlette's default JSON response so
JSON clients (including `/auth/tokens` for PAT CRUD) keep their
existing contract.
"""
if (
exc.status_code == 401
and request.method == "GET"
and not request.url.path.startswith(("/api/", "/auth/"))
):
next_param = quote(request.url.path, safe="")
return RedirectResponse(url=f"/login?next={next_param}", status_code=302)
from fastapi.exception_handlers import http_exception_handler
return await http_exception_handler(request, exc)
return app

102
app/web/router.py
View file

@ -8,6 +8,7 @@ import os
from datetime import datetime
from pathlib import Path
from typing import Optional
from urllib.parse import quote
from fastapi import APIRouter, Depends, Request, HTTPException
from fastapi.responses import HTMLResponse, RedirectResponse
@ -152,6 +153,11 @@ def _build_context(request: Request, user: Optional[dict] = None, **extra) -> di
return {k: v for k, v in theme.items() if v}
return {}
# Lines + server_url for the "Setup a new Claude Code" preview/clipboard
# partial; single source of truth lives in app/web/setup_instructions.py.
from app.web.setup_instructions import SETUP_INSTRUCTIONS_LINES
ctx_server_url = str(request.base_url).rstrip("/")
ctx = {
"request": request,
"config": ConfigProxy,
@ -162,8 +168,11 @@ def _build_context(request: Request, user: Optional[dict] = None, **extra) -> di
"get_flashed_messages": lambda **kwargs: [],
"url_for": lambda endpoint, **kw: _url_for_shim(endpoint, **kw),
"session": _FlexDict({"user": user}) if user else _FlexDict(),
"setup_instructions_lines": SETUP_INSTRUCTIONS_LINES,
"server_url": ctx_server_url,
}
# Flex all extra context values for template compatibility
# (but skip ones we just populated — extras with the same key win)
for k, v in extra.items():
ctx[k] = _flex(v) if isinstance(v, (dict, list)) else v
return ctx
@ -192,6 +201,10 @@ async def setup_wizard(request: Request, conn: duckdb.DuckDBPyConnection = Depen
@router.get("/login", response_class=HTMLResponse)
async def login_page(request: Request):
next_path = request.query_params.get("next", "")
if not next_path.startswith("/") or next_path.startswith("//"):
next_path = ""
providers = []
try:
from app.auth.providers.google import is_available as google_available
@ -211,39 +224,54 @@ async def login_page(request: Request):
login_buttons = []
for p in providers:
if p["name"] == "google":
login_buttons.append({"url": "/auth/google/login", "text": "Sign in with Google", "css_class": "btn-primary", "icon_html": ""})
_url = "/auth/google/login"
if next_path:
_url += f"?next={quote(next_path, safe='')}"
login_buttons.append({"url": _url, "text": "Sign in with Google", "css_class": "btn-primary", "icon_html": ""})
elif p["name"] == "password":
login_buttons.append({"url": "/login/password", "text": "Sign in with Email & Password", "css_class": "btn-secondary", "icon_html": ""})
_url = "/login/password"
if next_path:
_url += f"?next={quote(next_path, safe='')}"
login_buttons.append({"url": _url, "text": "Sign in with Email & Password", "css_class": "btn-secondary", "icon_html": ""})
elif p["name"] == "email":
login_buttons.append({"url": "/login/email", "text": "Sign in with Email Link", "css_class": "btn-secondary", "icon_html": ""})
_url = "/login/email"
if next_path:
_url += f"?next={quote(next_path, safe='')}"
login_buttons.append({"url": _url, "text": "Sign in with Email Link", "css_class": "btn-secondary", "icon_html": ""})
ctx = _build_context(request, providers=providers, login_buttons=login_buttons)
ctx = _build_context(request, providers=providers, login_buttons=login_buttons, next_path=next_path)
return templates.TemplateResponse(request, "login.html", ctx)
@router.get("/login/password", response_class=HTMLResponse)
async def login_password_page(request: Request):
"""Password login form (email + password)."""
next_path = request.query_params.get("next", "")
if not next_path.startswith("/") or next_path.startswith("//"):
next_path = ""
google_ok = False
try:
from app.auth.providers.google import is_available as google_available
google_ok = google_available()
except Exception:
pass
ctx = _build_context(request, google_available=google_ok)
ctx = _build_context(request, google_available=google_ok, next_path=next_path)
return templates.TemplateResponse(request, "login_email.html", ctx)
@router.get("/login/email", response_class=HTMLResponse)
async def login_email_page(request: Request):
"""Email magic link login form."""
next_path = request.query_params.get("next", "")
if not next_path.startswith("/") or next_path.startswith("//"):
next_path = ""
google_ok = False
try:
from app.auth.providers.google import is_available as google_available
google_ok = google_available()
except Exception:
pass
ctx = _build_context(request, google_available=google_ok)
ctx = _build_context(request, google_available=google_ok, next_path=next_path)
return templates.TemplateResponse(request, "login_email.html", ctx)
@ -288,7 +316,6 @@ async def dashboard(
account_status="active",
account_details=None,
telegram_status={"linked": False},
setup_instructions="Use 'da login' to connect your CLI tool.",
data_stats={
"tables": total_tables,
"total_tables": total_tables,
@ -495,6 +522,22 @@ async def activity_center(
return templates.TemplateResponse(request, "activity_center.html", ctx)
@router.get("/install", response_class=HTMLResponse)
async def install_page(
request: Request,
user: Optional[dict] = Depends(get_optional_user),
):
"""Public install instructions for the CLI."""
base_url = str(request.base_url).rstrip("/")
ctx = _build_context(
request,
user=user,
server_url=base_url,
agnes_version=os.environ.get("AGNES_VERSION", "dev"),
)
return templates.TemplateResponse(request, "install.html", ctx)
@router.get("/admin/tables", response_class=HTMLResponse)
async def admin_tables(
request: Request,
@ -517,3 +560,48 @@ async def admin_permissions_page(
"""Admin page for managing permissions and access requests."""
ctx = _build_context(request, user=user)
return templates.TemplateResponse(request, "admin_permissions.html", ctx)
@router.get("/admin/users", response_class=HTMLResponse)
async def admin_users_page(
request: Request,
user: dict = Depends(require_role(Role.ADMIN)),
):
"""Admin page for user management."""
ctx = _build_context(request, user=user)
return templates.TemplateResponse(request, "admin_users.html", ctx)
@router.get("/tokens", response_class=HTMLResponse)
async def my_tokens_page(
request: Request,
user: dict = Depends(get_current_user),
):
"""My tokens — ANY signed-in user (incl. admins' own).
Always shows the user's own PATs. Create + reveal + revoke-own flow.
Admins who need the org-wide view go to /admin/tokens.
"""
ctx = _build_context(request, user=user)
return templates.TemplateResponse(request, "my_tokens.html", ctx)
@router.get("/admin/tokens", response_class=HTMLResponse)
async def admin_tokens_page(
request: Request,
user: dict = Depends(require_role(Role.ADMIN)),
):
"""Admin — list of ALL tokens for incident response + offboarding.
Admin-only. No create form here (admins mint their own PATs via /tokens).
URL param ?user=<email> pre-fills the owner filter (deep-link from
/admin/users "Tokens" action).
"""
ctx = _build_context(request, user=user)
return templates.TemplateResponse(request, "admin_tokens.html", ctx)
@router.get("/profile")
async def profile_redirect(request: Request):
"""Back-compat: /profile (PAT CRUD) has been unified under /tokens."""
return RedirectResponse(url="/tokens", status_code=302)

79
app/web/setup_instructions.py Normal file
View file

@ -0,0 +1,79 @@
"""Single source of truth for the "Setup a new Claude Code" clipboard payload.
Both the JS-embedded clipboard renderer (`_claude_setup_instructions.jinja`)
and the read-only HTML preview on the dashboard and /install pages consume
these lines. Keep it in Python so there is exactly ONE place that edits.
Placeholders `{server_url}` and `{token}` are substituted at render time.
For the preview we substitute `{token}` with a user-visible placeholder
string styled distinctly in the HTML preview.
"""
from __future__ import annotations
SETUP_INSTRUCTIONS_LINES: list[str] = [
"Set up the Agnes CLI on this machine.",
"",
"Server: {server_url}",
"Personal access token: {token}",
"(Just generated; treat it as a secret.)",
"",
"Run these, in order. If any step fails, paste the exact error back and stop.",
"",
"1) Install the CLI:",
" uv tool install --force {server_url}/cli/agnes.whl",
"",
" If uv is not installed yet:",
" curl -LsSf https://astral.sh/uv/install.sh | sh",
"",
" If `da --version` fails after install because ~/.local/bin is not on PATH:",
" export PATH=\"$HOME/.local/bin:$PATH\"",
" # persist: append the same line to your ~/.zshrc or ~/.bashrc",
"",
"2) Log in (also saves the server URL):",
" da auth import-token --token \"{token}\" --server \"{server_url}\"",
"",
"3) Verify the login:",
" da auth whoami",
"",
"4) Run diagnostics:",
" da diagnose",
"",
" This should print \"Overall: healthy\" and a list of green checks. If",
" anything is yellow/red, paste the full output back.",
"",
"5) Skills (ask the user first):",
" The CLI ships with reusable markdown skills (setup, connectors,",
" corporate-memory, deploy, notifications, security, troubleshoot),",
" listable via `da skills list` and readable via `da skills show <name>`.",
"",
" Ask the user verbatim: \"Do you want me to copy the Agnes skills into",
" ~/.claude/skills/agnes/ so they are always loaded in Claude Code,",
" or should I pull them on-demand via `da skills show <name>` when",
" needed?\"",
"",
" If they say copy:",
" mkdir -p ~/.claude/skills/agnes",
" for s in $(da skills list | awk '{print $1}'); do",
" da skills show \"$s\" > ~/.claude/skills/agnes/\"$s\".md",
" done",
" echo \"Copied skills to ~/.claude/skills/agnes/\"",
"",
"6) Confirm:",
" Tell me \"Agnes CLI is ready\" and summarize:",
" - `da --version` output",
" - `da auth whoami` output (email + role)",
" - Whether skills were copied or left on-demand",
" - The `da diagnose` overall status",
]
def render_setup_instructions(server_url: str, token: str) -> str:
"""Render the setup instructions as a single string.
Used server-side for tests and any non-JS rendering path. The browser
clipboard flow uses the JS renderer embedded in the Jinja partial; both
must produce byte-identical output for a given (server_url, token).
"""
text = "\n".join(SETUP_INSTRUCTIONS_LINES)
return text.replace("{server_url}", server_url).replace("{token}", token)

161
app/web/static/style-custom.css
View file

@ -2040,3 +2040,164 @@ a.slack-badge:hover {
background: #d97706;
transform: translateY(-1px);
}
/* ─── Shared modern header (used by base.html + future pages) ─── */
/* Mirrors the inline header styles in dashboard.html so all pages share chrome. */
.app-header {
background: var(--surface, #fff);
border-bottom: 1px solid var(--border, #e5e7eb);
padding: 0 32px;
height: 72px;
display: flex;
align-items: center;
justify-content: space-between;
position: sticky;
top: 0;
z-index: 100;
}
.app-header-left {
display: flex;
flex-direction: column;
justify-content: center;
gap: 2px;
}
.app-header-logo {
display: inline-flex;
align-items: center;
text-decoration: none;
color: inherit;
font-weight: 600;
font-size: 16px;
}
.app-header-logo svg { display: block; }
a.app-header-logo:focus-visible {
outline: 2px solid var(--primary, #6366f1);
outline-offset: 2px;
border-radius: 4px;
}
.app-header-subtitle {
font-size: 11px;
font-weight: 500;
color: var(--text-secondary, #6b7280);
letter-spacing: 0.4px;
text-transform: uppercase;
margin-top: 2px;
}
.app-header-right {
display: flex;
align-items: center;
gap: 16px;
}
.app-header-email {
font-size: 13px;
color: var(--text-secondary, #6b7280);
font-weight: 500;
}
.app-avatar {
width: 36px;
height: 36px;
border-radius: 50%;
background: var(--primary-light, #eef2ff);
color: var(--primary, #6366f1);
font-size: 13px;
font-weight: 600;
display: flex;
align-items: center;
justify-content: center;
letter-spacing: 0.3px;
}
.app-avatar-img {
width: 36px;
height: 36px;
border-radius: 50%;
border: 2px solid var(--border, #e5e7eb);
}
.app-nav-link {
font-size: 13px;
font-weight: 500;
color: var(--text-secondary, #6b7280);
text-decoration: none;
padding: 6px 12px;
border-radius: 8px;
transition: all 0.15s ease;
}
.app-nav-link:hover {
color: var(--text-primary, #111827);
background: var(--border-light, #f3f4f6);
}
.app-nav-link.is-active {
color: var(--primary, #6366f1);
background: var(--primary-light, #eef2ff);
}
.app-btn-logout {
font-size: 13px;
font-weight: 500;
color: var(--text-secondary, #6b7280);
background: none;
border: 1px solid var(--border, #e5e7eb);
border-radius: 8px;
padding: 6px 14px;
cursor: pointer;
transition: all 0.15s ease;
text-decoration: none;
display: inline-block;
}
.app-btn-logout:hover {
color: var(--text-primary, #111827);
border-color: #d1d5db;
background: var(--border-light, #f3f4f6);
}
/* ── User menu (dropdown) ── */
.app-user-menu { position: relative; display: inline-flex; align-items: center; }
.app-user-menu-trigger {
display: inline-flex; align-items: center; gap: 6px;
background: none; border: 1px solid transparent;
border-radius: 999px; padding: 4px 10px 4px 4px;
cursor: pointer; transition: all 0.15s ease;
}
.app-user-menu-trigger:hover { background: var(--border-light, #f3f4f6); border-color: var(--border, #e5e7eb); }
.app-user-menu-trigger[aria-expanded="true"] { background: var(--border-light, #f3f4f6); border-color: var(--border, #e5e7eb); }
.app-user-menu-chevron { color: var(--text-secondary, #6b7280); transition: transform 0.15s ease; }
.app-user-menu-trigger[aria-expanded="true"] .app-user-menu-chevron { transform: rotate(180deg); }
.app-user-menu-panel {
position: absolute; top: calc(100% + 8px); right: 0;
min-width: 220px;
background: var(--surface, #fff);
border: 1px solid var(--border, #e5e7eb);
border-radius: 10px;
box-shadow: 0 8px 24px rgba(0, 0, 0, 0.08);
padding: 6px;
z-index: 50;
}
.app-user-menu-panel[hidden] { display: none; }
.app-user-menu-header {
padding: 10px 12px 8px;
border-bottom: 1px solid var(--border-light, #f3f4f6);
margin-bottom: 4px;
}
.app-user-menu-email { font-size: 13px; font-weight: 500; color: var(--text-primary, #111827); word-break: break-all; }
.app-user-menu-role { font-size: 11px; color: var(--text-secondary, #6b7280); margin-top: 2px; text-transform: uppercase; letter-spacing: 0.3px; }
.app-user-menu-item {
display: block; padding: 8px 12px;
font-size: 13px; color: var(--text-primary, #111827);
text-decoration: none; border-radius: 6px;
}
.app-user-menu-item:hover { background: var(--border-light, #f3f4f6); }
.app-user-menu-item.is-active { background: rgba(0, 115, 209, 0.08); color: var(--primary, #0073D1); font-weight: 500; }
@media (max-width: 720px) {
.app-header { padding: 0 16px; gap: 8px; }
.app-header-email { display: none; }
.app-nav-link { padding: 6px 8px; }
}

3
app/web/static/style.css
View file

@ -54,6 +54,9 @@ header {
gap: 16px;
}
a.logo { text-decoration: none; color: inherit; display: block; }
a.logo:hover h1 { color: var(--primary); }
.logo h1 {
font-size: 1.375rem;
font-weight: 600;

67
app/web/templates/_app_header.html Normal file
View file

@ -0,0 +1,67 @@
{# Shared modern header — used by base.html and dashboard.html.
Styles live in app/web/static/style-custom.css under the .app-* prefix. #}
{% if session.user %}
<header class="app-header">
<div class="app-header-left">
<a class="app-header-logo" href="/" aria-label="Home">
{% if config.LOGO_SVG %}{{ config.LOGO_SVG | safe }}{% else %}{{ config.INSTANCE_NAME or 'Data Analyst Portal' }}{% endif %}
</a>
<span class="app-header-subtitle">{{ config.INSTANCE_SUBTITLE or 'Data Analyst Portal' }}</span>
</div>
<div class="app-header-right">
{% set _path = request.url.path %}
<a class="app-nav-link {% if _path == '/dashboard' or _path == '/' %}is-active{% endif %}" href="/dashboard">Dashboard</a>
<a class="app-nav-link {% if _path.startswith('/install') %}is-active{% endif %}" href="/install">Install CLI</a>
{% if session.user.role == 'admin' %}
<a class="app-nav-link {% if _path.startswith('/admin/users') %}is-active{% endif %}" href="/admin/users">Users</a>
<a class="app-nav-link {% if _path.startswith('/admin/tokens') %}is-active{% endif %}" href="/admin/tokens">All tokens</a>
{% endif %}
<div class="app-user-menu" id="userMenu">
<button type="button" class="app-user-menu-trigger" id="userMenuTrigger"
aria-haspopup="menu" aria-expanded="false" aria-controls="userMenuPanel">
{% if session.user.picture %}
<img src="{{ session.user.picture }}" alt="" class="app-avatar-img">
{% else %}
<span class="app-avatar">{{ (session.user.name or session.user.email)[:2] | upper }}</span>
{% endif %}
<svg class="app-user-menu-chevron" width="12" height="12" viewBox="0 0 12 12" aria-hidden="true">
<path d="M2 4l4 4 4-4" fill="none" stroke="currentColor" stroke-width="1.5" stroke-linecap="round" stroke-linejoin="round"/>
</svg>
</button>
<div class="app-user-menu-panel" id="userMenuPanel" role="menu" hidden>
<div class="app-user-menu-header">
<div class="app-user-menu-email">{{ session.user.email }}</div>
{% if session.user.role %}
<div class="app-user-menu-role">{{ session.user.role | capitalize }}</div>
{% endif %}
</div>
<a class="app-user-menu-item {% if _path == '/tokens' or _path.startswith('/profile') %}is-active{% endif %}" role="menuitem" href="/tokens">My tokens</a>
<a class="app-user-menu-item" role="menuitem" href="{{ url_for('auth.logout') }}">Logout</a>
</div>
</div>
</div>
</header>
<script>
(function() {
var trigger = document.getElementById('userMenuTrigger');
var panel = document.getElementById('userMenuPanel');
if (!trigger || !panel) return;
function setOpen(open) {
trigger.setAttribute('aria-expanded', open ? 'true' : 'false');
if (open) { panel.removeAttribute('hidden'); }
else { panel.setAttribute('hidden', ''); }
}
trigger.addEventListener('click', function(e) {
e.stopPropagation();
setOpen(trigger.getAttribute('aria-expanded') !== 'true');
});
document.addEventListener('click', function(e) {
if (!panel.contains(e.target) && e.target !== trigger) setOpen(false);
});
document.addEventListener('keydown', function(e) {
if (e.key === 'Escape') { setOpen(false); trigger.focus(); }
});
})();
</script>
{% endif %}

44
app/web/templates/_claude_setup_instructions.jinja Normal file
View file

@ -0,0 +1,44 @@
{# Single source of truth for the "Setup a new Claude Code" clipboard payload.
Two modes:
* preview_mode=True → emits a read-only HTML <pre><code> block rendered
with the real server_url and a visible placeholder
for the token. Used inline on /dashboard and
/install so the reader can see exactly what will
land in their clipboard.
* preview_mode=False → emits the JS `SETUP_INSTRUCTIONS_TEMPLATE` array +
`renderSetupInstructions(server, token)` function.
Placed inside a <script> block; at click time the
server-issued token is substituted for "{token}".
The lines themselves come from `app/web/setup_instructions.py` via the
context variable `setup_instructions_lines`, so both modes stay in lockstep.
The .jinja extension is NOT in Jinja2's default autoescape list, so we
explicitly run every piece of user-visible text through the `e` filter
in preview mode (lines contain literal `<` and `>` e.g. `<name>`).
#}
{% if preview_mode %}
<pre class="setup-preview-pre"><code class="setup-preview-code">{% for line in setup_instructions_lines -%}
{% set rendered = line.replace("{server_url}", server_url) -%}
{% if "{token}" in rendered -%}
{% set parts = rendered.split("{token}") -%}
{{ parts[0] | e }}<span class="placeholder-token" aria-label="placeholder — real token is generated when you click the button">&lt;will be generated on click&gt;</span>{{ parts[1] | e }}
{% else -%}
{{ rendered | e }}
{% endif -%}
{% endfor %}</code></pre>
{% else %}
var SETUP_INSTRUCTIONS_TEMPLATE = [
{%- for line in setup_instructions_lines %}
{{ line | tojson }}{% if not loop.last %},{% endif %}
{%- endfor %}
].join("\n");
function renderSetupInstructions(serverUrl, token) {
return SETUP_INSTRUCTIONS_TEMPLATE
.split("{server_url}").join(serverUrl)
.split("{token}").join(token);
}
{% endif %}

22
app/web/templates/activity_center.html
View file

@ -1745,27 +1745,7 @@
</head>
<body>
<!-- Top Header Bar (matches Data Catalog pattern) -->
<header class="top-header">
<div class="top-header-left">
<a href="{{ url_for('dashboard') }}" class="header-back" title="Back to Dashboard">
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path d="M19 12H5M12 19l-7-7 7-7"/>
</svg>
</a>
<div class="header-logo-group">
<div class="header-logo">
{{ config.LOGO_SVG | safe }}
</div>
<span class="header-subtitle">Activity Center <span class="demo-badge">DEMO</span></span>
</div>
</div>
<div class="top-header-right">
{% if session.user.picture %}
<img src="{{ session.user.picture }}" alt="Profile" class="avatar-v2">
{% endif %}
{{ session.user.email }}
</div>
</header>
{% include '_app_header.html' %}
<div class="container-activity">
<!-- Executive Pulse - always visible -->

20
app/web/templates/admin_permissions.html
View file

@ -715,25 +715,7 @@
<body>
<!-- HEADER -->
<header class="header">
<div class="header-left">
<a href="{{ url_for('dashboard') }}" class="header-back" title="Back to Dashboard">
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path d="M19 12H5M12 19l-7-7 7-7"/>
</svg>
</a>
<div class="header-logo-group">
<div class="header-logo">
{{ config.LOGO_SVG | safe }}
</div>
<span class="header-subtitle">Permissions Management</span>
</div>
</div>
<div class="header-right">
<a href="/admin/tables" class="header-nav-link">Table Management</a>
<span>Admin</span>
</div>
</header>
{% include '_app_header.html' %}
<!-- PAGE TITLE -->
<div class="page-title">

20
app/web/templates/admin_tables.html
View file

@ -735,25 +735,7 @@
<body>
<!-- ═══════════════ HEADER ═══════════════ -->
<header class="header">
<div class="header-left">
<a href="{{ url_for('dashboard') }}" class="header-back" title="Back to Dashboard">
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path d="M19 12H5M12 19l-7-7 7-7"/>
</svg>
</a>
<div class="header-logo-group">
<div class="header-logo">
{{ config.LOGO_SVG | safe }}
</div>
<span class="header-subtitle">Table Management</span>
</div>
</div>
<div class="header-right">
<a href="/admin/permissions" style="font-size: 12px; font-weight: 500; color: var(--primary); text-decoration: none; padding: 6px 12px; border-radius: 6px; transition: all 0.15s ease;">Permissions</a>
<span>Admin</span>
</div>
</header>
{% include '_app_header.html' %}
<!-- ═══════════════ PAGE TITLE ═══════════════ -->
<div class="page-title">

1202
app/web/templates/admin_tokens.html Normal file
View file

File diff suppressed because it is too large Load diff

553
app/web/templates/admin_users.html Normal file
View file

@ -0,0 +1,553 @@
{% extends "base.html" %}
{% block title %}Users — {{ config.INSTANCE_NAME }}{% endblock %}
{% block content %}
<style>
.users-page { max-width: 1200px; margin: 24px auto; padding: 0 16px; }
.users-toolbar {
display: flex; justify-content: space-between; align-items: center;
gap: 16px; margin-bottom: 20px; flex-wrap: wrap;
}
.users-title { margin: 0; font-size: 22px; font-weight: 600; }
.users-search {
flex: 1; max-width: 360px;
padding: 8px 12px 8px 36px;
border: 1px solid var(--border, #e5e7eb);
border-radius: 8px;
font-size: 13px;
background: var(--surface, #fff) url("data:image/svg+xml;utf8,<svg xmlns='http://www.w3.org/2000/svg' width='14' height='14' viewBox='0 0 24 24' fill='none' stroke='%236b7280' stroke-width='2'><circle cx='11' cy='11' r='8'/><path d='m21 21-4.35-4.35'/></svg>") no-repeat 12px center;
}
.users-search:focus { outline: 2px solid var(--primary, #6366f1); outline-offset: -1px; }
.users-table-wrap {
background: var(--surface, #fff);
border: 1px solid var(--border, #e5e7eb);
border-radius: 12px;
overflow: hidden;
}
.users-table { width: 100%; border-collapse: collapse; font-size: 13px; }
.users-table thead th {
text-align: left; padding: 12px 16px;
background: var(--border-light, #f9fafb);
border-bottom: 1px solid var(--border, #e5e7eb);
font-weight: 600; color: var(--text-secondary, #6b7280);
font-size: 11px; text-transform: uppercase; letter-spacing: 0.4px;
}
.users-table tbody td {
padding: 12px 16px;
border-bottom: 1px solid var(--border-light, #f3f4f6);
vertical-align: middle;
}
.users-table tbody tr:last-child td { border-bottom: none; }
.users-table tbody tr.is-deactivated { opacity: 0.55; }
.users-table tbody tr:hover { background: var(--border-light, #fafafa); }
.user-cell { display: flex; align-items: center; gap: 10px; }
.user-avatar {
width: 32px; height: 32px; border-radius: 50%;
display: flex; align-items: center; justify-content: center;
font-size: 12px; font-weight: 600; color: #fff;
flex-shrink: 0;
}
.user-meta { display: flex; flex-direction: column; gap: 2px; min-width: 0; }
.user-meta .name { font-weight: 500; color: var(--text-primary, #111827); }
.user-meta .email { font-size: 11px; color: var(--text-secondary, #6b7280); }
.user-cell.no-name .name { display: none; }
.user-cell.no-name .email { font-size: 13px; color: var(--text-primary, #111827); font-weight: 500; }
.role-pill {
display: inline-block;
padding: 3px 10px; border-radius: 999px;
font-size: 11px; font-weight: 600;
text-transform: uppercase; letter-spacing: 0.4px;
cursor: pointer; border: 1px solid transparent;
}
.role-pill.role-admin { background: #fee2e2; color: #b91c1c; }
.role-pill.role-analyst { background: #dbeafe; color: #1e40af; }
.role-pill.role-km_admin { background: #ede9fe; color: #6d28d9; }
.role-pill.role-viewer { background: #f3f4f6; color: #4b5563; }
/* Toggle switch */
.toggle { position: relative; display: inline-block; width: 36px; height: 20px; }
.toggle input { opacity: 0; width: 0; height: 0; }
.toggle-slider {
position: absolute; cursor: pointer; inset: 0;
background: #cbd5e1; border-radius: 999px; transition: 0.2s;
}
.toggle-slider::before {
content: ""; position: absolute; left: 2px; top: 2px;
width: 16px; height: 16px; background: #fff; border-radius: 50%;
transition: 0.2s;
}
.toggle input:checked + .toggle-slider { background: #10b981; }
.toggle input:checked + .toggle-slider::before { transform: translateX(16px); }
.toggle input:focus-visible + .toggle-slider { outline: 2px solid var(--primary, #6366f1); outline-offset: 2px; }
.date-cell { color: var(--text-secondary, #6b7280); font-size: 12px; white-space: nowrap; }
.row-actions { display: flex; gap: 6px; justify-content: flex-end; }
.icon-btn {
background: transparent; border: 1px solid var(--border, #e5e7eb); border-radius: 6px;
padding: 5px 10px; font-size: 12px; cursor: pointer;
color: var(--text-secondary, #6b7280); transition: all 0.15s;
}
.icon-btn:hover { color: var(--text-primary, #111827); border-color: #cbd5e1; background: #f9fafb; }
.icon-btn.danger:hover { color: #b91c1c; border-color: #fecaca; background: #fef2f2; }
.users-empty, .users-loading {
text-align: center; padding: 48px 16px;
color: var(--text-secondary, #6b7280); font-size: 13px;
}
.users-empty .big { font-size: 15px; color: var(--text-primary, #111827); margin-bottom: 6px; font-weight: 500; }
.skeleton-row td { padding: 12px 16px; }
.skeleton-row .bar {
background: linear-gradient(90deg, #eef2f7 25%, #e2e8f0 37%, #eef2f7 63%);
background-size: 400% 100%; animation: skeleton 1.4s ease infinite;
height: 10px; border-radius: 4px;
}
@keyframes skeleton { 0% { background-position: 100% 50% } 100% { background-position: 0 50% } }
/* Modal */
.modal-backdrop {
position: fixed; inset: 0; background: rgba(15, 23, 42, 0.55);
display: none; align-items: center; justify-content: center; z-index: 1000;
padding: 16px;
}
.modal-backdrop.is-open { display: flex; }
.modal-card {
background: var(--surface, #fff); border-radius: 12px;
padding: 24px; width: 100%; max-width: 440px;
box-shadow: 0 20px 60px rgba(0, 0, 0, 0.25);
}
.modal-card h3 { margin: 0 0 6px; font-size: 17px; font-weight: 600; }
.modal-card p.sub { margin: 0 0 18px; font-size: 13px; color: var(--text-secondary, #6b7280); }
.modal-card label { display: block; font-size: 12px; font-weight: 500; color: var(--text-secondary, #6b7280); margin: 12px 0 6px; }
.modal-card input[type="text"], .modal-card input[type="email"], .modal-card input[type="password"], .modal-card select {
width: 100%; padding: 9px 12px; border: 1px solid var(--border, #e5e7eb);
border-radius: 8px; font-size: 13px; box-sizing: border-box;
background: var(--surface, #fff); color: var(--text-primary, #111827);
}
.modal-card input:focus, .modal-card select:focus { outline: 2px solid var(--primary, #6366f1); outline-offset: -1px; }
.modal-actions { display: flex; gap: 8px; justify-content: flex-end; margin-top: 20px; }
.modal-btn {
padding: 8px 16px; border-radius: 8px; font-size: 13px; font-weight: 500;
border: 1px solid var(--border, #e5e7eb); background: var(--surface, #fff);
cursor: pointer; transition: all 0.15s;
}
.modal-btn:hover { background: var(--border-light, #f9fafb); }
.modal-btn.primary { background: var(--primary, #6366f1); color: #fff; border-color: var(--primary, #6366f1); }
.modal-btn.primary:hover { filter: brightness(1.05); }
.modal-btn.danger { background: #dc2626; color: #fff; border-color: #dc2626; }
.modal-btn.danger:hover { filter: brightness(1.05); }
.token-reveal {
margin: 12px 0;
padding: 12px; border-radius: 8px;
background: #fffbeb; border: 1px solid #fcd34d;
font-family: var(--font-mono, ui-monospace, "SF Mono", Menlo, monospace);
font-size: 12px; word-break: break-all;
display: flex; align-items: center; gap: 8px;
}
.token-reveal code { flex: 1; }
.copy-btn {
background: var(--primary, #6366f1); color: #fff; border: none;
padding: 4px 10px; border-radius: 6px; font-size: 11px; font-weight: 500;
cursor: pointer; flex-shrink: 0;
}
.copy-btn.copied { background: #10b981; }
/* Toast */
.toast-stack {
position: fixed; bottom: 24px; right: 24px; z-index: 2000;
display: flex; flex-direction: column; gap: 8px;
pointer-events: none;
}
.toast {
background: #111827; color: #fff; padding: 10px 16px;
border-radius: 8px; font-size: 13px; box-shadow: 0 10px 30px rgba(0, 0, 0, 0.25);
opacity: 0; transform: translateY(8px);
transition: opacity 0.2s, transform 0.2s;
pointer-events: auto; max-width: 380px;
}
.toast.show { opacity: 1; transform: translateY(0); }
.toast.success { background: #047857; }
.toast.error { background: #b91c1c; }
</style>
<div class="users-page">
<div class="users-toolbar">
<h2 class="users-title">Users</h2>
<input id="user-search" type="search" class="users-search" placeholder="Filter by email or name…" autocomplete="off">
<button class="modal-btn primary" id="open-create-btn">+ Add user</button>
</div>
<div class="users-table-wrap">
<table class="users-table" id="users-table">
<thead>
<tr>
<th>User</th>
<th>Role</th>
<th>Active</th>
<th>Created</th>
<th>Deactivated</th>
<th style="text-align:right">Actions</th>
</tr>
</thead>
<tbody id="users-tbody"></tbody>
</table>
<div id="users-loading" class="users-loading">Loading users…</div>
<div id="users-empty" class="users-empty" style="display:none;">
<div class="big">No users yet</div>
<div>Click <strong>Add user</strong> to invite the first one.</div>
</div>
</div>
</div>
<!-- Create user modal -->
<div class="modal-backdrop" id="create-modal" role="dialog" aria-modal="true" aria-labelledby="create-modal-title">
<div class="modal-card">
<h3 id="create-modal-title">Add user</h3>
<p class="sub">Invites a new account. The user will need a password (set via the Reset link below) or a configured SSO provider to sign in.</p>
<label for="new-email">Email</label>
<input id="new-email" type="email" required autocomplete="off">
<label for="new-name">Name (optional)</label>
<input id="new-name" type="text" autocomplete="off">
<label for="new-role">Role</label>
<select id="new-role">
<option value="viewer">viewer</option>
<option value="analyst" selected>analyst</option>
<option value="km_admin">km_admin</option>
<option value="admin">admin</option>
</select>
<div class="modal-actions">
<button class="modal-btn" data-close-modal="create-modal">Cancel</button>
<button class="modal-btn primary" id="confirm-create-btn">Create</button>
</div>
</div>
</div>
<!-- Set password modal -->
<div class="modal-backdrop" id="setpwd-modal" role="dialog" aria-modal="true" aria-labelledby="setpwd-title">
<div class="modal-card">
<h3 id="setpwd-title">Set password</h3>
<p class="sub" id="setpwd-target"></p>
<label for="setpwd-input">New password (min 8 chars)</label>
<input id="setpwd-input" type="password" autocomplete="new-password">
<div class="modal-actions">
<button class="modal-btn" data-close-modal="setpwd-modal">Cancel</button>
<button class="modal-btn primary" id="confirm-setpwd-btn">Set password</button>
</div>
</div>
</div>
<!-- Reset token reveal modal
NOTE: The reset_token endpoint still exists for API-level future use,
but no matching "consume this token to set a new password" endpoint
ships today — the magic-link sender would log the user in without
prompting for a password, defeating the reset. Admins should use the
"Set pwd" action (/{id}/set-password) instead. This modal is retained
for API inspection only; the Reset button in the row actions is gone. -->
<div class="modal-backdrop" id="reset-modal" role="dialog" aria-modal="true" aria-labelledby="reset-title">
<div class="modal-card">
<h3 id="reset-title">Password reset token</h3>
<p class="sub" id="reset-target"></p>
<p class="sub">Admins should use <strong>Set password</strong> directly to assign a new password. The magic-link flow is not available for password-reset tokens in this build — this token currently has no matching consumer endpoint.</p>
<div class="token-reveal">
<code id="reset-token-text"></code>
<button class="copy-btn" id="reset-copy-btn">Copy</button>
</div>
<div class="modal-actions">
<button class="modal-btn primary" data-close-modal="reset-modal">Done</button>
</div>
</div>
</div>
<!-- Confirm dialog -->
<div class="modal-backdrop" id="confirm-modal" role="dialog" aria-modal="true" aria-labelledby="confirm-title">
<div class="modal-card">
<h3 id="confirm-title">Are you sure?</h3>
<p class="sub" id="confirm-text"></p>
<div class="modal-actions">
<button class="modal-btn" data-close-modal="confirm-modal">Cancel</button>
<button class="modal-btn danger" id="confirm-ok-btn">Confirm</button>
</div>
</div>
</div>
<div class="toast-stack" id="toast-stack" aria-live="polite"></div>
<script>
const API = "/api/users";
const ROLES = ["viewer", "analyst", "km_admin", "admin"];
function esc(s) {
const d = document.createElement("div");
d.textContent = s == null ? "" : String(s);
return d.innerHTML;
}
function fmtDate(s) { return s ? s.slice(0, 16).replace("T", " ") : "—"; }
function initials(u) {
const src = (u.name || u.email || "?").trim();
const parts = src.split(/[\s@.]+/).filter(Boolean);
return ((parts[0]?.[0] || "?") + (parts[1]?.[0] || "")).toUpperCase();
}
function avatarColor(s) {
// Stable hash → hue
let h = 0;
for (const c of s || "") h = (h * 31 + c.charCodeAt(0)) >>> 0;
return `hsl(${h % 360}, 55%, 50%)`;
}
// ── Toast ──
function toast(msg, kind = "") {
const el = document.createElement("div");
el.className = "toast " + kind;
el.textContent = msg;
document.getElementById("toast-stack").appendChild(el);
requestAnimationFrame(() => el.classList.add("show"));
setTimeout(() => {
el.classList.remove("show");
setTimeout(() => el.remove(), 250);
}, 3500);
}
// ── Modal helpers ──
function openModal(id) {
document.getElementById(id).classList.add("is-open");
const focusable = document.querySelector(`#${id} input, #${id} select, #${id} button.primary`);
focusable && focusable.focus();
}
function closeModal(id) {
document.getElementById(id).classList.remove("is-open");
}
document.querySelectorAll("[data-close-modal]").forEach(el =>
el.addEventListener("click", () => closeModal(el.dataset.closeModal)));
document.querySelectorAll(".modal-backdrop").forEach(el => {
el.addEventListener("click", e => { if (e.target === el) el.classList.remove("is-open"); });
});
document.addEventListener("keydown", e => {
if (e.key === "Escape") document.querySelectorAll(".modal-backdrop.is-open").forEach(m => m.classList.remove("is-open"));
});
// Generic confirm using the modal — returns a Promise<boolean>
function confirmModal(text) {
const modal = document.getElementById("confirm-modal");
document.getElementById("confirm-text").textContent = text;
return new Promise(resolve => {
const okBtn = document.getElementById("confirm-ok-btn");
const cancel = () => { closeModal("confirm-modal"); cleanup(); resolve(false); };
const ok = () => { closeModal("confirm-modal"); cleanup(); resolve(true); };
function cleanup() {
okBtn.removeEventListener("click", ok);
modal.removeEventListener("click", backdropCancel);
}
function backdropCancel(e) { if (e.target === modal) cancel(); }
okBtn.addEventListener("click", ok, { once: true });
modal.addEventListener("click", backdropCancel);
openModal("confirm-modal");
});
}
// ── State ──
let allUsers = [];
let filterText = "";
function renderUsers() {
const tbody = document.getElementById("users-tbody");
const loading = document.getElementById("users-loading");
const empty = document.getElementById("users-empty");
loading.style.display = "none";
const ft = filterText.trim().toLowerCase();
const filtered = ft
? allUsers.filter(u => (u.email || "").toLowerCase().includes(ft) || (u.name || "").toLowerCase().includes(ft))
: allUsers;
if (allUsers.length === 0) {
empty.style.display = "block";
tbody.innerHTML = "";
return;
}
empty.style.display = "none";
if (filtered.length === 0) {
tbody.innerHTML = `<tr><td colspan="6" class="users-loading">No matches for "${esc(filterText)}"</td></tr>`;
return;
}
tbody.innerHTML = "";
for (const u of filtered) {
const tr = document.createElement("tr");
if (!u.active) tr.classList.add("is-deactivated");
const role = u.role || "viewer";
const hasName = !!(u.name && u.name !== u.email);
tr.innerHTML = `
<td>
<div class="user-cell ${hasName ? "" : "no-name"}">
<div class="user-avatar" style="background:${avatarColor(u.email || u.id)}">${esc(initials(u))}</div>
<div class="user-meta">
<span class="name">${esc(u.name || "")}</span>
<span class="email">${esc(u.email)}</span>
</div>
</div>
</td>
<td>
<span class="role-pill role-${esc(role)}" data-action="edit-role" data-user-id="${esc(u.id)}" title="Click to change role">${esc(role)}</span>
</td>
<td>
<label class="toggle">
<input type="checkbox" ${u.active ? "checked" : ""} data-action="toggle-active" data-user-id="${esc(u.id)}">
<span class="toggle-slider"></span>
</label>
</td>
<td class="date-cell">${fmtDate(u.created_at)}</td>
<td class="date-cell">${u.deactivated_at ? fmtDate(u.deactivated_at) : "—"}</td>
<td>
<div class="row-actions">
<a class="icon-btn" href="/admin/tokens?user=${encodeURIComponent(u.email || "")}" title="View this user's personal access tokens">Tokens</a>
<button class="icon-btn" data-action="set-password" data-user-id="${esc(u.id)}" data-user-email="${esc(u.email)}" title="Assign a new password (the 'reset token' flow is not wired end-to-end in this build)">Set pwd</button>
<button class="icon-btn danger" data-action="delete-user" data-user-id="${esc(u.id)}" data-user-email="${esc(u.email)}">Delete</button>
</div>
</td>`;
tbody.appendChild(tr);
}
// Wire up actions via delegation-like loop
tbody.querySelectorAll('[data-action="edit-role"]').forEach(el =>
el.addEventListener("click", () => editRole(el.dataset.userId)));
tbody.querySelectorAll('[data-action="toggle-active"]').forEach(el =>
el.addEventListener("change", () => toggleActive(el.dataset.userId, el.checked)));
// Note: the "Reset" row action has been removed (the reset_token endpoint
// has no matching consumer in this build); admins use Set pwd instead.
// resetPassword() below is kept for API-level inspection / future use.
tbody.querySelectorAll('[data-action="set-password"]').forEach(el =>
el.addEventListener("click", () => openSetPassword(el.dataset.userId, el.dataset.userEmail)));
tbody.querySelectorAll('[data-action="delete-user"]').forEach(el =>
el.addEventListener("click", () => delUser(el.dataset.userId, el.dataset.userEmail)));
}
async function loadUsers() {
try {
const r = await fetch(API, { credentials: "include" });
if (!r.ok) throw new Error("HTTP " + r.status);
allUsers = await r.json();
renderUsers();
} catch (e) {
document.getElementById("users-loading").textContent = "Failed to load users: " + e.message;
toast("Failed to load users", "error");
}
}
document.getElementById("user-search").addEventListener("input", e => {
filterText = e.target.value;
renderUsers();
});
// ── Role editing via cycling pill click ──
async function editRole(id) {
const u = allUsers.find(x => x.id === id);
if (!u) return;
const next = ROLES[(ROLES.indexOf(u.role || "viewer") + 1) % ROLES.length];
if (!await confirmModal(`Change role for ${u.email} from "${u.role}" to "${next}"?`)) return;
await patch(id, { role: next }, `Role changed to ${next}`);
}
async function toggleActive(id, active) {
const path = active ? "activate" : "deactivate";
const r = await fetch(`${API}/${id}/${path}`, { method: "POST", credentials: "include" });
if (!r.ok) {
toast("Failed: " + (await r.text()), "error");
loadUsers();
return;
}
toast(active ? "User activated" : "User deactivated", "success");
loadUsers();
}
async function patch(id, body, successMsg) {
const r = await fetch(`${API}/${id}`, {
method: "PATCH", credentials: "include",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(body),
});
if (!r.ok) { toast("Failed: " + (await r.text()), "error"); return; }
toast(successMsg, "success");
loadUsers();
}
// ── Reset password ──
async function resetPassword(id, email) {
if (!await confirmModal(`Generate a reset token for ${email}?`)) return;
const r = await fetch(`${API}/${id}/reset-password`, { method: "POST", credentials: "include" });
const data = await r.json().catch(() => ({}));
if (!r.ok) { toast("Failed: " + (data.detail || r.status), "error"); return; }
document.getElementById("reset-target").textContent = `For ${email}`;
document.getElementById("reset-token-text").textContent = data.reset_token;
const copyBtn = document.getElementById("reset-copy-btn");
copyBtn.textContent = "Copy"; copyBtn.classList.remove("copied");
copyBtn.onclick = async () => {
try {
await navigator.clipboard.writeText(data.reset_token);
copyBtn.textContent = "Copied!"; copyBtn.classList.add("copied");
setTimeout(() => { copyBtn.textContent = "Copy"; copyBtn.classList.remove("copied"); }, 1500);
} catch { toast("Copy failed — select the text manually", "error"); }
};
openModal("reset-modal");
}
// ── Set password ──
function openSetPassword(id, email) {
document.getElementById("setpwd-target").textContent = `For ${email}`;
const input = document.getElementById("setpwd-input");
input.value = "";
openModal("setpwd-modal");
document.getElementById("confirm-setpwd-btn").onclick = async () => {
const pwd = input.value;
if (!pwd || pwd.length < 8) { toast("Password must be at least 8 characters", "error"); return; }
const r = await fetch(`${API}/${id}/set-password`, {
method: "POST", credentials: "include",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ password: pwd }),
});
if (!r.ok) { toast("Failed: " + (await r.text()), "error"); return; }
closeModal("setpwd-modal");
toast("Password updated", "success");
};
}
// ── Delete ──
async function delUser(id, email) {
if (!await confirmModal(`Delete ${email}? This cannot be undone.`)) return;
const r = await fetch(`${API}/${id}`, { method: "DELETE", credentials: "include" });
if (!r.ok) { toast("Failed: " + (await r.text()), "error"); return; }
toast("User deleted", "success");
loadUsers();
}
// ── Create ──
document.getElementById("open-create-btn").addEventListener("click", () => {
document.getElementById("new-email").value = "";
document.getElementById("new-name").value = "";
document.getElementById("new-role").value = "analyst";
openModal("create-modal");
});
document.getElementById("confirm-create-btn").addEventListener("click", async () => {
const email = document.getElementById("new-email").value.trim();
const name = document.getElementById("new-name").value.trim();
const role = document.getElementById("new-role").value;
if (!email) { toast("Email is required", "error"); return; }
const r = await fetch(API, {
method: "POST", credentials: "include",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ email, name: name || email.split("@")[0], role }),
});
if (!r.ok) { toast("Failed: " + (await r.text()), "error"); return; }
closeModal("create-modal");
toast("User created", "success");
loadUsers();
});
loadUsers();
</script>
{% endblock %}

20
app/web/templates/base.html
View file

@ -9,25 +9,9 @@
{% include '_theme.html' %}
</head>
<body>
<div class="container">
<header>
<div class="logo">
<h1>Data Analyst Portal</h1>
<p class="subtitle">{{ config.INSTANCE_SUBTITLE }}</p>
</div>
{% if session.user %}
<nav>
<span class="user-info">
{% if session.user.picture %}
<img src="{{ session.user.picture }}" alt="Profile" class="avatar">
{% endif %}
{{ session.user.email }}
</span>
<a href="{{ url_for('auth.logout') }}" class="btn btn-secondary btn-sm">Logout</a>
</nav>
{% endif %}
</header>
{% include '_app_header.html' %}
<div class="container">
{% with messages = get_flashed_messages(with_categories=true) %}
{% if messages %}
<div class="flash-messages">

19
app/web/templates/catalog.html
View file

@ -1494,24 +1494,7 @@
<body>
<!-- ═══════════════ HEADER ═══════════════ -->
<header class="header">
<div class="header-left">
<a href="{{ url_for('dashboard') }}" class="header-back" title="Back to Dashboard">
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
<path d="M19 12H5M12 19l-7-7 7-7"/>
</svg>
</a>
<div class="header-logo-group">
<div class="header-logo">
{{ config.LOGO_SVG | safe }}
</div>
<span class="header-subtitle">Data Catalog</span>
</div>
</div>
<div class="header-right">
{% if data_stats.last_updated %}Last sync: {{ data_stats.last_updated }}{% endif %}
</div>
</header>
{% include '_app_header.html' %}
<!-- ═══════════════ PAGE TITLE ═══════════════ -->
<div class="page-title">

44
app/web/templates/corporate_memory.html
View file

@ -514,49 +514,7 @@
<body>
<div class="container-memory">
<!-- Header -->
<header class="page-header">
<div class="header-left">
<a href="{{ url_for('dashboard') }}" class="back-link">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2">
<path d="M15 18l-6-6 6-6"/>
</svg>
Dashboard
</a>
<div class="page-title">
<span class="page-icon">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2">
<path d="M12 2L2 7l10 5 10-5-10-5z"/>
<path d="M2 17l10 5 10-5"/>
<path d="M2 12l10 5 10-5"/>
</svg>
</span>
<h1>Corporate Memory</h1>
</div>
</div>
<div style="display: flex; align-items: center; gap: var(--space-4);">
{% if governance.is_km_admin %}
<a href="{{ url_for('corporate_memory_admin') }}" class="admin-link-btn">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2">
<path d="M12 22s8-4 8-10V5l-8-3-8 3v7c0 6 8 10 8 10z"/>
</svg>
Admin Review
{% if governance.pending_count > 0 %}
<span class="pending-badge">{{ governance.pending_count }}</span>
{% endif %}
</a>
{% endif %}
<div class="user-info-v2">
{% if session.user.picture %}
<img src="{{ session.user.picture }}" alt="Profile" class="avatar-v2">
{% else %}
<div class="avatar-v2" style="background: var(--primary-light); display: flex; align-items: center; justify-content: center; font-weight: 600; color: var(--primary); font-size: 12px;">
{{ session.user.email[:2].upper() }}
</div>
{% endif %}
{{ session.user.email }}
</div>
</div>
</header>
{% include '_app_header.html' %}
<!-- Stats Bar -->
<div class="stats-bar">

29
app/web/templates/corporate_memory_admin.html
View file

@ -802,34 +802,7 @@
<body>
<div class="container-memory">
<!-- Header -->
<header class="page-header">
<div class="header-left">
<a href="{{ url_for('corporate_memory') }}" class="back-link">
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2">
<path d="M15 18l-6-6 6-6"/>
</svg>
Corporate Memory
</a>
<div class="page-title">
<span class="page-icon">
<svg width="18" height="18" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2">
<path d="M12 22s8-4 8-10V5l-8-3-8 3v7c0 6 8 10 8 10z"/>
</svg>
</span>
<h1>Corporate Memory &mdash; Admin</h1>
</div>
</div>
<div class="user-info-v2">
{% if session.user.picture %}
<img src="{{ session.user.picture }}" alt="Profile" class="avatar-v2">
{% else %}
<div class="avatar-v2" style="background: var(--primary-light); display: flex; align-items: center; justify-content: center; font-weight: 600; color: var(--primary); font-size: 12px;">
{{ session.user.email[:2].upper() }}
</div>
{% endif %}
{{ session.user.email }}
</div>
</header>
{% include '_app_header.html' %}
<!-- Stats Bar -->
<div class="stats-bar">

326
app/web/templates/dashboard.html
View file

@ -32,9 +32,20 @@
gap: 2px;
}
.header-logo {
display: inline-flex;
align-items: center;
text-decoration: none;
color: inherit;
}
.header-logo svg {
display: block;
}
a.header-logo:focus-visible {
outline: 2px solid var(--primary);
outline-offset: 2px;
border-radius: 4px;
}
.header-subtitle {
font-size: 11px;
@ -1221,6 +1232,147 @@
margin-left: auto;
}
/* Error banner for setup flow */
.setup-error {
margin-top: 12px;
padding: 10px 14px;
background: rgba(234, 88, 12, 0.12);
border-left: 3px solid #EA580C;
border-radius: 6px;
color: #FFF;
font-size: 13px;
}
.env-setup-cta .btn-setup[disabled] {
opacity: 0.7;
cursor: wait;
}
/* ── Setup instructions preview (read-only card inside env-setup-cta) ── */
.setup-preview-card {
margin-top: 18px;
background: rgba(0, 0, 0, 0.25);
border: 1px solid rgba(255, 255, 255, 0.12);
border-radius: 8px;
padding: 14px 16px;
}
.setup-preview-summary {
list-style: none;
cursor: pointer;
display: flex;
align-items: center;
gap: 8px;
user-select: none;
}
.setup-preview-summary::-webkit-details-marker { display: none; }
.setup-preview-chevron {
display: inline-block;
font-size: 11px;
color: rgba(255, 255, 255, 0.7);
transition: transform 0.15s ease;
}
details[open] > .setup-preview-summary .setup-preview-chevron { transform: rotate(90deg); }
.setup-preview-title {
font-size: 12px;
font-weight: 600;
letter-spacing: 0.5px;
text-transform: uppercase;
color: rgba(255, 255, 255, 0.85);
margin: 0;
}
.setup-preview-sub {
font-size: 12px;
color: rgba(255, 255, 255, 0.65);
margin: 0 0 10px 0;
}
.setup-preview-pre {
background: #1e1e2e;
border-radius: 6px;
padding: 14px 16px;
margin: 0;
max-height: 400px;
overflow-y: auto;
font-family: var(--font-mono);
font-size: 12.5px;
line-height: 1.55;
color: #cdd6f4;
white-space: pre-wrap;
word-break: break-word;
}
.setup-preview-code { font-family: inherit; font-size: inherit; }
.setup-preview-pre .placeholder-token {
background: rgba(249, 226, 175, 0.12);
color: #f9e2af;
padding: 0 4px;
border-radius: 3px;
font-style: italic;
}
/* Fallback modal (when clipboard is blocked) */
.setup-fallback-overlay {
position: fixed;
inset: 0;
background: rgba(0, 0, 0, 0.45);
display: flex;
align-items: center;
justify-content: center;
z-index: 1000;
}
.setup-fallback-modal {
background: var(--surface);
border-radius: 12px;
padding: 20px;
max-width: 720px;
width: calc(100% - 32px);
max-height: calc(100vh - 64px);
display: flex;
flex-direction: column;
gap: 12px;
box-shadow: 0 20px 48px rgba(0, 0, 0, 0.3);
}
.setup-fallback-modal h4 {
margin: 0;
font-size: 15px;
color: var(--text-primary);
}
.setup-fallback-modal p {
margin: 0;
font-size: 13px;
color: var(--text-secondary);
}
.setup-fallback-modal textarea {
flex: 1;
min-height: 260px;
font-family: var(--font-mono);
font-size: 12px;
padding: 10px;
border: 1px solid var(--border);
border-radius: 8px;
background: var(--background);
color: var(--text-primary);
resize: vertical;
}
.setup-fallback-actions {
display: flex;
justify-content: flex-end;
gap: 8px;
}
.setup-fallback-actions button {
font-family: var(--font-primary);
font-size: 13px;
font-weight: 500;
padding: 8px 16px;
border-radius: 6px;
border: 1px solid var(--border);
background: var(--surface);
color: var(--text-primary);
cursor: pointer;
}
.setup-fallback-actions button.primary {
background: var(--primary);
color: #FFF;
border-color: var(--primary);
}
/* ── Setup Banner (bottom, for returning users) ── */
.setup-banner {
background: var(--background);
@ -1834,26 +1986,8 @@
</head>
<body>
<!-- ═══════════════ HEADER ═══════════════ -->
<header class="header">
<div class="header-left">
<div class="header-logo">
{{ config.LOGO_SVG | safe }}
</div>
<span class="header-subtitle">Data Analyst Portal</span>
</div>
{% if session.user %}
<div class="header-right">
<span class="header-email">{{ session.user.email }}</span>
{% if session.user.picture %}
<img src="{{ session.user.picture }}" alt="Profile" class="avatar-img">
{% else %}
<div class="avatar">{{ (user.name or user.email)[:2] | upper }}</div>
{% endif %}
<a href="{{ url_for('auth.logout') }}" class="btn-logout">Logout</a>
</div>
{% endif %}
</header>
<!-- ═══════════════ HEADER (shared partial) ═══════════════ -->
{% include '_app_header.html' %}
{% with messages = get_flashed_messages(with_categories=true) %}
{% if messages %}
@ -1875,19 +2009,33 @@
{% if not account_details or not account_details.last_sync_display %}
<!-- ═══════════════ ENVIRONMENT SETUP CTA ═══════════════ -->
<div class="env-setup-cta">
<h3>Set up your local environment</h3>
<p class="env-subtitle">Run Claude Code in your project folder and paste the setup instructions to configure SSH, sync data, and initialize DuckDB.</p>
<h3>Set up a new Claude Code</h3>
<p class="env-subtitle">Generates a personal access token and copies a ready-to-paste setup script to your clipboard. Paste into Claude Code to finish.</p>
<div class="env-setup-row">
<span class="code-pill">cd {{ project_dir }} && claude</span>
<button onclick="copyBootstrapInstructions(this)" class="btn-setup" id="bootstrapCopyBtn">
<button type="button" onclick="setupNewClaude(this)" class="btn-setup" id="setupClaudeBtn">
<svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2">
<rect x="9" y="9" width="13" height="13" rx="2" ry="2"></rect>
<path d="M5 15H4a2 2 0 0 1-2-2V4a2 2 0 0 1 2-2h9a2 2 0 0 1 2 2v1"></path>
</svg>
Copy Setup Instructions
Setup a new Claude Code
</button>
<span class="env-hint">Paste into Claude Code to complete setup</span>
<span class="env-hint">Valid 90 days · token stays in clipboard only</span>
</div>
<div id="setupClaudeError" class="setup-error" role="alert" style="display:none;"></div>
<details class="setup-preview-card" aria-label="Preview of the clipboard payload">
<summary class="setup-preview-summary">
<span class="setup-preview-chevron" aria-hidden="true">▸</span>
<span class="setup-preview-title">What Claude Code will receive</span>
</summary>
<p class="setup-preview-sub">
Read-only preview. The real token is generated the moment
you click the button above and is placed directly in your
clipboard — never shown on this page.
</p>
{% with preview_mode=True %}
{% include "_claude_setup_instructions.jinja" %}
{% endwith %}
</details>
</div>
{% endif %}
@ -2266,17 +2414,6 @@
</div><!-- /right-column -->
</div><!-- /dashboard-grid -->
<!-- ═══════════════ SETUP BANNER ═══════════════ -->
<div class="setup-banner">
<div class="setup-banner-text">
<div class="setup-banner-title">Set up a new machine</div>
<div class="setup-banner-desc">Copy instructions and paste into Claude Code to configure another local environment.</div>
</div>
<button onclick="copyBootstrapInstructions(this)" class="btn-setup-secondary" id="bootstrapCopyBtnBottom">
Copy Setup Instructions
</button>
</div>
</main>
{% else %}
@ -2421,19 +2558,112 @@
});
}
function copyBootstrapInstructions(btn) {
var instructions = {{ setup_instructions | tojson }};
// ══════════════════════════════════════════════════════════════════
// "Setup a new Claude Code" one-click flow
// ══════════════════════════════════════════════════════════════════
// Template + renderer included from _claude_setup_instructions.jinja
// so dashboard.html and install.html always render the same payload.
{% include "_claude_setup_instructions.jinja" %}
var button = btn || document.getElementById('bootstrapCopyBtn');
var origText = button.textContent;
copyToClipboard(instructions).then(function() {
button.textContent = 'Copied!';
button.classList.add('copied');
setTimeout(function() {
button.textContent = origText;
button.classList.remove('copied');
}, 2000);
function defaultTokenName() {
var stamp = new Date().toISOString().slice(0, 16).replace("T", " ");
return "Claude Code — " + stamp;
}
function showSetupFallback(instructions) {
// Clipboard blocked (non-secure context, permission denied, etc.).
// Show a modal with the instructions preselected so the user can Ctrl+C.
var overlay = document.createElement('div');
overlay.className = 'setup-fallback-overlay';
overlay.innerHTML =
'<div class="setup-fallback-modal" role="dialog" aria-modal="true" aria-labelledby="setupFallbackTitle">' +
'<h4 id="setupFallbackTitle">Copy these setup instructions</h4>' +
'<p>Your browser blocked automatic clipboard access. Select all, copy, then paste into Claude Code.</p>' +
'<textarea readonly></textarea>' +
'<div class="setup-fallback-actions">' +
'<button type="button" data-action="close">Close</button>' +
'<button type="button" class="primary" data-action="select">Select all</button>' +
'</div>' +
'</div>';
document.body.appendChild(overlay);
var ta = overlay.querySelector('textarea');
ta.value = instructions;
ta.focus();
ta.select();
overlay.addEventListener('click', function(ev) {
if (ev.target === overlay) { document.body.removeChild(overlay); }
});
overlay.querySelector('[data-action="close"]').addEventListener('click', function() {
document.body.removeChild(overlay);
});
overlay.querySelector('[data-action="select"]').addEventListener('click', function() {
ta.focus();
ta.select();
});
}
async function setupNewClaude(btn) {
var errEl = document.getElementById('setupClaudeError');
if (errEl) { errEl.style.display = 'none'; errEl.textContent = ''; }
var origText = btn.textContent;
btn.disabled = true;
btn.textContent = 'Generating token…';
try {
var resp = await fetch('/auth/tokens', {
method: 'POST',
credentials: 'include',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
name: defaultTokenName(),
expires_in_days: 90,
}),
});
if (resp.status === 401) {
// Session expired mid-flight — bounce to login and come back.
window.location.href = '/login?next=' + encodeURIComponent(window.location.pathname);
return;
}
if (!resp.ok) {
var detail = 'HTTP ' + resp.status;
try {
var body = await resp.json();
if (body && body.detail) { detail = body.detail; }
} catch (_) { /* non-JSON */ }
throw new Error(detail);
}
var data = await resp.json();
if (!data || !data.token) {
throw new Error('Server did not return a token.');
}
var serverUrl = window.location.origin;
var instructions = renderSetupInstructions(serverUrl, data.token);
try {
await copyToClipboard(instructions);
btn.textContent = 'Copied! Paste into Claude Code';
btn.classList.add('copied');
setTimeout(function() {
btn.textContent = origText;
btn.classList.remove('copied');
btn.disabled = false;
}, 3000);
} catch (clipErr) {
// Clipboard denied — fall back to modal. Re-enable the button immediately.
btn.textContent = origText;
btn.disabled = false;
showSetupFallback(instructions);
}
// Token is NOT stored in DOM after the modal closes / flash disappears.
} catch (err) {
btn.textContent = origText;
btn.disabled = false;
if (errEl) {
errEl.textContent = 'Setup failed: ' + (err && err.message ? err.message : err);
errEl.style.display = 'block';
} else {
alert('Setup failed: ' + (err && err.message ? err.message : err));
}
}
}
async function updateSyncSettings() {

1077
app/web/templates/install.html Normal file
View file

File diff suppressed because it is too large Load diff

1
app/web/templates/login_email.html
View file

@ -19,6 +19,7 @@
<!-- Sign In Tab -->
<div id="signin-tab" class="auth-tab-content active">
<form method="POST" action="/auth/password/login/web" class="login-form">
<input type="hidden" name="next" value="{{ next_path|default('', true) }}">
<div class="form-group">
<label for="email-signin">Email Address</label>
<input type="email"

1294
app/web/templates/my_tokens.html Normal file
View file

File diff suppressed because it is too large Load diff

5
cli/client.py
View file

@ -35,6 +35,11 @@ def api_delete(path: str, **kwargs) -> httpx.Response:
return client.delete(path, **kwargs)
def api_patch(path: str, **kwargs) -> httpx.Response:
with get_client() as client:
return client.patch(path, **kwargs)
def stream_download(path: str, target_path: str, progress_callback=None) -> int:
"""Stream download a file from the API. Returns bytes written."""
with get_client(timeout=300.0) as client:

96
cli/commands/admin.py
View file

@ -4,7 +4,7 @@ import json
import typer
from cli.client import api_get, api_post, api_delete
from cli.client import api_get, api_post, api_delete, api_patch
admin_app = typer.Typer(help="Admin operations (requires admin role)")
@ -38,7 +38,10 @@ def list_users(as_json: bool = typer.Option(False, "--json")):
typer.echo(json.dumps(users, indent=2))
else:
for u in users:
typer.echo(f" {u['email']:30s} role={u['role']:10s} id={u['id'][:8]}")
status_str = "active" if u.get("active", True) else "DEACTIVATED"
typer.echo(
f" {u['email']:30s} role={u['role']:10s} {status_str:12s} id={u['id'][:8]}"
)
@admin_app.command("remove-user")
@ -240,3 +243,92 @@ def metadata_apply(
typer.echo(f"Pushed metadata for {table_id} to source.")
else:
typer.echo(f"Failed to push {table_id}: {resp.json().get('detail', resp.text)}", err=True)
# ---- User management (#11) ----
def _resolve_user_id(ref: str) -> str:
"""Accept either a UUID or an email; look up email → id via list."""
if "@" not in ref:
return ref
resp = api_get("/api/users")
if resp.status_code != 200:
typer.echo(f"Could not list users: {resp.text}", err=True)
raise typer.Exit(1)
for u in resp.json():
if u.get("email") == ref:
return u["id"]
typer.echo(f"User not found: {ref}", err=True)
raise typer.Exit(1)
def _print_user_result(resp, ok_msg: str) -> None:
if resp.status_code in (200, 204):
typer.echo(ok_msg)
else:
try:
detail = resp.json().get("detail", resp.text)
except Exception:
detail = resp.text
typer.echo(f"Failed: {detail}", err=True)
raise typer.Exit(1)
@admin_app.command("set-role")
def set_role(
user_ref: str = typer.Argument(..., help="User id or email"),
role: str = typer.Argument(..., help="viewer | analyst | km_admin | admin"),
):
"""Set a user's role."""
uid = _resolve_user_id(user_ref)
resp = api_patch(f"/api/users/{uid}", json={"role": role})
_print_user_result(resp, f"Updated role for {user_ref} → {role}")
@admin_app.command("deactivate")
def deactivate(user_ref: str = typer.Argument(..., help="User id or email")):
"""Deactivate a user (blocks login, existing tokens also rejected)."""
uid = _resolve_user_id(user_ref)
resp = api_post(f"/api/users/{uid}/deactivate")
_print_user_result(resp, f"Deactivated {user_ref}")
@admin_app.command("activate")
def activate(user_ref: str = typer.Argument(..., help="User id or email")):
"""Re-activate a deactivated user."""
uid = _resolve_user_id(user_ref)
resp = api_post(f"/api/users/{uid}/activate")
_print_user_result(resp, f"Activated {user_ref}")
@admin_app.command("reset-password")
def reset_password(user_ref: str = typer.Argument(..., help="User id or email")):
"""Generate a reset token (emailed if SMTP/SendGrid configured)."""
uid = _resolve_user_id(user_ref)
resp = api_post(f"/api/users/{uid}/reset-password")
if resp.status_code == 200:
data = resp.json()
typer.echo(f"Reset token: {data['reset_token']}")
typer.echo(f"Email sent: {data['email_sent']}")
else:
typer.echo(f"Failed: {resp.json().get('detail', resp.text)}", err=True)
raise typer.Exit(1)
@admin_app.command("set-password")
def set_password(
user_ref: str = typer.Argument(..., help="User id or email"),
password: str = typer.Option(
..., prompt=True, hide_input=True, confirmation_prompt=True,
help="New password (hidden input)",
),
):
"""Set a user's password directly (force-reset flow)."""
uid = _resolve_user_id(user_ref)
resp = api_post(f"/api/users/{uid}/set-password", json={"password": password})
if resp.status_code == 204:
typer.echo(f"Password set for {user_ref}")
else:
typer.echo(f"Failed: {resp.json().get('detail', resp.text)}", err=True)
raise typer.Exit(1)

163
cli/commands/auth.py
View file

@ -1,9 +1,17 @@
"""Auth commands — da login, da logout, da whoami."""
"""Auth commands — da login, da logout, da whoami, da auth import-token."""
import httpx
import typer
from cli.client import api_post, api_get
from cli.config import save_token, clear_token, get_token, get_server_url
from cli.config import (
save_token,
clear_token,
get_token,
get_server_url,
save_config,
load_config,
)
auth_app = typer.Typer(help="Authentication commands")
@ -11,22 +19,50 @@ auth_app = typer.Typer(help="Authentication commands")
@auth_app.command()
def login(
email: str = typer.Option(..., prompt=True, help="Your email address"),
password: str = typer.Option(
"", prompt="Password (leave empty for magic-link / OAuth accounts)",
hide_input=True, help="Your password (if the account has one)",
),
server: str = typer.Option(None, help="Server URL override"),
):
"""Login and obtain a JWT token."""
"""Login and obtain a JWT token.
Password-enabled accounts: enter the password when prompted.
Magic-link / OAuth accounts: leave the password empty — the server will
respond with guidance pointing you to the correct auth provider.
"""
if server:
import os
os.environ["DA_SERVER"] = server
body = {"email": email}
if password:
body["password"] = password
try:
resp = api_post("/auth/token", json={"email": email})
resp = api_post("/auth/token", json=body)
if resp.status_code == 200:
data = resp.json()
save_token(data["access_token"], data["email"], data["role"])
typer.echo(f"Logged in as {data['email']} (role: {data['role']})")
return
# Helpful error for accounts that cannot login via password.
try:
detail = resp.json().get("detail", resp.text)
except Exception:
detail = resp.text
if resp.status_code == 401 and "external authentication" in str(detail).lower():
typer.echo(
"This account uses a magic link / OAuth provider. "
"Sign in via the web UI, open /tokens, and create a personal "
"access token — then export it as DA_TOKEN.",
err=True,
)
else:
typer.echo(f"Login failed: {resp.json().get('detail', resp.text)}", err=True)
raise typer.Exit(1)
typer.echo(f"Login failed: {detail}", err=True)
raise typer.Exit(1)
except typer.Exit:
raise
except Exception as e:
typer.echo(f"Connection error: {e}", err=True)
raise typer.Exit(1)
@ -56,3 +92,118 @@ def whoami():
except Exception:
typer.echo("Invalid token. Run: da login")
raise typer.Exit(1)
@auth_app.command("import-token")
def import_token(
token: str = typer.Option(..., "--token", help="JWT / Personal Access Token to import"),
server: str = typer.Option(
None,
"--server",
help="Server URL (defaults to ~/.config/da/config.yaml or $DA_SERVER)",
),
email: str = typer.Option(
None,
"--email",
help="Override email (used only if the JWT lacks an 'email' claim)",
),
role: str = typer.Option(
None,
"--role",
help="Override role (used only if the JWT lacks a 'role' claim)",
),
skip_verify: bool = typer.Option(
False,
"--skip-verify",
help="Skip the server-side verification step (offline import)",
),
):
"""Import a personal access token non-interactively.
Decodes the JWT locally to extract the email/role claims, verifies it
against the server, and writes it to ~/.config/da/token.json using the
canonical format so subsequent `da auth whoami` / `da sync` calls
authenticate cleanly.
Example:
da auth import-token --token "$AGNES_PAT"
da auth import-token --token "$AGNES_PAT" --server https://agnes.example.com
"""
import os
import jwt as pyjwt
# 1) Seed server URL so the verify call below uses the right base URL.
if server:
save_config({"server": server})
os.environ["DA_SERVER"] = server
else:
cfg = load_config()
if not os.environ.get("DA_SERVER") and not cfg.get("server"):
typer.echo(
"No server configured. Pass --server https://<host> or set "
"DA_SERVER, or seed ~/.config/da/config.yaml first.",
err=True,
)
raise typer.Exit(1)
# 2) Decode JWT without signature verification — we only need the claims.
resolved_email = email
resolved_role = role
try:
payload = pyjwt.decode(token, options={"verify_signature": False})
resolved_email = resolved_email or payload.get("email")
resolved_role = resolved_role or payload.get("role")
except Exception as e:
typer.echo(f"Could not decode token as JWT: {e}", err=True)
raise typer.Exit(1)
# 3) Server-side verification. The server has no dedicated /auth/me — we
# use /api/catalog/tables which is the lightest endpoint that every
# authenticated user can call and also exercises the PAT validation
# path (revocation, expiry, token_hash match).
verify_url = get_server_url()
if not skip_verify:
headers = {"Authorization": f"Bearer {token}"}
try:
with httpx.Client(base_url=verify_url, headers=headers, timeout=15.0) as client:
resp = client.get("/api/catalog/tables")
except Exception as e:
typer.echo(f"Could not reach server {verify_url}: {e}", err=True)
raise typer.Exit(1)
if resp.status_code == 401:
detail = "unauthorized"
try:
detail = resp.json().get("detail", detail)
except Exception:
pass
typer.echo(f"Token rejected by server ({verify_url}): {detail}", err=True)
raise typer.Exit(1)
if resp.status_code >= 500:
typer.echo(
f"Server error from {verify_url} during verification "
f"(HTTP {resp.status_code}). Re-run with --skip-verify to bypass.",
err=True,
)
raise typer.Exit(1)
# 4) Fallback claim lookup via a response the server might include.
# /api/catalog/tables doesn't return user info, but other JWT
# issuers might later gain an /auth/me. For now, we rely on JWT
# claims + the CLI overrides.
# 5) If we still lack email/role, refuse rather than writing a partial record.
if not resolved_email or not resolved_role:
typer.echo(
"Token is missing 'email' and/or 'role' claims. Re-issue the token "
"or pass --email and --role explicitly.",
err=True,
)
raise typer.Exit(1)
# 6) Persist in the canonical on-disk format used by cli/config.py.
save_token(token, resolved_email, resolved_role)
typer.echo(f"Imported token for {resolved_email} (role: {resolved_role}).")
from cli.commands.tokens import token_app
auth_app.add_typer(token_app, name="token")

97
cli/commands/tokens.py Normal file
View file

@ -0,0 +1,97 @@
"""`da auth token` — manage personal access tokens (#12)."""
import json as _json
import re
from typing import Optional
import typer
from cli.client import api_post, api_get, api_delete
token_app = typer.Typer(help="Personal access tokens (long-lived CLI/CI auth)")
def _parse_ttl(ttl: Optional[str]) -> Optional[int]:
"""Parse "30d", "90d", "365d", "never" → days (int) or None."""
if not ttl or ttl.lower() in ("never", "none", "no-expiry"):
return None
m = re.fullmatch(r"(\d+)d", ttl.lower().strip())
if not m:
raise typer.BadParameter(f"Invalid TTL: {ttl}. Use e.g. 30d, 90d, 365d, or 'never'.")
return int(m.group(1))
@token_app.command("create")
def create(
name: str = typer.Option(..., "--name", help="Human label for the token"),
ttl: str = typer.Option("90d", "--ttl", help="Lifetime (e.g. 30d, 90d, 365d, never)"),
raw: bool = typer.Option(False, "--raw", help="Print only the raw token (for CI)"),
):
"""Create a new personal access token."""
body = {"name": name, "expires_in_days": _parse_ttl(ttl)}
resp = api_post("/auth/tokens", json=body)
if resp.status_code != 201:
typer.echo(f"Failed: {resp.json().get('detail', resp.text)}", err=True)
raise typer.Exit(1)
data = resp.json()
if raw:
typer.echo(data["token"])
return
typer.echo("Personal access token created — this is shown ONCE:")
typer.echo("")
typer.echo(f" {data['token']}")
typer.echo("")
typer.echo(f"id: {data['id']}")
typer.echo(f"name: {data['name']}")
typer.echo(f"expires: {data.get('expires_at') or 'never'}")
typer.echo("")
typer.echo("Export it so `da` can use it:")
typer.echo(f" export DA_TOKEN={data['token']}")
@token_app.command("list")
def list_tokens(as_json: bool = typer.Option(False, "--json")):
"""List your personal access tokens."""
resp = api_get("/auth/tokens")
if resp.status_code != 200:
typer.echo(f"Failed: {resp.json().get('detail', resp.text)}", err=True)
raise typer.Exit(1)
rows = resp.json()
if as_json:
typer.echo(_json.dumps(rows, indent=2))
return
if not rows:
typer.echo("No tokens yet. Create one with: da auth token create --name <label>")
return
typer.echo(f"{'ID':36s} {'NAME':20s} {'PREFIX':10s} {'EXPIRES':20s} {'LAST USED':20s} STATUS")
for r in rows:
status = "revoked" if r.get("revoked_at") else "active"
typer.echo(
f"{r['id']:36s} {r['name']:20s} {r['prefix']:10s} "
f"{(r.get('expires_at') or 'never'):20s} "
f"{(r.get('last_used_at') or '-'):20s} {status}"
)
@token_app.command("revoke")
def revoke(
ident: str = typer.Argument(..., help="Token id, prefix, or name"),
):
"""Revoke a token."""
resp = api_get("/auth/tokens")
if resp.status_code != 200:
typer.echo(f"Failed to list tokens: {resp.text}", err=True)
raise typer.Exit(1)
rows = resp.json()
match = next(
(r for r in rows if r["id"] == ident or r["prefix"] == ident or r["name"] == ident),
None,
)
if not match:
typer.echo(f"No token matches {ident}", err=True)
raise typer.Exit(1)
del_resp = api_delete(f"/auth/tokens/{match['id']}")
if del_resp.status_code != 204:
typer.echo(f"Failed: {del_resp.text}", err=True)
raise typer.Exit(1)
typer.echo(f"Revoked token {match['id']} ({match['name']})")

5
cli/skills/security.md
View file

@ -32,6 +32,9 @@ User scripts run in isolated subprocess with:
- Stdout/stderr size cap (64KB)
## JWT Tokens
- Issued on login, valid 30 days
- Session tokens: issued on interactive login (`da login`), valid 24 hours.
- For long-lived CLI / CI use, create a Personal Access Token via the UI
(`/tokens` → New token) or CLI (`da auth token create`).
- PATs are revocable and auditable; session tokens are not.
- Contains: user_id, email, role
- Set JWT_SECRET_KEY in .env (min 32 chars)

45
docs/HEADLESS_USAGE.md Normal file
View file

@ -0,0 +1,45 @@
# Headless / CI usage
For unattended clients (CI, cron, Claude Code), authenticate with a Personal Access Token (PAT) rather than an interactive session.
## Create a PAT
**Via UI:** sign in, open `/tokens`, create a token. Copy the raw value — it is shown exactly once.
**Via CLI (requires an interactive session):**
```bash
da auth token create --name "github-actions" --ttl 365d --raw
```
The `--raw` flag prints only the token, suitable for piping into a secret store.
## Use the PAT
Set the `DA_TOKEN` env var:
```bash
export DA_TOKEN=<your-token>
da query "SELECT 1"
```
### GitHub Actions example
```yaml
- name: Sync data
env:
DA_TOKEN: ${{ secrets.AGNES_TOKEN }}
DA_SERVER: https://agnes.example.com
run: |
pip install data-analyst
da sync --all
```
## Revoke
```bash
da auth token list
da auth token revoke <id|prefix|name>
```
Or from `/tokens` → Revoke.

259
docs/superpowers/plans/2026-04-09-dead-code-cleanup.md Normal file
View file

@ -0,0 +1,259 @@
# Dead Code & Legacy Artifact Cleanup
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Remove 17 dead files, fix broken Makefile, and clean up legacy artifacts that survived the v1→v2 migration.
**Architecture:** Pure deletion + one Makefile rewrite. No functional changes — only removing code/files that are never imported, referenced, or executed.
**Tech Stack:** git rm, pytest
**Source:** Deep audit of all tracked files with grep-verified zero-reference confirmation (2026-04-09).
---
### Task 1: Remove dead scripts
These scripts have zero references anywhere in the codebase.
**Files to delete:**
- `scripts/collect_session.py` — unused SessionEnd hook
- `scripts/generate_user_sync_configs.py` — replaced by DuckDB API
- `scripts/standalone_profiler.py` — replaced by `src/profiler.py`
- `scripts/remote_query.sh` — references non-existent module
- `scripts/update.sh` — calls `src.data_sync` and `docs/data_description.md` (both gone)
- `scripts/setup_views.sh` — depends on deleted `sync_data.sh`
- `scripts/test_sync.sh` — rsync diagnostics for resolved Issue #197
- `scripts/activate_venv.sh` — Docker uses direct venv paths
- `scripts/backfill_gap.sh` — one-time Jira backfill with hardcoded issue ranges
- `scripts/sync_config_template.yaml` — v1 sync config template
- [ ] **Step 1: Delete all dead scripts**
```bash
git rm scripts/collect_session.py \
scripts/generate_user_sync_configs.py \
scripts/standalone_profiler.py \
scripts/remote_query.sh \
scripts/update.sh \
scripts/setup_views.sh \
scripts/test_sync.sh \
scripts/activate_venv.sh \
scripts/backfill_gap.sh \
scripts/sync_config_template.yaml
```
- [ ] **Step 2: Run tests**
Run: `pytest tests/ -q --tb=short`
Expected: All 654 pass (none of these scripts are imported by tests)
- [ ] **Step 3: Commit**
```bash
git commit -m "chore: remove 10 dead scripts from v1 architecture"
```
---
### Task 2: Remove legacy config example and root artifacts
**Files to delete:**
- `config/data_description.md.example` — v1 markdown-based table config, replaced by DuckDB `table_registry`
- `llms.txt` — describes v1 modules (`src/data_sync.py`, `webapp/app.py`, etc.) that don't exist
**Note on `data_description.md`:** `src/profiler.py` still references `docs/data_description.md` (line 95) but handles its absence gracefully (logs warning, skips). The `.example` file is just a template — removing it doesn't affect runtime.
- [ ] **Step 1: Delete files**
```bash
git rm config/data_description.md.example llms.txt
```
- [ ] **Step 2: Run tests**
Run: `pytest tests/ -q --tb=short`
Expected: All pass
- [ ] **Step 3: Commit**
```bash
git commit -m "chore: remove legacy data_description.md.example and outdated llms.txt"
```
---
### Task 3: Remove completed planning docs from dev_docs
These are implementation plans for features that are done.
**Files to delete:**
- `dev_docs/plan-rsync-fix.md` — rsync fix (Issue #197, resolved)
- `dev_docs/plan_parquet_types_fix.md` — parquet type fix (Issues #185-187, resolved)
- `dev_docs/plan-corporate-memory.md` — corporate memory governance (fully implemented)
- [ ] **Step 1: Delete files**
```bash
git rm dev_docs/plan-rsync-fix.md \
dev_docs/plan_parquet_types_fix.md \
dev_docs/plan-corporate-memory.md
```
- [ ] **Step 2: Commit**
```bash
git commit -m "chore: remove completed planning docs (rsync fix, parquet types, corporate memory)"
```
---
### Task 4: Remove unused notification examples
`examples/notifications/` contains 3 Python scripts for a notification feature that was never built.
**Files to delete:**
- `examples/notifications/data_freshness.py`
- `examples/notifications/metric_report.py`
- `examples/notifications/revenue_drop.py`
- [ ] **Step 1: Check if examples/ has anything else**
```bash
git ls-files examples/
```
If only these 3 files, delete the entire directory.
- [ ] **Step 2: Delete**
```bash
git rm -r examples/
```
- [ ] **Step 3: Commit**
```bash
git commit -m "chore: remove unused notification examples (feature not implemented)"
```
---
### Task 5: Fix or replace broken Makefile
The `validate-config` target imports `src.config.Config` which doesn't exist. The Makefile also references `.venv/bin/python` (not Docker).
**File:** `Makefile`
- [ ] **Step 1: Rewrite Makefile**
Replace entire content with a minimal, working version:
```makefile
# Agnes AI Data Analyst — Development Makefile
.PHONY: help test lint dev docker
help:
@echo "Available targets:"
@echo " make test Run test suite"
@echo " make dev Start FastAPI dev server"
@echo " make docker Build and start Docker Compose"
@echo " make lint Run ruff linter (if installed)"
test:
pytest tests/ -v --tb=short
dev:
uvicorn app.main:app --reload
docker:
docker compose up --build
lint:
@ruff check . 2>/dev/null || echo "ruff not installed: pip install ruff"
```
- [ ] **Step 2: Run `make test`**
Run: `make test`
Expected: All tests pass
- [ ] **Step 3: Commit**
```bash
git add Makefile
git commit -m "fix: rewrite Makefile — remove broken validate-config, add working targets"
```
---
### Task 6: Update scripts/README.md
After deleting 10 scripts, the README should reflect what's left.
**File:** `scripts/README.md`
- [ ] **Step 1: Rewrite scripts/README.md**
```markdown
# Scripts
Utility and migration scripts for Agnes AI Data Analyst.
## Active Scripts
| Script | Purpose |
|--------|---------|
| `generate_sample_data.py` | Generate sample data for development/demo |
| `duckdb_manager.py` | DuckDB database management utilities |
| `init.sh` | Initial server setup (install deps, create dirs) |
## Migration Scripts (one-time use)
| Script | Purpose |
|--------|---------|
| `migrate_json_to_duckdb.py` | Migrate v1 JSON state files to DuckDB |
| `migrate_parquets_to_extracts.py` | Migrate v1 parquet layout to extract.duckdb |
| `migrate_registry_to_duckdb.py` | Migrate v1 table registry to DuckDB |
```
- [ ] **Step 2: Commit**
```bash
git add scripts/README.md
git commit -m "docs: update scripts/README.md after dead script cleanup"
```
---
## Execution Order
All tasks are independent except Task 6 (depends on Task 1).
Recommended: run sequentially (Task 1-6) for clean git history.
**Verification after all tasks:**
```bash
# Tests pass
pytest tests/ -v --tb=short
# No broken imports
python -c "from app.main import create_app; print('OK')"
# Makefile works
make test
```
## Summary
| Action | Files | Lines removed (est.) |
|--------|-------|---------------------|
| Dead scripts | 10 files | ~800 |
| Legacy config + llms.txt | 2 files | ~250 |
| Completed plans | 3 files | ~300 |
| Notification examples | 3 files | ~150 |
| Makefile rewrite | 1 file | ~60 (replaced) |
| scripts/README.md | 1 file | updated |
| **Total** | **19 files removed, 2 rewritten** | **~1,500 lines** |

495
docs/superpowers/plans/2026-04-09-deployment-readiness.md Normal file
View file

@ -0,0 +1,495 @@
# Deployment & Multi-Instance Readiness Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Make the platform deployable to N customer instances with minimal manual effort.
**Architecture:** Docker image on GHCR + per-instance config (instance.yaml + .env) + Terraform provisioning. One image, many instances.
**Tech Stack:** Docker, Terraform (GCP), GitHub Actions, Caddy (TLS proxy)
**Source:** Deployment readiness + multi-instance architecture reviews 2026-04-09 (findings C5-C7, I4-I9)
---
## File Map
| File | Responsibility | Tasks |
|------|---------------|-------|
| `config/.env.template` | Complete env var reference | 1 |
| `docker-compose.yml` | Add restart policy, config mount, image ref, Caddy proxy | 2, 3 |
| `docker-compose.prod.yml` | Production override with GHCR image + Caddy | 2, 3 |
| `.github/workflows/deploy.yml` | Image versioning with SHA tag | 4 |
| `infra/main.tf` | Remote state backend, instance.yaml generation | 5 |
| `services/telegram_bot/config.py` | Fix hardcoded paths | 6 |
| `src/profiler.py` | Fix PROFILER_DATA_DIR | 6 |
| `docs/DEPLOYMENT.md` | Update for multi-instance | 7 |
---
### Task 1: Complete .env.template with all env vars
The template lists only 8 of ~15 needed variables.
**Files:**
- Modify: `config/.env.template`
- [ ] **Step 1: Rewrite .env.template**
```bash
# Agnes AI Data Analyst - Environment Variables
# =============================================
# Copy to .env: cp config/.env.template .env
# .env is gitignored - NEVER commit it.
# ── REQUIRED ────────────────────────────────────────
JWT_SECRET_KEY= # python -c "import secrets; print(secrets.token_hex(32))"
SESSION_SECRET= # python -c "import secrets; print(secrets.token_hex(32))"
# ── GOOGLE OAUTH (required for Google login) ────────
# GOOGLE_CLIENT_ID=
# GOOGLE_CLIENT_SECRET=
# ── KEBOOLA (required for Keboola data source) ──────
# KEBOOLA_STORAGE_TOKEN=
# KEBOOLA_STACK_URL=https://connection.keboola.com
# ── BIGQUERY (required for BigQuery data source) ─────
# BIGQUERY_PROJECT=
# BIGQUERY_LOCATION=us
# ── BOOTSTRAP (first deploy only) ───────────────────
# SEED_ADMIN_EMAIL=admin@example.com
# ── EMAIL / SMTP (required for magic link auth) ─────
# SMTP_HOST=smtp.gmail.com
# SMTP_PORT=587
# SMTP_USER=
# SMTP_PASSWORD=
# ── OPTIONAL SERVICES ───────────────────────────────
# TELEGRAM_BOT_TOKEN=
# JIRA_WEBHOOK_SECRET=
# JIRA_API_TOKEN=
# ANTHROPIC_API_KEY=
# LLM_API_KEY=
# ── DESKTOP APP ─────────────────────────────────────
# DESKTOP_JWT_SECRET= # Separate secret for desktop app tokens
# ── DEPLOYMENT ──────────────────────────────────────
# DATA_DIR=/data # Default: /data in Docker, ./data locally
# LOG_LEVEL=info # debug, info, warning, error
# CORS_ORIGINS=http://localhost:3000,http://localhost:8000
```
- [ ] **Step 2: Commit**
```bash
git add config/.env.template
git commit -m "docs: complete .env.template with all 20+ env vars"
```
---
### Task 2: Fix docker-compose for production (I7, I5, I8)
Add restart policy to app, config volume mount, and GHCR image reference.
**Files:**
- Modify: `docker-compose.yml`
- Create: `docker-compose.prod.yml` (production override)
- [ ] **Step 1: Add restart policy and config mount to docker-compose.yml**
In `docker-compose.yml`, add to the `app` service:
```yaml
app:
build: .
command: uvicorn app.main:app --host 0.0.0.0 --port 8000
ports:
- "8000:8000"
volumes:
- data:/data
- ./config:/app/config:ro
env_file: .env
environment:
- DATA_DIR=/data
restart: unless-stopped
healthcheck:
test: ["CMD", "curl", "-sf", "http://localhost:8000/api/health"]
interval: 30s
timeout: 5s
retries: 3
```
Key changes: `restart: unless-stopped` added, `./config:/app/config:ro` volume mount added.
- [ ] **Step 2: Create docker-compose.prod.yml**
```yaml
# Production override — uses pre-built GHCR image instead of local build.
# Usage: docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
services:
app:
image: ghcr.io/keboola/agnes-the-ai-analyst:latest
build: !reset null
scheduler:
image: ghcr.io/keboola/agnes-the-ai-analyst:latest
build: !reset null
extract:
image: ghcr.io/keboola/agnes-the-ai-analyst:latest
build: !reset null
telegram-bot:
image: ghcr.io/keboola/agnes-the-ai-analyst:latest
build: !reset null
ws-gateway:
image: ghcr.io/keboola/agnes-the-ai-analyst:latest
build: !reset null
corporate-memory:
image: ghcr.io/keboola/agnes-the-ai-analyst:latest
build: !reset null
session-collector:
image: ghcr.io/keboola/agnes-the-ai-analyst:latest
build: !reset null
```
- [ ] **Step 3: Commit**
```bash
git add docker-compose.yml docker-compose.prod.yml
git commit -m "feat: add restart policy, config mount, production compose override with GHCR images"
```
---
### Task 3: Add Caddy reverse proxy for TLS (I4)
No HTTPS in Docker Compose — data transits in plaintext.
**Files:**
- Create: `Caddyfile`
- Modify: `docker-compose.yml` (add caddy service)
- [ ] **Step 1: Create Caddyfile**
```
{$DOMAIN:localhost} {
reverse_proxy app:8000
}
```
- [ ] **Step 2: Add Caddy service to docker-compose.yml**
Add to services section:
```yaml
caddy:
image: caddy:2-alpine
ports:
- "80:80"
- "443:443"
volumes:
- ./Caddyfile:/etc/caddy/Caddyfile:ro
- caddy_data:/data
- caddy_config:/config
environment:
- DOMAIN=${DOMAIN:-localhost}
depends_on:
app:
condition: service_healthy
restart: unless-stopped
profiles:
- production
```
Add volumes:
```yaml
volumes:
data:
caddy_data:
caddy_config:
```
- [ ] **Step 3: Update DEPLOYMENT.md**
Add section:
```markdown
### HTTPS with Caddy (production)
Set `DOMAIN=data.yourcompany.com` in `.env`, then:
```bash
docker compose --profile production up -d
```
Caddy automatically provisions Let's Encrypt TLS certificates.
```
- [ ] **Step 4: Commit**
```bash
git add Caddyfile docker-compose.yml docs/DEPLOYMENT.md
git commit -m "feat: add Caddy reverse proxy for automatic HTTPS in production"
```
---
### Task 4: Add Docker image versioning with commit SHA (C7)
Images are only tagged `:latest` — no versioning, no rollback.
**Files:**
- Modify: `.github/workflows/deploy.yml`
- [ ] **Step 1: Update image tagging**
In `.github/workflows/deploy.yml`, replace the build-and-push step:
```yaml
- name: Build and push
uses: docker/build-push-action@v7
with:
push: true
tags: |
ghcr.io/${{ github.repository }}:latest
ghcr.io/${{ github.repository }}:${{ github.sha }}
```
- [ ] **Step 2: Commit**
```bash
git add -f .github/workflows/deploy.yml
git commit -m "feat: tag Docker images with commit SHA for versioning and rollback"
```
---
### Task 5: Add Terraform remote state backend (I6)
Local tfstate blocks multi-operator and multi-instance Terraform.
**Files:**
- Modify: `infra/main.tf`
- Modify: `infra/variables.tf`
- [ ] **Step 1: Add GCS backend to main.tf**
In `infra/main.tf`, inside the `terraform {}` block:
```hcl
terraform {
required_version = ">= 1.5"
backend "gcs" {
bucket = "agnes-terraform-state"
prefix = "instances"
}
required_providers {
google = {
source = "hashicorp/google"
version = "~> 5.0"
}
random = {
source = "hashicorp/random"
version = "~> 3.0"
}
}
}
```
- [ ] **Step 2: Add instance.yaml generation to startup script**
In `infra/main.tf`, in the `startup_script` local, after the `.env` generation:
```bash
echo "=== Creating instance.yaml ==="
cat > "$APP_DIR/config/instance.yaml" << 'YAMLEOF'
instance:
name: "${var.instance_name}"
subtitle: "Data Analytics Platform"
server:
host: "${google_compute_address.data_analyst.address}"
hostname: "${var.domain != "" ? var.domain : google_compute_address.data_analyst.address}"
port: 8000
auth:
allowed_domain: "${var.admin_email != "" ? join("", [split("@", var.admin_email)[1]]) : ""}"
data_source:
type: "${var.keboola_token != "" ? "keboola" : "local"}"
YAMLEOF
sed -i 's/^ //' "$APP_DIR/config/instance.yaml"
```
- [ ] **Step 3: Update repo URL in startup script**
Replace line 73 `git clone https://github.com/padak/tmp_oss.git` with:
```bash
git clone https://github.com/keboola/agnes-the-ai-analyst.git "$APP_DIR"
```
And line 75 `git checkout feature/v2-fastapi-duckdb-docker-cli` with:
```bash
# main branch is default, no checkout needed
```
- [ ] **Step 4: Commit**
```bash
git add infra/main.tf infra/variables.tf
git commit -m "feat: add Terraform GCS remote state, instance.yaml generation, update repo URL"
```
---
### Task 6: Fix hardcoded paths in services (I9)
telegram_bot and profiler use hardcoded `/data/...` paths instead of `DATA_DIR`.
**Files:**
- Modify: `services/telegram_bot/config.py:14`
- Modify: `src/profiler.py:87`
- Modify: `services/telegram_bot/dispatch.py:17`
- [ ] **Step 1: Fix telegram_bot config**
In `services/telegram_bot/config.py`, replace line 14:
```python
NOTIFICATIONS_DIR = os.path.join(os.environ.get("DATA_DIR", "/data"), "notifications")
```
- [ ] **Step 2: Fix profiler**
In `src/profiler.py`, replace line 87:
```python
DATA_DIR = Path(os.environ.get("DATA_DIR", "/data")) / "src_data"
```
Remove `PROFILER_DATA_DIR` reference — use standard `DATA_DIR` like everywhere else.
- [ ] **Step 3: Fix dispatch.py**
In `services/telegram_bot/dispatch.py`, replace line 17:
```python
WS_GATEWAY_SOCKET_PATH = os.environ.get("WS_GATEWAY_SOCKET", "/run/ws-gateway/ws.sock")
```
- [ ] **Step 4: Run tests**
Run: `pytest tests/ -q --tb=short`
Expected: All pass
- [ ] **Step 5: Commit**
```bash
git add services/telegram_bot/config.py services/telegram_bot/dispatch.py src/profiler.py
git commit -m "fix: use DATA_DIR env var everywhere — remove hardcoded /data paths"
```
---
### Task 7: Update DEPLOYMENT.md for multi-instance
Add production deployment with GHCR images, Caddy TLS, and multi-instance guidance.
**Files:**
- Modify: `docs/DEPLOYMENT.md`
- [ ] **Step 1: Add sections to DEPLOYMENT.md**
Add these sections:
**Production with GHCR images:**
```markdown
### Production Deployment (pre-built images)
Instead of building locally, pull from GitHub Container Registry:
```bash
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
```
Pin to a specific version:
```bash
# In docker-compose.prod.yml, change :latest to :COMMIT_SHA
image: ghcr.io/keboola/agnes-the-ai-analyst:abc1234
```
```
**Multi-instance:**
```markdown
## Multi-Instance Deployment
Each customer gets a separate VM with isolated data and config.
1. Copy `infra/terraform.tfvars.example` to `infra/instances/customer-name.tfvars`
2. Fill in customer-specific values
3. Apply: `cd infra && terraform workspace new customer-name && terraform apply -var-file=instances/customer-name.tfvars`
4. SSH in and create `config/instance.yaml` from `config/instance.yaml.example`
5. Start: `docker compose -f docker-compose.yml -f docker-compose.prod.yml --profile production up -d`
6. Bootstrap: `curl -X POST http://IP:8000/auth/bootstrap -d '{"email":"admin@customer.com"}'`
```
**Update/rollback:**
```markdown
## Updating an Instance
```bash
# Pull latest image
docker compose -f docker-compose.yml -f docker-compose.prod.yml pull
# Restart with new image
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
# Rollback to specific version
# Edit docker-compose.prod.yml: change :latest to :PREVIOUS_SHA
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
```
```
- [ ] **Step 2: Commit**
```bash
git add docs/DEPLOYMENT.md
git commit -m "docs: add multi-instance deployment, GHCR images, update/rollback procedures"
```
---
## Execution Order
Sequential recommended (some tasks depend on earlier ones):
1. **Task 1** — .env.template (no deps)
2. **Task 2** — docker-compose fixes (no deps)
3. **Task 3** — Caddy TLS (depends on Task 2)
4. **Task 4** — image versioning (no deps)
5. **Task 5** — Terraform remote state (no deps)
6. **Task 6** — hardcoded paths (no deps)
7. **Task 7** — documentation (depends on all above)
Tasks 1, 2, 4, 5, 6 can run in parallel.
**Verification after all tasks:**
```bash
# Tests still pass
pytest tests/ -v --tb=short
# Docker builds
docker compose build
# Production compose validates
docker compose -f docker-compose.yml -f docker-compose.prod.yml config
```

187
docs/superpowers/plans/2026-04-09-final-polish.md Normal file
View file

@ -0,0 +1,187 @@
# Final Polish — Remaining P2 Fixes
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Clean up all remaining P2 issues from reviews + port 3 fixes from padak/tmp_oss + update stale docs.
**Architecture:** Small, independent fixes grouped by area. No architectural changes.
**Tech Stack:** Python 3.13, FastAPI, DuckDB, pytest
---
### Task 1: Fix argon2 error handling and imports (3 files)
Bare `except Exception` swallows non-auth errors. argon2 imported inside function body.
**Files:**
- Modify: `app/auth/router.py:51-56`
- Modify: `app/auth/providers/password.py`
**Changes:**
1. In `app/auth/router.py`, add at top:
```python
from argon2 import PasswordHasher
from argon2.exceptions import VerifyMismatchError
```
Replace lines 51-56 (the try/except inside the password_hash check):
```python
try:
ph = PasswordHasher()
ph.verify(user["password_hash"], request.password)
except VerifyMismatchError:
raise HTTPException(status_code=401, detail="Invalid password")
except Exception:
import logging
logging.getLogger(__name__).exception("Password verification error for %s", request.email)
raise HTTPException(status_code=500, detail="Authentication error")
```
2. In `app/auth/providers/password.py`, apply the same pattern — top-level import, catch `VerifyMismatchError` specifically.
3. Run: `pytest tests/test_auth_providers.py tests/test_security.py -v`
4. Commit: `fix: specific argon2 exception handling, top-level imports`
---
### Task 2: Fix duplicate import and raw SQL (2 files)
**Files:**
- Modify: `app/api/upload.py:7-8` — remove duplicate `from pathlib import Path as _Path`, use `Path` everywhere
- Modify: `app/api/memory.py:236` — route admin_edit through KnowledgeRepository instead of raw SQL
**Changes:**
1. In `upload.py`, remove line 8 (`from pathlib import Path as _Path`), replace all `_Path` usages with `Path`.
2. In `memory.py`, find the `admin_edit` function. Replace raw `conn.execute(f"UPDATE knowledge_items SET {set_clause}...")` with a call through the repository. Check if `KnowledgeRepository` has an `update` method; if not, add one.
3. Run: `pytest tests/test_api_complete.py -v`
4. Commit: `fix: remove duplicate import, route admin_edit through repository`
---
### Task 3: Fix Google OAuth connection management
**File:** `app/auth/providers/google.py:79-86`
Manual `get_system_db()` / `conn.close()` instead of using DI. If exception occurs between open and close, connection leaks.
**Changes:**
Wrap in try/finally:
```python
conn = get_system_db()
try:
repo = UserRepository(conn)
# ... existing user lookup/creation logic ...
finally:
conn.close()
```
Run: `pytest tests/test_auth_providers.py -v`
Commit: `fix: wrap Google OAuth DB access in try/finally`
---
### Task 4: Add auth event audit logging
Login, token creation, bootstrap — no audit trail.
**File:** `app/auth/router.py`
**Changes:**
After successful token creation (line ~58) and bootstrap (line ~104), add:
```python
from src.repositories.audit import AuditRepository
try:
audit_conn = get_system_db()
AuditRepository(audit_conn).log(
user_id=user["id"], action="token_created", resource="auth",
params={"email": user["email"]},
)
audit_conn.close()
except Exception:
pass # Audit failure should not block auth
```
Check what `AuditRepository.log()` signature looks like first — read `src/repositories/audit.py`.
Run: `pytest tests/test_auth_providers.py -v`
Commit: `feat: add audit logging for auth events (token, bootstrap)`
---
### Task 5: Port profiler union_by_name fix from padak
`src/profiler.py` crashes on partitioned tables when parquet files have schema evolution.
**File:** `src/profiler.py`
**Changes:**
Find all `read_parquet(` calls in profiler.py. Add `union_by_name=true` parameter. For example:
```python
# Before:
conn.execute(f"SELECT * FROM read_parquet('{path}')")
# After:
conn.execute(f"SELECT * FROM read_parquet('{path}', union_by_name=true)")
```
Run: `pytest tests/test_auto_profiling.py -v`
Commit: `fix: add union_by_name=true to profiler parquet reads (schema evolution support)`
---
### Task 6: Port strip_html fix for enricher from padak
`connectors/openmetadata/enricher.py` passes raw HTML to templates.
**File:** `connectors/openmetadata/enricher.py`
**Changes:**
1. Import strip_html from transformer:
```python
from connectors.openmetadata.transformer import strip_html
```
2. Find where table/column descriptions are set in `_parse_table_response()` or similar. Apply `strip_html()` to description fields before returning.
Run: `pytest tests/test_openmetadata_enricher.py -v`
Commit: `fix: strip HTML from catalog descriptions in enricher`
---
### Task 7: Update stale docs (3 files)
**Files:**
- `docs/CONFIGURATION.md` — references SendGrid, WEBAPP_SECRET_KEY, old patterns
- `dev_docs/disaster-recovery.md` — describes v1 architecture (systemd, nginx, /home)
- `dev_docs/server.md` — partially stale
**Changes:**
1. `docs/CONFIGURATION.md`: Read it, remove all Flask/SendGrid/WEBAPP_SECRET_KEY references. Update to match current env vars (JWT_SECRET_KEY, SESSION_SECRET). Reference `.env.template` for the full list.
2. `dev_docs/disaster-recovery.md`: Read it. If mostly v1, either rewrite for Docker-based backup (GCP disk snapshots + DuckDB export) or delete and add a brief backup section to DEPLOYMENT.md.
3. `dev_docs/server.md`: Read it. Remove rsync/SSH/systemd sections. Keep any still-relevant Docker deployment info.
Commit: `docs: update CONFIGURATION.md, disaster-recovery, server.md for v2 architecture`
---
## Execution Order
Tasks 1-6 are independent (different files). Task 7 is docs-only.
All can run in parallel.
**Verification:**
```bash
pytest tests/ -v --tb=short # All 654+ tests pass
```

623
docs/superpowers/plans/2026-04-09-security-fixes.md Normal file
View file

@ -0,0 +1,623 @@
# Security Fixes for Production Deployment
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Fix all critical and important security findings before deploying to paying customers.
**Architecture:** Targeted fixes to auth, RBAC, query endpoint, script sandbox, and upload endpoints. No architectural changes — just closing specific holes identified by security review.
**Tech Stack:** Python 3.13, FastAPI, DuckDB, argon2-cffi, PyJWT
**Source:** Security posture review 2026-04-09 (findings C1-C3, I1-I5, I10-I11)
---
## File Map
| File | Responsibility | Tasks |
|------|---------------|-------|
| `app/auth/router.py` | Token endpoint auth | 1 |
| `app/web/router.py` | Web UI route guards | 2 |
| `app/api/query.py` | SQL query blocklist | 3 |
| `app/api/scripts.py` | Script execution RBAC | 4 |
| `app/api/catalog.py` | Catalog profile access control | 5 |
| `app/api/upload.py` | Upload path leak fix | 6 |
| `app/auth/providers/google.py` | Cookie secure flag | 7 |
| `app/instance_config.py` | Instance name YAML path fix | 8 |
---
### Task 1: Block /auth/token for OAuth-only users (C1)
Users without `password_hash` (OAuth-only) can get a JWT by just sending their email. This is an account takeover vulnerability.
**Files:**
- Modify: `app/auth/router.py:47-56`
- Test: `tests/test_auth_providers.py`
- [ ] **Step 1: Write the failing test**
```python
# In tests/test_auth_providers.py, add to TestTokenEndpoint:
def test_token_rejected_for_oauth_only_user(self, client, e2e_env):
"""OAuth-only users (no password_hash) cannot get token via /auth/token."""
from src.db import get_system_db
from src.repositories.users import UserRepository
conn = get_system_db()
repo = UserRepository(conn)
# Create user without password (simulates Google OAuth user)
repo.create(id="oauth-user", email="oauth@test.com", role="analyst")
conn.close()
resp = client.post("/auth/token", json={"email": "oauth@test.com"})
assert resp.status_code == 401
assert "password" in resp.json()["detail"].lower() or "provider" in resp.json()["detail"].lower()
```
- [ ] **Step 2: Run test to verify it fails**
Run: `pytest tests/test_auth_providers.py::TestTokenEndpoint::test_token_rejected_for_oauth_only_user -v`
Expected: FAIL — currently returns 200
- [ ] **Step 3: Fix the auth logic**
In `app/auth/router.py`, replace lines 47-56 with:
```python
# Require authentication proof
if user.get("password_hash"):
# User has password — require and verify it
if not request.password:
raise HTTPException(status_code=401, detail="Password required")
try:
from argon2 import PasswordHasher
ph = PasswordHasher()
ph.verify(user["password_hash"], request.password)
except Exception:
raise HTTPException(status_code=401, detail="Invalid password")
else:
# No password set — user must use their auth provider (Google, magic link)
raise HTTPException(
status_code=401,
detail="This account uses external authentication. Please log in via your configured provider.",
)
```
Also update the docstring on line 41:
```python
"""Issue a JWT token. Requires password for password-protected accounts.
OAuth-only accounts must use their auth provider instead."""
```
- [ ] **Step 4: Run test to verify it passes**
Run: `pytest tests/test_auth_providers.py -v`
Expected: All pass
- [ ] **Step 5: Verify bootstrap still works**
Run: `pytest tests/test_bootstrap.py -v`
Expected: All pass (bootstrap creates user WITH password or returns token directly)
- [ ] **Step 6: Commit**
```bash
git add app/auth/router.py tests/test_auth_providers.py
git commit -m "fix: block /auth/token for OAuth-only users — require password or external provider"
```
---
### Task 2: Add role checks to web admin pages (C2)
`/admin/tables`, `/admin/permissions`, `/corporate-memory/admin` are accessible to any authenticated user.
**Files:**
- Modify: `app/web/router.py:431-452,388-394`
- Test: `tests/test_web_ui.py`
- [ ] **Step 1: Write the failing tests**
```python
# In tests/test_web_ui.py, add:
@pytest.fixture
def analyst_cookie(web_client, tmp_path, monkeypatch):
"""Create analyst user (non-admin) and return cookie."""
from src.db import get_system_db
from src.repositories.users import UserRepository
conn = get_system_db()
UserRepository(conn).create(id="analyst1", email="analyst@test.com", name="Analyst", role="analyst")
conn.close()
resp = web_client.post("/auth/token", json={"email": "analyst@test.com"})
# analyst has no password_hash — need to give them one first
# Actually: after Task 1, this will fail. Create with password instead:
from argon2 import PasswordHasher
from src.db import get_system_db as gsdb
c = gsdb()
c.execute("UPDATE users SET password_hash = ? WHERE id = ?",
[PasswordHasher().hash("testpass"), "analyst1"])
c.close()
resp = web_client.post("/auth/token", json={"email": "analyst@test.com", "password": "testpass"})
assert resp.status_code == 200
return {"access_token": resp.json()["access_token"]}
class TestWebUIRBAC:
def test_admin_tables_requires_admin(self, web_client, analyst_cookie):
resp = web_client.get("/admin/tables", cookies=analyst_cookie)
assert resp.status_code == 403
def test_admin_permissions_requires_admin(self, web_client, analyst_cookie):
resp = web_client.get("/admin/permissions", cookies=analyst_cookie)
assert resp.status_code == 403
def test_corporate_memory_admin_requires_km_admin(self, web_client, analyst_cookie):
resp = web_client.get("/corporate-memory/admin", cookies=analyst_cookie)
assert resp.status_code == 403
def test_admin_can_access_admin_tables(self, web_client, admin_cookie):
resp = web_client.get("/admin/tables", cookies=admin_cookie)
assert resp.status_code == 200
def test_admin_can_access_admin_permissions(self, web_client, admin_cookie):
resp = web_client.get("/admin/permissions", cookies=admin_cookie)
assert resp.status_code == 200
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `pytest tests/test_web_ui.py::TestWebUIRBAC -v`
Expected: analyst can access admin pages (403 expected, gets 200)
- [ ] **Step 3: Add role checks to web routes**
In `app/web/router.py`, add import at top:
```python
from src.rbac import Role
from app.auth.dependencies import require_role
```
Replace line 434 (`user: dict = Depends(get_current_user)`) in `admin_tables`:
```python
user: dict = Depends(require_role(Role.ADMIN)),
```
Replace line 448 in `admin_permissions_page`:
```python
user: dict = Depends(require_role(Role.ADMIN)),
```
Replace line 391 in `corporate_memory_admin`:
```python
user: dict = Depends(require_role(Role.KM_ADMIN)),
```
- [ ] **Step 4: Run tests**
Run: `pytest tests/test_web_ui.py -v`
Expected: All pass
- [ ] **Step 5: Commit**
```bash
git add app/web/router.py tests/test_web_ui.py
git commit -m "fix: require admin/km_admin role for web admin pages"
```
---
### Task 3: Expand SQL query blocklist with DuckDB metadata (C3)
The query endpoint doesn't block `information_schema`, `duckdb_tables()`, `duckdb_columns()`, relative paths, or `pragma_` functions.
**Files:**
- Modify: `app/api/query.py:40-54`
- Test: `tests/test_security.py`
- [ ] **Step 1: Write the failing tests**
```python
# In tests/test_security.py, add to TestQuerySecurity:
def test_blocks_information_schema(self, client, auth_headers):
resp = client.post("/api/query", json={"sql": "SELECT * FROM information_schema.tables"}, headers=auth_headers)
assert resp.status_code == 400
def test_blocks_duckdb_tables(self, client, auth_headers):
resp = client.post("/api/query", json={"sql": "SELECT * FROM duckdb_tables()"}, headers=auth_headers)
assert resp.status_code == 400
def test_blocks_duckdb_columns(self, client, auth_headers):
resp = client.post("/api/query", json={"sql": "SELECT * FROM duckdb_columns()"}, headers=auth_headers)
assert resp.status_code == 400
def test_blocks_duckdb_databases(self, client, auth_headers):
resp = client.post("/api/query", json={"sql": "SELECT * FROM duckdb_databases()"}, headers=auth_headers)
assert resp.status_code == 400
def test_blocks_relative_path(self, client, auth_headers):
resp = client.post("/api/query", json={"sql": "SELECT * FROM '../secret.parquet'"}, headers=auth_headers)
assert resp.status_code == 400
def test_blocks_pragma_table_info(self, client, auth_headers):
resp = client.post("/api/query", json={"sql": "SELECT * FROM pragma_table_info('users')"}, headers=auth_headers)
assert resp.status_code == 400
```
- [ ] **Step 2: Run to verify failures**
Run: `pytest tests/test_security.py::TestQuerySecurity -v`
Expected: New tests FAIL
- [ ] **Step 3: Expand the blocklist**
In `app/api/query.py`, replace the `blocked` list (lines 40-54):
```python
blocked = [
# DDL/DML
"drop ", "delete ", "insert ", "update ", "alter ", "create ",
"copy ", "attach ", "detach ", "load ", "install ",
"export ", "import ", "pragma ", "call ",
# File access functions
"read_csv", "read_json", "read_parquet", "read_text",
"write_csv", "write_parquet", "read_blob", "read_ndjson",
"parquet_scan", "parquet_metadata", "parquet_schema",
"json_scan", "csv_scan",
"query_table", "iceberg_scan", "delta_scan",
"glob(", "list_files",
# URL/path schemes
"'/", '"/','http://', 'https://', 's3://', 'gcs://',
"'../", '"../',
# DuckDB metadata (leaks schema info regardless of RBAC)
"information_schema", "duckdb_tables", "duckdb_columns",
"duckdb_databases", "duckdb_settings", "duckdb_functions",
"duckdb_views", "duckdb_indexes", "duckdb_schemas",
"pragma_table_info", "pragma_storage_info",
# Multiple statements
";",
]
```
- [ ] **Step 4: Run tests**
Run: `pytest tests/test_security.py::TestQuerySecurity -v`
Expected: All pass
- [ ] **Step 5: Commit**
```bash
git add app/api/query.py tests/test_security.py
git commit -m "fix: block DuckDB metadata functions and relative paths in query endpoint"
```
---
### Task 4: Restrict script execution to analyst role (I2/I4)
Any authenticated user (including viewers) can deploy and execute scripts.
**Files:**
- Modify: `app/api/scripts.py:53-56,72-73,83-84`
- Test: `tests/test_security.py`
- [ ] **Step 1: Write the failing test**
```python
# In tests/test_security.py, add:
class TestScriptRBAC:
def test_viewer_cannot_run_scripts(self, client):
"""Viewers should not be able to execute scripts."""
from src.db import get_system_db
from src.repositories.users import UserRepository
from app.auth.jwt import create_access_token
conn = get_system_db()
UserRepository(conn).create(id="viewer1", email="viewer@test.com", role="viewer")
conn.close()
token = create_access_token(user_id="viewer1", email="viewer@test.com", role="viewer")
headers = {"Authorization": f"Bearer {token}"}
resp = client.post("/api/scripts/run", json={
"name": "test", "source": "print('hi')"
}, headers=headers)
assert resp.status_code == 403
def test_viewer_cannot_deploy_scripts(self, client):
from src.db import get_system_db
from src.repositories.users import UserRepository
from app.auth.jwt import create_access_token
conn = get_system_db()
try:
UserRepository(conn).create(id="viewer2", email="viewer2@test.com", role="viewer")
except Exception:
pass
conn.close()
token = create_access_token(user_id="viewer2", email="viewer2@test.com", role="viewer")
headers = {"Authorization": f"Bearer {token}"}
resp = client.post("/api/scripts/deploy", json={
"name": "test", "source": "print('hi')", "schedule": ""
}, headers=headers)
assert resp.status_code == 403
```
- [ ] **Step 2: Run to verify failures**
Run: `pytest tests/test_security.py::TestScriptRBAC -v`
Expected: FAIL — viewers get 200
- [ ] **Step 3: Add role requirements**
In `app/api/scripts.py`, add import:
```python
from app.auth.dependencies import require_role
from src.rbac import Role
```
Replace `get_current_user` with `require_role(Role.ANALYST)` on these endpoints:
- `deploy_script` (line 56): `user: dict = Depends(require_role(Role.ANALYST)),`
- `run_ad_hoc` (line 73): `user: dict = Depends(require_role(Role.ANALYST)),`
- `run_deployed` (line 84): `user: dict = Depends(require_role(Role.ANALYST)),`
- `list_scripts` (line 46): keep as `get_current_user` (read-only, safe for all)
- `undeploy_script` (line 101): `user: dict = Depends(require_role(Role.ADMIN)),`
- [ ] **Step 4: Run tests**
Run: `pytest tests/test_security.py tests/test_api_scripts.py -v`
Expected: All pass
- [ ] **Step 5: Commit**
```bash
git add app/api/scripts.py tests/test_security.py
git commit -m "fix: restrict script deploy/execute to analyst role, undeploy to admin"
```
---
### Task 5: Add access control to catalog profile endpoints (I5)
`/api/catalog/profile/{table_name}` returns profile data without checking table access.
**Files:**
- Modify: `app/api/catalog.py:18-39`
- Test: `tests/test_access_control.py`
- [ ] **Step 1: Write the failing test**
```python
# In tests/test_access_control.py, add to TestPrivateTablesRestricted or new class:
class TestCatalogProfileAccessControl:
def test_profile_denied_for_private_table(self, client, e2e_env):
"""Analyst without explicit access should not see profile of private table."""
# Assumes 'private_table' is registered as private in e2e_env
# and the test user doesn't have access
from app.auth.jwt import create_access_token
token = create_access_token(user_id="analyst-no-access", email="noaccess@test.com", role="analyst")
headers = {"Authorization": f"Bearer {token}"}
resp = client.get("/api/catalog/profile/private_table", headers=headers)
assert resp.status_code == 403
```
- [ ] **Step 2: Run to verify failure**
- [ ] **Step 3: Add access check**
In `app/api/catalog.py`, add import:
```python
from src.rbac import can_access_table
```
In `get_table_profile` (line 18), after `user: dict = Depends(get_current_user)`, add:
```python
# Check table-level access
if not can_access_table(user, table_name, conn):
raise HTTPException(status_code=403, detail=f"Access denied to table '{table_name}'")
```
Add the same check to the `/profile/{table_name}/refresh` endpoint if it exists.
- [ ] **Step 4: Run tests**
Run: `pytest tests/test_access_control.py -v`
Expected: All pass
- [ ] **Step 5: Commit**
```bash
git add app/api/catalog.py tests/test_access_control.py
git commit -m "fix: add per-table access control to catalog profile endpoints"
```
---
### Task 6: Stop leaking internal file paths in upload responses (I10)
Upload endpoints return `"path": str(target)` exposing server filesystem structure.
**Files:**
- Modify: `app/api/upload.py:37,59`
- Test: `tests/test_api_complete.py`
- [ ] **Step 1: Write the failing test**
```python
# In tests/test_api_complete.py, add to TestUpload:
def test_upload_does_not_leak_absolute_path(self, client, admin_headers):
"""Upload response should not contain absolute filesystem paths."""
import io
resp = client.post(
"/api/upload/session/test-session",
files={"file": ("test.txt", io.BytesIO(b"hello"), "text/plain")},
headers=admin_headers,
)
assert resp.status_code == 200
data = resp.json()
assert not data.get("path", "").startswith("/"), "Response should not leak absolute path"
```
- [ ] **Step 2: Run to verify failure**
Run: `pytest tests/test_api_complete.py::TestUpload::test_upload_does_not_leak_absolute_path -v`
Expected: FAIL — returns `/data/user_sessions/...`
- [ ] **Step 3: Fix upload responses**
In `app/api/upload.py`, replace `"path": str(target)` with `"filename": filename` in both endpoints:
Line 37:
```python
return {"status": "ok", "filename": filename, "size": len(content)}
```
Line 59:
```python
return {"status": "ok", "filename": filename, "size": len(content)}
```
- [ ] **Step 4: Run tests**
Run: `pytest tests/test_api_complete.py -v`
Expected: All pass
- [ ] **Step 5: Commit**
```bash
git add app/api/upload.py tests/test_api_complete.py
git commit -m "fix: return filename instead of absolute path in upload responses"
```
---
### Task 7: Force secure cookie flag in production (I11)
Google OAuth callback sets `secure=True` only when the request is HTTPS. Behind a TLS-terminating proxy, the app sees HTTP.
**Files:**
- Modify: `app/auth/providers/google.py:92-98`
- [ ] **Step 1: Fix the cookie setting**
In `app/auth/providers/google.py`, replace lines 92-98:
```python
is_production = os.environ.get("TESTING", "").lower() not in ("1", "true")
response = RedirectResponse(url="/dashboard", status_code=302)
response.set_cookie(
key="access_token", value=jwt_token,
httponly=True, max_age=86400, samesite="lax",
secure=is_production, # Always secure in production (behind TLS proxy)
)
```
Note: `max_age` reduced from `86400 * 30` (30 days) to `86400` (1 day) to match JWT expiry.
- [ ] **Step 2: Run auth tests**
Run: `pytest tests/test_auth_providers.py -v`
Expected: All pass
- [ ] **Step 3: Commit**
```bash
git add app/auth/providers/google.py
git commit -m "fix: force secure cookie flag in production, align cookie max_age with JWT expiry"
```
---
### Task 8: Fix instance_config YAML path for instance name (C4)
`get_instance_name()` reads flat key `instance_name` but YAML structure is `instance.name`.
**Files:**
- Modify: `app/instance_config.py:48-53`
- Test: `tests/test_instance_config.py`
- [ ] **Step 1: Write the failing test**
```python
# In tests/test_instance_config.py, add:
def test_reads_nested_instance_name(self, tmp_path, monkeypatch):
"""get_instance_name should read instance.name from YAML, not flat instance_name."""
monkeypatch.setenv("DATA_DIR", str(tmp_path))
monkeypatch.setenv("TESTING", "1")
monkeypatch.setenv("JWT_SECRET_KEY", "test-secret-key-min-32-characters!!")
config_dir = tmp_path / "config"
config_dir.mkdir(exist_ok=True)
(config_dir / "instance.yaml").write_text(
"instance:\n name: Acme Analytics\n subtitle: Data Team\n"
)
import importlib
import app.instance_config as mod
importlib.reload(mod)
assert mod.get_instance_name() == "Acme Analytics"
assert mod.get_instance_subtitle() == "Data Team"
```
- [ ] **Step 2: Run to verify failure**
Run: `pytest tests/test_instance_config.py::TestInstanceConfig::test_reads_nested_instance_name -v`
Expected: FAIL — returns "AI Data Analyst" instead of "Acme Analytics"
- [ ] **Step 3: Fix the accessor functions**
In `app/instance_config.py`, replace lines 48-53:
```python
def get_instance_name() -> str:
return get_value("instance", "name", default="AI Data Analyst")
def get_instance_subtitle() -> str:
return get_value("instance", "subtitle", default="")
```
- [ ] **Step 4: Run tests**
Run: `pytest tests/test_instance_config.py -v`
Expected: All pass
- [ ] **Step 5: Commit**
```bash
git add app/instance_config.py tests/test_instance_config.py
git commit -m "fix: get_instance_name reads nested instance.name from YAML"
```
---
## Execution Order
Tasks are independent and can run in parallel (different files). Recommended order by impact:
1. **Task 1** (C1) — account takeover via /auth/token
2. **Task 2** (C2) — admin pages exposed
3. **Task 3** (C3) — SQL metadata leaks
4. **Task 4** (I4) — script execution RBAC
5. **Task 5** (I5) — catalog profile access control
6. **Task 8** (C4) — instance name config
7. **Task 6** (I10) — upload path leak
8. **Task 7** (I11) — cookie secure flag
**Verification after all tasks:**
```bash
pytest tests/ -v --tb=short # All 650+ tests pass
```

848
docs/superpowers/plans/2026-04-21-hackathon-dry-run.md Normal file
View file

@ -0,0 +1,848 @@
# Hackathon E2E Dry-Run Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Validate the full developer→dev-VM→merge→prod flow end-to-end the day before a multi-developer hackathon, so any broken link is found and fixed before participants arrive.
**Architecture:** This is an operational dry-run, not a code feature. The executing agent pushes a throwaway feature branch to the public repo, verifies that CI produces a per-branch Docker image tag on GHCR, switches the shared `agnes-dev` VM onto that tag via the existing auto-upgrade cron, verifies that the CI test gate blocks a deliberately-broken PR from reaching `:stable`, and produces a helper script + report. The plan is **strictly non-destructive for prod** — prod-pinning (point 6 of the original outline) is explicitly out of scope and left to the user.
**Tech Stack:** Bash / `gcloud` / `gh` / `git` / `docker` / `curl` / Python (`pytest`) / Terraform (plan only, no apply). No app code changes.
---
## Out of Scope (do NOT do)
- Any `terraform apply` against real infrastructure. TF `plan` is allowed; TF `apply` is forbidden.
- Pinning `prod_instance.image_tag` in `agnes-infra-keboola`. User will do this themselves after the dry-run succeeds.
- Rotating admin passwords, Keboola tokens, or JWT secrets.
- Modifying `main` branch of any repo. All changes happen on throwaway branches, which are deleted at the end.
- Creating new GCP resources (VMs, disks, IPs, secrets, SAs).
If any step would require doing one of the above, **STOP and ask the user**.
---
## Prerequisites
Before starting, the executing agent MUST verify all of the following. If any fails, abort and report which prerequisite is missing — do NOT try to fix it.
- [ ] **Working directory** is the `tmp_oss` checkout at `/Users/zdeneksrotyr/Library/Mobile Documents/com~apple~CloudDocs/Sources/VsCode/component_factory/tmp_oss`. Current branch can be anything; the plan will create a new branch.
- [ ] **`gh auth status`** shows authenticated, with `workflow` scope. Run:
```bash
gh auth status 2>&1 | grep -E "(Logged in|Token scopes)"
```
Expected: line containing `Logged in to github.com` and a line listing scopes that include `workflow`. If `workflow` scope is missing, abort with message: `Run: gh auth refresh -h github.com -s workflow`.
- [ ] **`gcloud` authenticated** to project `kids-ai-data-analysis`. Run:
```bash
gcloud config get-value project
gcloud auth list --filter=status:ACTIVE --format="value(account)"
```
Expected: project is `kids-ai-data-analysis`, at least one active account. If not, abort with message: `Run: gcloud config set project kids-ai-data-analysis && gcloud auth login`.
- [ ] **SSH to `agnes-dev` works** (OS Login). Run:
```bash
gcloud compute ssh agnes-dev --zone=europe-west1-b --command="echo ok" --quiet
```
Expected: output contains `ok`. First connection may take ~20s while OS Login provisions. If fails with permission error, abort with message: `User needs compute.osLogin role on agnes-dev VM`.
- [ ] **`docker` CLI available** locally (for `docker manifest inspect`). Run: `docker --version`. Expected: version output. If missing, abort.
- [ ] **Public GHCR pull works**. Run:
```bash
docker manifest inspect ghcr.io/keboola/agnes-the-ai-analyst:stable > /dev/null && echo ok
```
Expected: `ok`. If fails, abort — something is wrong with public image visibility.
- [ ] **Clone of `agnes-infra-keboola` exists or can be cloned** at `/tmp/agnes-infra-keboola`. Run:
```bash
if [ ! -d /tmp/agnes-infra-keboola ]; then
gh repo clone keboola/agnes-infra-keboola /tmp/agnes-infra-keboola
fi
cd /tmp/agnes-infra-keboola && git status --short
```
Expected: clone succeeds, `git status` is clean. If clone fails, skip Task 4 (TF plan verification) and note it in the final report.
**Gate:** All 7 prerequisite checks pass, OR the agent has clearly reported which ones failed and reduced scope accordingly. Only then proceed to Task 1.
---
## Task 1: Baseline Snapshot
**Purpose:** Record the current state of both VMs and the TF outputs so the agent can detect drift at the end and prove it left everything as it found it.
**Files:**
- Create: `/tmp/dryrun-baseline/prod-health.json`
- Create: `/tmp/dryrun-baseline/dev-health.json`
- Create: `/tmp/dryrun-baseline/prod-image.txt`
- Create: `/tmp/dryrun-baseline/dev-image.txt`
- Create: `/tmp/dryrun-baseline/dev-env.txt`
- [ ] **Step 1.1: Create baseline directory**
```bash
mkdir -p /tmp/dryrun-baseline
```
- [ ] **Step 1.2: Capture prod health**
```bash
curl -sf --max-time 10 http://34.77.102.61:8000/api/health > /tmp/dryrun-baseline/prod-health.json
cat /tmp/dryrun-baseline/prod-health.json | python3 -m json.tool
```
Expected: JSON with `"status"` field equal to `"healthy"` or `"degraded"`. If `"unhealthy"` or curl times out, abort with message: `Prod is not in acceptable baseline state — investigate before dry-run`.
- [ ] **Step 1.3: Capture dev health**
```bash
curl -sf --max-time 10 http://34.77.94.14:8000/api/health > /tmp/dryrun-baseline/dev-health.json
cat /tmp/dryrun-baseline/dev-health.json | python3 -m json.tool
```
Expected: JSON with `"status"` in `{healthy, degraded}`. Same abort condition as 1.2.
- [ ] **Step 1.4: Capture current image tags on both VMs**
```bash
gcloud compute ssh agnes-prod --zone=europe-west1-b --quiet --command \
"docker inspect \$(docker ps -qf name=app) --format '{{.Config.Image}}'" \
> /tmp/dryrun-baseline/prod-image.txt
gcloud compute ssh agnes-dev --zone=europe-west1-b --quiet --command \
"docker inspect \$(docker ps -qf name=app) --format '{{.Config.Image}}'" \
> /tmp/dryrun-baseline/dev-image.txt
cat /tmp/dryrun-baseline/prod-image.txt /tmp/dryrun-baseline/dev-image.txt
```
Expected: each file contains exactly one line like `ghcr.io/keboola/agnes-the-ai-analyst:stable` or `:stable-2026.04.XX`. Non-empty.
- [ ] **Step 1.5: Capture `agnes-dev` `.env` AGNES_TAG line**
```bash
gcloud compute ssh agnes-dev --zone=europe-west1-b --quiet --command \
"sudo grep -E '^AGNES_TAG=' /data/.env || echo 'AGNES_TAG_NOT_SET'" \
> /tmp/dryrun-baseline/dev-env.txt
cat /tmp/dryrun-baseline/dev-env.txt
```
Expected: output is `AGNES_TAG=dev` or similar. Record exact value for restoration in Task 6. If `AGNES_TAG_NOT_SET`, abort — the VM is in an unknown config state.
- [ ] **Step 1.6: Record baseline to report buffer**
Append to a running report at `/tmp/dryrun-report.md` (create if not exists):
```bash
cat > /tmp/dryrun-report.md <<EOF
# Hackathon Dry-Run Report
**Run at:** $(date -u +"%Y-%m-%dT%H:%M:%SZ")
## Baseline (Task 1)
- Prod health status: $(jq -r '.status' /tmp/dryrun-baseline/prod-health.json)
- Dev health status: $(jq -r '.status' /tmp/dryrun-baseline/dev-health.json)
- Prod image: $(cat /tmp/dryrun-baseline/prod-image.txt)
- Dev image: $(cat /tmp/dryrun-baseline/dev-image.txt)
- Dev AGNES_TAG: $(cat /tmp/dryrun-baseline/dev-env.txt)
EOF
cat /tmp/dryrun-report.md
```
Expected: report file exists, all fields populated (no empty values).
**Task 1 gate:** baseline directory has 5 non-empty files, report has 5 non-empty bullet lines. Proceed.
---
## Task 2: Verify Per-Branch GHCR Build
**Purpose:** Push a throwaway feature branch to the public repo, wait for the release workflow, and confirm that the per-branch `:dev-<slug>` tag appears on GHCR.
**Files:**
- Create (throwaway): branch `feature/hack-dryrun-<timestamp>` in `tmp_oss` + one trivial commit touching `docs/QUICKSTART.md`
**Branch naming:** the agent MUST use `feature/hack-dryrun-<epoch>` (e.g. `feature/hack-dryrun-1745254321`) so the slug is unique per run and cleanup is deterministic.
- [ ] **Step 2.1: Compute branch name and expected slug**
Per `.github/workflows/release.yml:92-98` logic: strip `feature/` prefix, sanitise `[^a-zA-Z0-9-]` to `-`, lowercase, cut 50 chars.
```bash
EPOCH=$(date +%s)
BRANCH="feature/hack-dryrun-${EPOCH}"
SLUG=$(echo "$BRANCH" | sed 's|^feature/||' | sed 's|[^a-zA-Z0-9-]|-|g' | tr '[:upper:]' '[:lower:]' | cut -c1-50)
echo "BRANCH=$BRANCH"
echo "SLUG=$SLUG"
echo "EXPECTED_TAG=ghcr.io/keboola/agnes-the-ai-analyst:dev-$SLUG"
# Persist for later steps
echo "$BRANCH" > /tmp/dryrun-baseline/branch-name.txt
echo "$SLUG" > /tmp/dryrun-baseline/slug.txt
```
Expected: BRANCH like `feature/hack-dryrun-1745254321`, SLUG like `hack-dryrun-1745254321`. Persisted.
- [ ] **Step 2.2: Create branch with trivial commit**
```bash
cd "/Users/zdeneksrotyr/Library/Mobile Documents/com~apple~CloudDocs/Sources/VsCode/component_factory/tmp_oss"
# Save current branch so we can return
git rev-parse --abbrev-ref HEAD > /tmp/dryrun-baseline/starting-branch.txt
BRANCH=$(cat /tmp/dryrun-baseline/branch-name.txt)
git checkout -b "$BRANCH"
echo "<!-- dryrun $(date -u +%FT%TZ) -->" >> docs/QUICKSTART.md
git add docs/QUICKSTART.md
git commit -m "dryrun: verify per-branch GHCR tag"
git push -u origin "$BRANCH"
```
Expected: branch created, one commit, push succeeds with upstream tracking. If push is rejected (e.g. protection), abort.
- [ ] **Step 2.3: Wait for release workflow to complete**
```bash
cd "/Users/zdeneksrotyr/Library/Mobile Documents/com~apple~CloudDocs/Sources/VsCode/component_factory/tmp_oss"
BRANCH=$(cat /tmp/dryrun-baseline/branch-name.txt)
# Get the most recent run id for this branch + workflow
sleep 10 # give GH a moment to register the run
RUN_ID=$(gh run list --branch "$BRANCH" --workflow release.yml --limit 1 --json databaseId --jq '.[0].databaseId')
echo "Watching run $RUN_ID"
gh run watch "$RUN_ID" --exit-status --interval 15
echo "Workflow exit: $?"
```
Expected: exit status 0 after ~3-5 min. If exit != 0, print the logs:
```bash
gh run view "$RUN_ID" --log-failed | tail -100
```
and abort with message: `Release workflow failed for throwaway branch — investigate before hackathon`.
- [ ] **Step 2.4: Verify per-branch tag exists on GHCR**
```bash
SLUG=$(cat /tmp/dryrun-baseline/slug.txt)
EXPECTED="ghcr.io/keboola/agnes-the-ai-analyst:dev-$SLUG"
docker manifest inspect "$EXPECTED" > /tmp/dryrun-baseline/ghcr-manifest.json
DIGEST=$(jq -r '.config.digest // .manifests[0].digest' /tmp/dryrun-baseline/ghcr-manifest.json)
echo "Tag exists: $EXPECTED"
echo "Digest: $DIGEST"
echo "$DIGEST" > /tmp/dryrun-baseline/expected-digest.txt
```
Expected: `docker manifest inspect` returns JSON (exit 0), a non-empty digest is extracted. If the tag is missing, abort with message: `release.yml did not produce :dev-<slug> tag — check build-and-push step logs`.
- [ ] **Step 2.5: Record Task 2 result**
```bash
SLUG=$(cat /tmp/dryrun-baseline/slug.txt)
cat >> /tmp/dryrun-report.md <<EOF
## Task 2: Per-Branch GHCR Build — PASS
- Branch: $(cat /tmp/dryrun-baseline/branch-name.txt)
- Slug: $SLUG
- Tag: ghcr.io/keboola/agnes-the-ai-analyst:dev-$SLUG
- Digest: $(cat /tmp/dryrun-baseline/expected-digest.txt)
EOF
```
**Task 2 gate:** `:dev-<slug>` manifest exists. Proceed.
---
## Task 3: Dev VM Switch Flow
**Purpose:** Simulate the hackathon developer path — have the shared `agnes-dev` VM pick up the per-branch image via the existing auto-upgrade cron, verify the new image is running, then (in Task 6) roll back.
**Files touched (reversibly):**
- `/data/.env` on `agnes-dev` VM — one-line `AGNES_TAG=` change (rollback is captured in baseline from Step 1.5)
- [ ] **Step 3.1: Switch `agnes-dev` `.env` AGNES_TAG to the per-branch tag**
```bash
SLUG=$(cat /tmp/dryrun-baseline/slug.txt)
NEW_TAG="dev-$SLUG"
gcloud compute ssh agnes-dev --zone=europe-west1-b --quiet --command "\
sudo cp /data/.env /data/.env.dryrun-bak && \
sudo sed -i 's|^AGNES_TAG=.*|AGNES_TAG=$NEW_TAG|' /data/.env && \
sudo grep -E '^AGNES_TAG=' /data/.env"
```
Expected: final line is `AGNES_TAG=dev-<slug>`. If sed didn't match (no `AGNES_TAG=` line existed), abort and manually investigate.
- [ ] **Step 3.2: Trigger auto-upgrade cron script immediately**
```bash
gcloud compute ssh agnes-dev --zone=europe-west1-b --quiet --command \
"sudo /usr/local/bin/agnes-auto-upgrade.sh 2>&1 | tail -30"
```
Expected: output shows `docker compose pull` + `docker compose up -d` activity. If the script doesn't exist or errors, abort with message: `auto-upgrade script missing or broken on agnes-dev`.
- [ ] **Step 3.3: Wait for app container to become healthy**
```bash
# Poll /api/health for up to 90s
for i in $(seq 1 30); do
STATUS=$(curl -s --max-time 5 http://34.77.94.14:8000/api/health | jq -r '.status' 2>/dev/null || echo "down")
echo "[$i/30] status=$STATUS"
if [ "$STATUS" = "healthy" ] || [ "$STATUS" = "degraded" ]; then
break
fi
sleep 3
done
[ "$STATUS" = "healthy" ] || [ "$STATUS" = "degraded" ] || { echo "FAIL: dev never healthy"; exit 1; }
```
Expected: reaches `healthy`/`degraded` within 90s.
- [ ] **Step 3.4: Verify the running image is the per-branch one**
```bash
SLUG=$(cat /tmp/dryrun-baseline/slug.txt)
EXPECTED_DIGEST=$(cat /tmp/dryrun-baseline/expected-digest.txt)
RUNNING_IMAGE=$(gcloud compute ssh agnes-dev --zone=europe-west1-b --quiet --command \
"docker inspect \$(docker ps -qf name=app) --format '{{.Image}}'")
echo "Running image digest: $RUNNING_IMAGE"
# The running image line will be sha256:xxxxx. Compare to the manifest digest we recorded.
# They should match (or differ only by multi-arch manifest indirection — compare via docker inspect on remote)
gcloud compute ssh agnes-dev --zone=europe-west1-b --quiet --command \
"docker inspect \$(docker ps -qf name=app) --format '{{.Config.Image}}' && \
docker image inspect \$(docker ps -qf name=app --format '{{.Image}}' | head -1) --format '{{.RepoTags}}{{.RepoDigests}}'"
```
Expected: `RepoTags` or `RepoDigests` output includes either `:dev-$SLUG` or the digest from Step 2.4. If neither matches, the cron didn't pull the new tag — record as FAIL and continue (cleanup is still required).
- [ ] **Step 3.5: Record Task 3 result**
The agent must judge PASS/FAIL based on Step 3.4 output: PASS iff `RepoTags` or `RepoDigests` contained `:dev-$SLUG` or the digest captured in Step 2.4.
```bash
SLUG=$(cat /tmp/dryrun-baseline/slug.txt)
# Replace <RESULT> with PASS or FAIL based on the Step 3.4 output the agent observed.
# Replace <IMAGE_OUTPUT> with the RepoTags/RepoDigests line from Step 3.4.
# Replace <SECONDS> with the loop iteration count from Step 3.3 × 3.
cat >> /tmp/dryrun-report.md <<EOF
## Task 3: Dev VM Switch — <RESULT>
- Switched agnes-dev to AGNES_TAG=dev-$SLUG
- Health after switch: reached healthy/degraded within 90s
- Running image: <IMAGE_OUTPUT>
- Time from cron trigger to healthy: <SECONDS>s
EOF
```
**Task 3 gate:** health reached OK state; running image verified. Proceed even if image verification was inconclusive — rollback still required.
---
## Task 4: Terraform Plan Verification (Private Repo)
**Purpose:** Validate that adding a new entry to `dev_instances` produces a clean `terraform plan` (not apply) in `agnes-infra-keboola`. This proves the TF module accepts the variable shape the hackathon docs will recommend.
**Skip condition:** If prerequisites check found that `/tmp/agnes-infra-keboola` clone failed, skip this entire task and record `SKIPPED — repo unavailable` in the report.
**Files touched (throwaway branch only):**
- `/tmp/agnes-infra-keboola/terraform/terraform.tfvars` (throwaway edit)
- [ ] **Step 4.1: Create throwaway branch in private repo**
```bash
cd /tmp/agnes-infra-keboola
git checkout main
git pull
EPOCH=$(date +%s)
BRANCH="dryrun-tfplan-${EPOCH}"
echo "$BRANCH" > /tmp/dryrun-baseline/tf-branch.txt
git checkout -b "$BRANCH"
```
Expected: clean checkout of main, new branch created.
- [ ] **Step 4.2: Add throwaway dev_instance entry**
Read `terraform/terraform.tfvars` first to understand the current `dev_instances` shape. Then append a new entry.
The `dev_instances` variable schema (from `infra/modules/customer-instance/variables.tf:41-49`) is:
```hcl
list(object({
name = string
machine_type = optional(string, "e2-small")
image_tag = optional(string, "dev")
}))
```
Modify the `dev_instances` list to append:
```hcl
{ name = "agnes-hack-dryrun", image_tag = "dev-<slug-from-task-2>" }
```
The agent should detect the current tfvars format and insert accordingly. If the file does not already contain `dev_instances`, abort and report format-mismatch.
```bash
SLUG=$(cat /tmp/dryrun-baseline/slug.txt)
# Show current tfvars for context
cat /tmp/agnes-infra-keboola/terraform/terraform.tfvars | grep -A 20 "dev_instances"
# Agent must edit the file to add the new entry — use the Edit tool rather than sed to be safe.
```
After editing, show the diff:
```bash
cd /tmp/agnes-infra-keboola
git diff terraform/terraform.tfvars
```
Expected: diff adds exactly one new entry to `dev_instances` list with `name = "agnes-hack-dryrun"` and `image_tag = "dev-<slug>"`.
- [ ] **Step 4.3: Run `terraform plan` locally (no apply)**
```bash
cd /tmp/agnes-infra-keboola/terraform
export GOOGLE_APPLICATION_CREDENTIALS="$HOME/.agnes-keys/agnes-deploy-kids-ai-data-analysis-key.json"
[ -f "$GOOGLE_APPLICATION_CREDENTIALS" ] || { echo "SA key not found — skipping plan"; exit 2; }
terraform init -input=false -upgrade=false
terraform plan -input=false -no-color -out=/tmp/dryrun-tfplan.bin > /tmp/dryrun-tfplan.txt 2>&1
RC=$?
echo "terraform plan exit: $RC"
tail -40 /tmp/dryrun-tfplan.txt
```
Expected:
- exit 0 or 2 (2 = changes detected, which is what we want)
- output ends with `Plan: N to add, M to change, K to destroy.` where `N >= 1` (at least the new VM + disk + IP) and `K == 0` (we must NOT be destroying anything)
If `K > 0` or `terraform plan` errors, abort and DO NOT proceed to Step 4.4. Report the plan output verbatim in the final report.
- [ ] **Step 4.4: Discard throwaway branch (no push, no apply)**
```bash
cd /tmp/agnes-infra-keboola
git checkout main
BRANCH=$(cat /tmp/dryrun-baseline/tf-branch.txt)
git branch -D "$BRANCH"
# Branch was never pushed, so nothing to clean up remotely.
```
Expected: branch deleted locally, main is current, working tree clean.
- [ ] **Step 4.5: Record Task 4 result**
```bash
ADDS=$(grep -E "Plan:" /tmp/dryrun-tfplan.txt | head -1)
DESTROYS_OK=$(grep -E "Plan:.*0 to destroy" /tmp/dryrun-tfplan.txt && echo yes || echo no)
cat >> /tmp/dryrun-report.md <<EOF
## Task 4: TF Plan for New Dev VM — <PASS|SKIPPED|FAIL>
- Plan summary: $ADDS
- Zero destroys: $DESTROYS_OK
- Full plan output: see /tmp/dryrun-tfplan.txt
EOF
```
**Task 4 gate:** plan produced with 0 destroys and ≥1 add. Proceed.
---
## Task 5: Verify Smoke-Test Gate Blocks Broken PR
**Purpose:** Confirm that a pull request with a deliberately-failing test does NOT produce a passing CI — which is the safety net that keeps `:stable` from auto-promoting broken images to prod.
**Files touched (throwaway branch only):**
- `tests/test_dryrun_should_fail.py` (new file on throwaway branch)
**Important:** This task creates a PR (not a merge). The PR is closed without merging in Step 5.5.
- [ ] **Step 5.1: Create throwaway branch with failing test**
```bash
cd "/Users/zdeneksrotyr/Library/Mobile Documents/com~apple~CloudDocs/Sources/VsCode/component_factory/tmp_oss"
git checkout main
git pull
EPOCH=$(date +%s)
BRANCH="dryrun-break-smoke-${EPOCH}"
echo "$BRANCH" > /tmp/dryrun-baseline/smoke-branch.txt
git checkout -b "$BRANCH"
cat > tests/test_dryrun_should_fail.py <<'PYEOF'
def test_intentional_fail_for_dryrun():
"""Intentional failure to verify CI gate blocks broken PRs. Remove after dryrun."""
assert False, "dryrun: this test is supposed to fail"
PYEOF
git add tests/test_dryrun_should_fail.py
git commit -m "dryrun: intentional failing test (will be reverted)"
git push -u origin "$BRANCH"
```
Expected: push succeeds.
- [ ] **Step 5.2: Open PR**
```bash
cd "/Users/zdeneksrotyr/Library/Mobile Documents/com~apple~CloudDocs/Sources/VsCode/component_factory/tmp_oss"
PR_URL=$(gh pr create --title "dryrun: verify CI gate (DO NOT MERGE)" \
--body "Intentionally failing test to verify CI blocks bad merges. Will be closed immediately after CI result." \
--base main)
echo "$PR_URL" > /tmp/dryrun-baseline/pr-url.txt
echo "Opened: $PR_URL"
```
Expected: PR URL returned.
- [ ] **Step 5.3: Wait for CI `test` job to complete (expected: FAIL)**
```bash
cd "/Users/zdeneksrotyr/Library/Mobile Documents/com~apple~CloudDocs/Sources/VsCode/component_factory/tmp_oss"
BRANCH=$(cat /tmp/dryrun-baseline/smoke-branch.txt)
sleep 15
RUN_ID=$(gh run list --branch "$BRANCH" --workflow release.yml --limit 1 --json databaseId --jq '.[0].databaseId')
echo "Watching run $RUN_ID (expected to FAIL)"
# Use --exit-status WITHOUT `set -e`; we expect non-zero
set +e
gh run watch "$RUN_ID" --exit-status --interval 15
EXIT=$?
set -e
echo "Exit code: $EXIT (non-zero is EXPECTED here)"
```
Expected: exit code != 0. If exit code IS 0, that means CI passed despite `assert False` → the test suite is not being run, or the file was excluded → record as **FAIL — CI gate broken**.
- [ ] **Step 5.4: Verify PR mergeability check shows failure**
```bash
PR_URL=$(cat /tmp/dryrun-baseline/pr-url.txt)
PR_NUM=$(basename "$PR_URL")
STATE=$(gh pr view "$PR_NUM" --json statusCheckRollup --jq '.statusCheckRollup[] | select(.name=="test") | .conclusion')
echo "test job conclusion: $STATE"
```
Expected: `FAILURE`. If `SUCCESS`, the gate is broken.
- [ ] **Step 5.5: Close PR and delete branch**
```bash
cd "/Users/zdeneksrotyr/Library/Mobile Documents/com~apple~CloudDocs/Sources/VsCode/component_factory/tmp_oss"
PR_URL=$(cat /tmp/dryrun-baseline/pr-url.txt)
PR_NUM=$(basename "$PR_URL")
gh pr close "$PR_NUM" --delete-branch --comment "dryrun complete — CI gate verified, closing without merge"
# Also delete locally
git checkout main
BRANCH=$(cat /tmp/dryrun-baseline/smoke-branch.txt)
git branch -D "$BRANCH" 2>/dev/null || true
```
Expected: PR closed, local branch gone.
- [ ] **Step 5.6: Check whether `main` has required status checks configured**
```bash
gh api repos/keboola/agnes-the-ai-analyst/branches/main/protection 2>/tmp/dryrun-protection-err.txt > /tmp/dryrun-protection.json
RC=$?
if [ $RC -ne 0 ]; then
echo "No branch protection on main (or insufficient permissions to read it)"
cat /tmp/dryrun-protection-err.txt
PROTECTION_NOTE="NONE — branch is unprotected; broken PRs can be merged. Recommend adding 'test' as required status check."
else
REQUIRED=$(jq -r '.required_status_checks.contexts[]?' /tmp/dryrun-protection.json 2>/dev/null | tr '\n' ',')
echo "Required checks: $REQUIRED"
if echo "$REQUIRED" | grep -q "test"; then
PROTECTION_NOTE="OK — 'test' is required."
else
PROTECTION_NOTE="PARTIAL — protection exists but 'test' is not required. Contexts: $REQUIRED"
fi
fi
echo "$PROTECTION_NOTE" > /tmp/dryrun-baseline/protection-note.txt
```
Expected: note written. Does not abort — informational only.
- [ ] **Step 5.7: Record Task 5 result**
```bash
cat >> /tmp/dryrun-report.md <<EOF
## Task 5: CI Gate — <PASS|FAIL>
- Throwaway PR: $(cat /tmp/dryrun-baseline/pr-url.txt) (closed)
- CI 'test' job result on broken code: <FAILURE expected>
- Branch protection on main: $(cat /tmp/dryrun-baseline/protection-note.txt)
EOF
```
**Task 5 gate:** broken PR's CI status is FAILURE. Proceed. If `PROTECTION_NOTE` says NONE/PARTIAL, the final report must flag this as a **hackathon-blocking recommendation**.
---
## Task 6: Cleanup and Baseline Restoration
**Purpose:** Leave the system in exactly the state recorded in Task 1. This is the most important task — a dirty dry-run poisons the hackathon.
- [ ] **Step 6.1: Restore `agnes-dev` AGNES_TAG**
```bash
ORIG_LINE=$(cat /tmp/dryrun-baseline/dev-env.txt)
# ORIG_LINE looks like: AGNES_TAG=dev
ORIG_VALUE=$(echo "$ORIG_LINE" | cut -d= -f2-)
gcloud compute ssh agnes-dev --zone=europe-west1-b --quiet --command "\
sudo sed -i 's|^AGNES_TAG=.*|AGNES_TAG=$ORIG_VALUE|' /data/.env && \
sudo rm -f /data/.env.dryrun-bak && \
sudo grep -E '^AGNES_TAG=' /data/.env && \
sudo /usr/local/bin/agnes-auto-upgrade.sh 2>&1 | tail -20"
```
Expected: AGNES_TAG line matches original, auto-upgrade pulls back to the original tag.
- [ ] **Step 6.2: Wait for dev VM to return to healthy state on original tag**
```bash
for i in $(seq 1 30); do
STATUS=$(curl -s --max-time 5 http://34.77.94.14:8000/api/health | jq -r '.status' 2>/dev/null || echo down)
echo "[$i/30] status=$STATUS"
[ "$STATUS" = "healthy" ] || [ "$STATUS" = "degraded" ] && break
sleep 3
done
```
Expected: reaches healthy/degraded within 90s.
- [ ] **Step 6.3: Verify running image matches baseline**
```bash
RESTORED=$(gcloud compute ssh agnes-dev --zone=europe-west1-b --quiet --command \
"docker inspect \$(docker ps -qf name=app) --format '{{.Config.Image}}'")
ORIG=$(cat /tmp/dryrun-baseline/dev-image.txt)
echo "Restored: $RESTORED"
echo "Original: $ORIG"
[ "$RESTORED" = "$ORIG" ] && echo MATCH || echo "MISMATCH — investigate"
```
Expected: MATCH. If MISMATCH, the baseline-tag digest may have advanced (auto-upgrade pulled newer `:stable`/`:dev` floating image during the run) — that is acceptable as long as the `.Config.Image` *tag* matches. Record exact difference in report.
- [ ] **Step 6.4: Delete throwaway branches in public repo**
```bash
cd "/Users/zdeneksrotyr/Library/Mobile Documents/com~apple~CloudDocs/Sources/VsCode/component_factory/tmp_oss"
STARTING=$(cat /tmp/dryrun-baseline/starting-branch.txt)
git checkout "$STARTING"
FEAT_BRANCH=$(cat /tmp/dryrun-baseline/branch-name.txt)
SMOKE_BRANCH=$(cat /tmp/dryrun-baseline/smoke-branch.txt 2>/dev/null || echo "")
# Local delete
git branch -D "$FEAT_BRANCH" 2>/dev/null || true
[ -n "$SMOKE_BRANCH" ] && git branch -D "$SMOKE_BRANCH" 2>/dev/null || true
# Remote delete (smoke branch was already deleted via `gh pr close --delete-branch` in Step 5.5)
git push origin --delete "$FEAT_BRANCH" 2>/dev/null || echo "(feature branch already gone)"
```
Expected: local branches gone, remote feature branch deleted. QUICKSTART.md commit on throwaway branch vanishes from origin.
- [ ] **Step 6.5: Final health check on prod (must match baseline)**
```bash
curl -sf --max-time 10 http://34.77.102.61:8000/api/health > /tmp/dryrun-baseline/prod-health-after.json
BEFORE=$(jq -r '.status' /tmp/dryrun-baseline/prod-health.json)
AFTER=$(jq -r '.status' /tmp/dryrun-baseline/prod-health-after.json)
echo "Prod status before: $BEFORE / after: $AFTER"
[ "$BEFORE" = "$AFTER" ] && echo UNCHANGED || echo DRIFT
```
Expected: UNCHANGED. (Note: prod was never touched, so this is sanity only.)
- [ ] **Step 6.6: Record Task 6 result**
```bash
cat >> /tmp/dryrun-report.md <<EOF
## Task 6: Cleanup — <PASS|FAIL>
- agnes-dev AGNES_TAG restored to: $(cat /tmp/dryrun-baseline/dev-env.txt)
- agnes-dev health after restore: $(curl -s --max-time 5 http://34.77.94.14:8000/api/health | jq -r '.status')
- agnes-dev image: matches baseline? <MATCH|MISMATCH — paste both>
- Throwaway branches deleted: feature, smoke
- Prod status unchanged: <UNCHANGED|DRIFT>
EOF
```
**Task 6 gate:** dev VM back on its baseline tag, branches gone, prod untouched.
---
## Task 7: Generate Deliverables
**Purpose:** Produce the artefacts the user needs tomorrow: a helper script for the hackathon team and a consolidated report.
**Files:**
- Create: `scripts/switch-dev-vm.sh` (new)
- Create (already being built): `/tmp/dryrun-report.md`
- [ ] **Step 7.1: Write `scripts/switch-dev-vm.sh`**
Create file at `scripts/switch-dev-vm.sh`:
```bash
#!/usr/bin/env bash
# switch-dev-vm.sh — point the shared hackathon dev VM at the caller's branch image.
#
# Usage:
# scripts/switch-dev-vm.sh <branch-slug>
# scripts/switch-dev-vm.sh hack-zs-metrics
#
# Prerequisite: your branch has been pushed and the release.yml workflow has completed,
# producing ghcr.io/keboola/agnes-the-ai-analyst:dev-<slug>.
#
# The slug is derived from your branch name by stripping the leading "feature/" and
# replacing non-alphanumeric chars with "-". For branch "feature/hack-zs-metrics" the slug
# is "hack-zs-metrics".
set -euo pipefail
if [ $# -ne 1 ]; then
echo "Usage: $0 <branch-slug>" >&2
echo "Example: $0 hack-zs-metrics" >&2
exit 2
fi
SLUG="$1"
VM="agnes-dev"
ZONE="europe-west1-b"
TAG="dev-$SLUG"
IMAGE="ghcr.io/keboola/agnes-the-ai-analyst:$TAG"
echo "[1/4] Verifying $IMAGE exists on GHCR..."
docker manifest inspect "$IMAGE" > /dev/null || {
echo "ERROR: $IMAGE not found on GHCR. Did your release.yml run finish?" >&2
echo "Check: gh run list --branch feature/$SLUG --workflow release.yml" >&2
exit 1
}
echo "[2/4] Updating AGNES_TAG on $VM to $TAG..."
gcloud compute ssh "$VM" --zone="$ZONE" --quiet --command "\
sudo sed -i 's|^AGNES_TAG=.*|AGNES_TAG=$TAG|' /data/.env && \
sudo grep -E '^AGNES_TAG=' /data/.env"
echo "[3/4] Triggering auto-upgrade..."
gcloud compute ssh "$VM" --zone="$ZONE" --quiet --command \
"sudo /usr/local/bin/agnes-auto-upgrade.sh 2>&1 | tail -10"
echo "[4/4] Waiting for app to become healthy..."
for i in $(seq 1 30); do
STATUS=$(curl -s --max-time 5 http://34.77.94.14:8000/api/health | python3 -c 'import sys,json; print(json.load(sys.stdin).get("status","down"))' 2>/dev/null || echo down)
echo " [$i/30] status=$STATUS"
if [ "$STATUS" = "healthy" ] || [ "$STATUS" = "degraded" ]; then
echo "OK — agnes-dev now running $TAG. Open http://34.77.94.14:8000"
exit 0
fi
sleep 3
done
echo "ERROR: agnes-dev did not become healthy in 90s. SSH in and check: docker compose logs" >&2
exit 1
```
```bash
chmod +x scripts/switch-dev-vm.sh
bash -n scripts/switch-dev-vm.sh # syntax check
```
Expected: syntax-check passes, file executable.
- [ ] **Step 7.2: Commit the script on a fresh branch and open PR**
```bash
cd "/Users/zdeneksrotyr/Library/Mobile Documents/com~apple~CloudDocs/Sources/VsCode/component_factory/tmp_oss"
git checkout -b feature/hackathon-dryrun-deliverables
git add scripts/switch-dev-vm.sh
git commit -m "chore: add switch-dev-vm.sh helper for hackathon"
git push -u origin HEAD
gh pr create --title "chore: add switch-dev-vm.sh helper for hackathon" \
--body "Adds scripts/switch-dev-vm.sh. Produced by the 2026-04-21 hackathon dry-run. Reviewed by user before merge." \
--base main > /tmp/dryrun-baseline/deliverable-pr.txt
cat /tmp/dryrun-baseline/deliverable-pr.txt
```
Expected: PR URL. **Do not merge** — leave for user review.
- [ ] **Step 7.3: Finalise report with overall verdict**
Determine overall verdict by inspecting each Task's PASS/FAIL line in `/tmp/dryrun-report.md`. Overall is PASS only if all tasks PASS (SKIPPED Task 4 is acceptable — note it).
Append to report:
```bash
cat >> /tmp/dryrun-report.md <<EOF
---
## Overall Verdict
<PASS | PASS WITH GAPS | FAIL>
## Recommendations for the User Before Hackathon Starts
1. <If protection-note said NONE/PARTIAL:> Configure required status check 'test' on main branch of keboola/agnes-the-ai-analyst.
2. Pin prod image_tag in agnes-infra-keboola/terraform/terraform.tfvars from "stable" to "stable-2026.04.XX" (current running version). Revert after hackathon.
3. Rotate admin password '1234' on prod (34.77.102.61:8000/login) and dev (34.77.94.14:8000/login).
4. Wire notification_channel_ids in tfvars so uptime alerts actually notify someone.
5. Share the hackathon 1-pager + switch-dev-vm.sh via the team Slack channel.
6. Review PR $(cat /tmp/dryrun-baseline/deliverable-pr.txt) and merge if switch-dev-vm.sh looks good.
## Artefacts
- Full report: /tmp/dryrun-report.md (this file)
- Baseline snapshots: /tmp/dryrun-baseline/*.{json,txt}
- TF plan output: /tmp/dryrun-tfplan.txt (if Task 4 ran)
- Deliverable PR: $(cat /tmp/dryrun-baseline/deliverable-pr.txt)
EOF
cat /tmp/dryrun-report.md
```
Expected: full report printed.
- [ ] **Step 7.4: Print final summary to chat**
Agent should output, in its final message to the user:
- Overall verdict (one line)
- Each task's result (one line each)
- Any unresolved anomalies
- Link to deliverable PR
- Path to full report
**Task 7 gate:** report complete, PR open, all artefacts listed.
---
## Abort / Rollback Procedures
If any task fails mid-execution, the agent must still perform Task 6 cleanup before reporting failure. Specifically:
- If Task 2 push succeeded but Task 3 failed → still run Task 6 Steps 6.1-6.4 to restore dev VM and delete the branch.
- If Task 5 PR was opened but workflow didn't finish → close the PR with `gh pr close --delete-branch` and log it.
- If Task 4 TF plan showed destroys → abort immediately, do NOT attempt apply, record in report, continue to Task 6.
If Task 6 itself fails (dev VM won't come back healthy on original tag), the agent must:
1. Print the baseline values (from `/tmp/dryrun-baseline/dev-env.txt`, `/tmp/dryrun-baseline/dev-image.txt`) so the user can manually SSH and fix.
2. Attempt `gcloud compute ssh agnes-dev --zone=europe-west1-b --command "docker compose -f /opt/agnes/docker-compose.yml logs --tail 100"` and include output in the report.
3. Mark overall verdict as FAIL and stop.
## What a Successful Run Looks Like
- Task 1 baseline: captured with prod+dev healthy/degraded
- Task 2: GHCR manifest exists for `:dev-hack-dryrun-<epoch>`
- Task 3: agnes-dev briefly running the per-branch image, healthy within 90s
- Task 4: `terraform plan` showed `1+ to add, 0 to destroy` (or SKIPPED)
- Task 5: CI `test` job reported FAILURE on the broken PR, PR closed
- Task 6: agnes-dev back on its baseline AGNES_TAG, healthy, branches gone
- Task 7: `scripts/switch-dev-vm.sh` committed on PR for user review, full report in `/tmp/dryrun-report.md`
- Final agent message: verdict + 6 bullet results + deliverable PR link
Duration: ~45-75 minutes, bounded primarily by CI workflow runs (~3-5 min each, two runs) and TF init (~30s-2min cold).

490
docs/superpowers/plans/2026-04-21-issues-14-and-10.md Normal file
View file

@ -0,0 +1,490 @@
# Issues #14 + #10 Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Resolve two independent GitHub issues — add `scripts/switch-dev-vm.sh` helper for the hackathon (#14) and make unauthenticated HTML routes redirect to `/login` instead of returning raw JSON 401 (#10).
**Architecture:** Two independent changes on separate branches, shipped as two separate PRs.
- **#14** is a new standalone shell script plus a one-liner in `docs/QUICKSTART.md`. No app code touched. Blast radius = zero.
- **#10** adds a single FastAPI exception handler in `app/main.py` that intercepts `HTTPException(401)` for non-`/api/*` paths and redirects to `/login?next=<path>`. Implementation choice: path-scoped global handler (not per-route dep-swap) because it's deterministic, keeps `app/web/router.py` unchanged, and guarantees API routes under `/api/*` keep their existing JSON-401 contract. The `?next=` round-trip is honored only for the password web-login form (`/auth/password/login/web`) — the most common path. Google OAuth and email-link logins continue to land on `/dashboard` as today (documented follow-up, no regression).
**Tech Stack:** Bash, FastAPI, Starlette, pytest, Jinja2.
---
## Out of Scope
- Rewriting Google OAuth or email-magic-link flows to honor `?next=`. Those land on `/dashboard` today; this PR does not change that. A follow-up issue can track it.
- Any refactor of `app/auth/dependencies.py` beyond what's needed. `get_current_user` and `require_role` stay untouched.
- Any change to `/api/*` auth behavior. JSON 401 remains the contract for API callers.
---
## Task 1: Add `scripts/switch-dev-vm.sh` helper (Issue #14)
**Files:**
- Create: `scripts/switch-dev-vm.sh`
- Modify: `docs/QUICKSTART.md` (append a "Hackathon" section)
Reference implementation lives in `docs/superpowers/plans/2026-04-21-hackathon-dry-run.md` lines 694–750 (Task 7.1). Copy that script verbatim.
- [ ] **Step 1.1: Create the feature branch**
```bash
cd "/Users/zdeneksrotyr/Library/Mobile Documents/com~apple~CloudDocs/Sources/VsCode/component_factory/tmp_oss"
git checkout main
git pull --ff-only
git checkout -b feature/switch-dev-vm-helper
```
- [ ] **Step 1.2: Create `scripts/switch-dev-vm.sh`**
Write the file at `scripts/switch-dev-vm.sh` with this exact content:
```bash
#!/usr/bin/env bash
# switch-dev-vm.sh — point the shared hackathon dev VM at the caller's branch image.
#
# Usage:
# scripts/switch-dev-vm.sh <branch-slug>
# scripts/switch-dev-vm.sh hack-zs-metrics
#
# Prerequisite: your branch has been pushed and the release.yml workflow has completed,
# producing ghcr.io/keboola/agnes-the-ai-analyst:dev-<slug>.
#
# The slug is derived from your branch name by stripping the leading "feature/" and
# replacing non-alphanumeric chars with "-". For branch "feature/hack-zs-metrics" the slug
# is "hack-zs-metrics".
set -euo pipefail
if [ $# -ne 1 ]; then
echo "Usage: $0 <branch-slug>" >&2
echo "Example: $0 hack-zs-metrics" >&2
exit 2
fi
SLUG="$1"
VM="agnes-dev"
ZONE="europe-west1-b"
TAG="dev-$SLUG"
IMAGE="ghcr.io/keboola/agnes-the-ai-analyst:$TAG"
echo "[1/4] Verifying $IMAGE exists on GHCR..."
docker manifest inspect "$IMAGE" > /dev/null || {
echo "ERROR: $IMAGE not found on GHCR. Did your release.yml run finish?" >&2
echo "Check: gh run list --branch feature/$SLUG --workflow release.yml" >&2
exit 1
}
echo "[2/4] Updating AGNES_TAG on $VM to $TAG..."
gcloud compute ssh "$VM" --zone="$ZONE" --quiet --command "\
sudo sed -i 's|^AGNES_TAG=.*|AGNES_TAG=$TAG|' /data/.env && \
sudo grep -E '^AGNES_TAG=' /data/.env"
echo "[3/4] Triggering auto-upgrade..."
gcloud compute ssh "$VM" --zone="$ZONE" --quiet --command \
"sudo /usr/local/bin/agnes-auto-upgrade.sh 2>&1 | tail -10"
echo "[4/4] Waiting for app to become healthy..."
for i in $(seq 1 30); do
STATUS=$(curl -s --max-time 5 http://34.77.94.14:8000/api/health | python3 -c 'import sys,json; print(json.load(sys.stdin).get("status","down"))' 2>/dev/null || echo down)
echo " [$i/30] status=$STATUS"
if [ "$STATUS" = "healthy" ] || [ "$STATUS" = "degraded" ]; then
echo "OK — agnes-dev now running $TAG. Open http://34.77.94.14:8000"
exit 0
fi
sleep 3
done
echo "ERROR: agnes-dev did not become healthy in 90s. SSH in and check: docker compose logs" >&2
exit 1
```
- [ ] **Step 1.3: Make executable and syntax-check**
```bash
chmod +x scripts/switch-dev-vm.sh
bash -n scripts/switch-dev-vm.sh
```
Expected: `bash -n` prints nothing and exits 0.
- [ ] **Step 1.4: Append a Hackathon section to `docs/QUICKSTART.md`**
At the end of `docs/QUICKSTART.md`, append:
```markdown
## Hackathon: switch the shared dev VM to your branch
During the hackathon the shared VM `agnes-dev` can be pointed at any per-branch image built by `release.yml` (`ghcr.io/keboola/agnes-the-ai-analyst:dev-<slug>`).
```bash
# Slug = branch name without "feature/" prefix, non-alphanumeric → "-"
scripts/switch-dev-vm.sh hack-zs-metrics
```
The script verifies the image exists on GHCR, updates `AGNES_TAG` in `/data/.env` on the VM, triggers the auto-upgrade, and polls `/api/health` for up to 90 s. Requires `gcloud`, `docker`, `curl`, and `python3`.
```
- [ ] **Step 1.5: Commit**
```bash
git add scripts/switch-dev-vm.sh docs/QUICKSTART.md
git commit -m "chore: add switch-dev-vm.sh helper for hackathon (#14)"
```
- [ ] **Step 1.6: Push and open PR**
```bash
git push -u origin HEAD
gh pr create \
--base main \
--title "chore: add switch-dev-vm.sh helper for hackathon (#14)" \
--body "$(cat <<'EOF'
## Summary
Adds `scripts/switch-dev-vm.sh` — one-shot helper for the hackathon that points `agnes-dev` at the caller's per-branch image and waits for the app to become healthy.
Script verbatim from `docs/superpowers/plans/2026-04-21-hackathon-dry-run.md` Task 7.1.
Closes #14.
## Test plan
- [ ] `bash -n scripts/switch-dev-vm.sh` passes
- [ ] Running against a non-existent tag exits non-zero before touching the VM
- [ ] Running against a real `dev-<slug>` tag leaves `agnes-dev` healthy within 90 s (verified manually during hackathon dry-run)
EOF
)"
```
Expected: PR URL printed. Do not merge — leave for user review.
---
## Task 2: HTML routes redirect to `/login` on missing auth (Issue #10)
**Files:**
- Modify: `app/main.py` (register the exception handler)
- Modify: `app/web/router.py:193-221` (`login_page` reads `?next=` from query and passes into template context)
- Modify: `app/web/templates/login_email.html` (add hidden `<input name="next">` to the password form)
- Modify: `app/auth/providers/password.py` (`password_login_web` accepts `next` form field and redirects to sanitized path)
- Modify: `tests/test_web_ui.py` (add regression tests)
**Approach:** A single `HTTPException` handler on the FastAPI app instance. When the raised status is `401` *and* the request path does not start with `/api/`, return `RedirectResponse("/login?next=<path>", 302)`. Otherwise fall through to Starlette's default JSON response. API routes are unaffected by path scoping.
The `?next=` round-trip is implemented only for the `password_login_web` path (most common). The login template receives the raw `next` query-string param and embeds it as a hidden form field. The web-login handler sanitizes (must start with `/`, must not start with `//`, must not contain a scheme) and redirects to that path on success.
- [ ] **Step 2.1: Create the feature branch**
```bash
cd "/Users/zdeneksrotyr/Library/Mobile Documents/com~apple~CloudDocs/Sources/VsCode/component_factory/tmp_oss"
git checkout main
git pull --ff-only
git checkout -b fix/web-auth-redirect-to-login
```
- [ ] **Step 2.2: Write the failing test for unauthenticated HTML redirect**
Append to `tests/test_web_ui.py` (inside the existing file, after `TestWebUISmoke` or as a new class at the bottom):
```python
class TestUnauthenticatedHtmlRedirects:
def test_dashboard_unauthenticated_redirects_to_login(self, web_client):
resp = web_client.get("/dashboard", follow_redirects=False)
assert resp.status_code == 302
assert resp.headers["location"].startswith("/login")
assert "next=%2Fdashboard" in resp.headers["location"]
def test_catalog_unauthenticated_redirects_to_login(self, web_client):
resp = web_client.get("/catalog", follow_redirects=False)
assert resp.status_code == 302
assert resp.headers["location"].startswith("/login")
assert "next=%2Fcatalog" in resp.headers["location"]
def test_api_route_still_returns_json_401(self, web_client):
# /api/sync/status requires auth; must keep JSON 401 (no redirect).
resp = web_client.get("/api/sync/status", follow_redirects=False)
assert resp.status_code == 401
assert resp.headers["content-type"].startswith("application/json")
```
- [ ] **Step 2.3: Run the new tests — confirm they fail**
```bash
source .venv/bin/activate
pytest tests/test_web_ui.py::TestUnauthenticatedHtmlRedirects -v
```
Expected: `test_dashboard_unauthenticated_redirects_to_login` and `test_catalog_unauthenticated_redirects_to_login` FAIL with `assert 401 == 302`. `test_api_route_still_returns_json_401` passes already (baseline behavior).
- [ ] **Step 2.4: Register the exception handler in `app/main.py`**
Open `app/main.py` and add these imports at the top of the file (alongside existing imports near line 9–14):
```python
from urllib.parse import quote
from starlette.exceptions import HTTPException as StarletteHTTPException
from fastapi.responses import RedirectResponse
```
Then, inside `create_app()` after the routers are registered (after all `app.include_router(...)` calls but before `return app`), add:
```python
@app.exception_handler(StarletteHTTPException)
async def _html_auth_redirect_handler(request, exc: StarletteHTTPException):
"""Redirect unauthenticated HTML requests to /login; leave /api/* as JSON 401."""
if exc.status_code == 401 and not request.url.path.startswith("/api/"):
next_param = quote(request.url.path, safe="")
return RedirectResponse(url=f"/login?next={next_param}", status_code=302)
# Fall back to Starlette's default JSON handler
from starlette.exceptions import HTTPException as _SE
from fastapi.exception_handlers import http_exception_handler
return await http_exception_handler(request, exc)
```
Note: registering a handler for `StarletteHTTPException` catches both FastAPI's `HTTPException` (which subclasses it) and any direct Starlette-raised one.
- [ ] **Step 2.5: Re-run the failing tests — confirm they now pass**
```bash
pytest tests/test_web_ui.py::TestUnauthenticatedHtmlRedirects -v
```
Expected: all three tests PASS.
- [ ] **Step 2.6: Run the full `test_web_ui.py` suite — confirm no regressions**
```bash
pytest tests/test_web_ui.py -v
```
Expected: all tests PASS (existing + new).
- [ ] **Step 2.7: Pass `?next=` into the login page context**
In `app/web/router.py`, in the `login_page` function (around line 193), read the query param. Replace the existing function body:
```python
@router.get("/login", response_class=HTMLResponse)
async def login_page(request: Request):
next_path = request.query_params.get("next", "")
# Safety: only accept same-origin paths (must start with "/", must not start with "//").
if not next_path.startswith("/") or next_path.startswith("//"):
next_path = ""
providers = []
try:
from app.auth.providers.google import is_available as google_available
if google_available():
providers.append({"name": "google", "display_name": "Google", "icon": "google"})
except Exception:
pass
providers.append({"name": "password", "display_name": "Email & Password", "icon": "key"})
try:
from app.auth.providers.email import is_available as email_available
if email_available():
providers.append({"name": "email", "display_name": "Email Link", "icon": "mail"})
except Exception:
pass
login_buttons = []
for p in providers:
if p["name"] == "google":
login_buttons.append({"url": "/auth/google/login", "text": "Sign in with Google", "css_class": "btn-primary", "icon_html": ""})
elif p["name"] == "password":
login_buttons.append({"url": "/login/password", "text": "Sign in with Email & Password", "css_class": "btn-secondary", "icon_html": ""})
elif p["name"] == "email":
login_buttons.append({"url": "/login/email", "text": "Sign in with Email Link", "css_class": "btn-secondary", "icon_html": ""})
ctx = _build_context(request, providers=providers, login_buttons=login_buttons, next_path=next_path)
return templates.TemplateResponse(request, "login.html", ctx)
```
Also update `login_password_page` (around line 224) the same way — it renders `login_email.html` which contains the password form:
```python
@router.get("/login/password", response_class=HTMLResponse)
async def login_password_page(request: Request):
next_path = request.query_params.get("next", "")
if not next_path.startswith("/") or next_path.startswith("//"):
next_path = ""
google_ok = False
try:
from app.auth.providers.google import is_available as google_available
google_ok = google_available()
except Exception:
pass
ctx = _build_context(request, google_available=google_ok, next_path=next_path)
return templates.TemplateResponse(request, "login_email.html", ctx)
```
- [ ] **Step 2.8: Add the hidden `next` field to the password login form**
First, inspect the current form markup to find the right insertion point:
```bash
grep -n "action=\"/auth/password/login/web\"\|<form" app/web/templates/login_email.html
```
Then open `app/web/templates/login_email.html` and, inside the `<form>` that POSTs to `/auth/password/login/web`, add **as the first child of the form** (right after the opening `<form ...>` tag):
```html
<input type="hidden" name="next" value="{{ next_path|default('', true) }}">
```
If `login.html` also contains a direct-POST login form that hits `/auth/password/login/web`, add the same hidden input there too. (Grep first: `grep -n "/auth/password/login/web" app/web/templates/*.html`.)
- [ ] **Step 2.9: Honor `next` in `password_login_web`**
In `app/auth/providers/password.py`, replace the `password_login_web` function body so that the redirect target is derived from the `next` form field. The updated function:
```python
@router.post("/login/web")
async def password_login_web(
email: str = Form(...),
password: str = Form(""),
next: str = Form(""),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
"""Web form login — sets cookie and redirects to `next` (or /dashboard)."""
repo = UserRepository(conn)
user = repo.get_by_email(email)
if not user or not user.get("password_hash"):
return RedirectResponse(url="/login/password?error=invalid", status_code=302)
try:
ph = PasswordHasher()
ph.verify(user["password_hash"], password)
except (VerifyMismatchError, Exception):
return RedirectResponse(url="/login/password?error=invalid", status_code=302)
token = create_access_token(user["id"], user["email"], user["role"])
use_secure = os.environ.get("DOMAIN", "") != ""
# Sanitize next: must start with "/" and not with "//" (prevent open-redirect).
target = next if (next.startswith("/") and not next.startswith("//")) else "/dashboard"
response = RedirectResponse(url=target, status_code=302)
response.set_cookie(
key="access_token", value=token,
httponly=True, max_age=86400, samesite="lax",
secure=use_secure,
)
return response
```
- [ ] **Step 2.10: Add a test for the `?next=` round-trip**
Append to `TestUnauthenticatedHtmlRedirects` in `tests/test_web_ui.py`:
```python
def test_password_login_honors_next(self, web_client, tmp_path):
from argon2 import PasswordHasher
from src.db import get_system_db
from src.repositories.users import UserRepository
password = "TestPass1!"
conn = get_system_db()
UserRepository(conn).create(
id="u1", email="u1@test.com", name="U1", role="admin",
password_hash=PasswordHasher().hash(password),
)
conn.close()
resp = web_client.post(
"/auth/password/login/web",
data={"email": "u1@test.com", "password": password, "next": "/catalog"},
follow_redirects=False,
)
assert resp.status_code == 302
assert resp.headers["location"] == "/catalog"
def test_password_login_rejects_open_redirect(self, web_client, tmp_path):
from argon2 import PasswordHasher
from src.db import get_system_db
from src.repositories.users import UserRepository
password = "TestPass1!"
conn = get_system_db()
UserRepository(conn).create(
id="u2", email="u2@test.com", name="U2", role="admin",
password_hash=PasswordHasher().hash(password),
)
conn.close()
resp = web_client.post(
"/auth/password/login/web",
data={"email": "u2@test.com", "password": password, "next": "//evil.example/"},
follow_redirects=False,
)
assert resp.status_code == 302
assert resp.headers["location"] == "/dashboard"
```
- [ ] **Step 2.11: Run the new tests — confirm they pass**
```bash
pytest tests/test_web_ui.py::TestUnauthenticatedHtmlRedirects -v
```
Expected: all five tests PASS.
- [ ] **Step 2.12: Run the broader suite to check for regressions**
```bash
pytest tests/test_web_ui.py tests/test_auth_providers.py tests/test_access_control.py -v
```
Expected: all PASS. If any fail that previously passed, investigate before committing.
- [ ] **Step 2.13: Commit**
```bash
git add app/main.py app/web/router.py app/web/templates/login_email.html app/auth/providers/password.py tests/test_web_ui.py
# Only stage login.html if it was actually edited in step 2.8:
git status --short
git commit -m "fix: redirect unauthenticated HTML routes to /login (#10)"
```
- [ ] **Step 2.14: Push and open PR**
```bash
git push -u origin HEAD
gh pr create \
--base main \
--title "fix: redirect unauthenticated HTML routes to /login (#10)" \
--body "$(cat <<'EOF'
## Summary
Unauthenticated access to HTML pages like `/dashboard` returned a raw JSON 401 body; it now redirects to `/login?next=<path>` and the password login form honors `next` on success.
Implementation: a single `StarletteHTTPException` handler registered on the FastAPI app. When status is `401` and the request path does not start with `/api/`, return `302 /login?next=<path>`. Otherwise fall through to Starlette's default JSON response, so `/api/*` routes keep their existing JSON 401 contract.
Closes #10.
## What this PR does
- `/dashboard`, `/catalog`, `/corporate-memory`, `/activity-center`, admin pages, etc. → 302 to `/login?next=<path>`
- `/api/*` routes unchanged — still return JSON 401
- Password web-login form carries `next` through as a hidden field and redirects there on success (sanitized: must start with `/`, must not start with `//`)
## Out of scope (follow-up)
Google OAuth and the email-magic-link provider still land on `/dashboard` after login regardless of `next`. No regression vs. today; tracked for a follow-up issue.
## Test plan
- [x] `pytest tests/test_web_ui.py -v` passes
- [x] New tests cover: HTML redirect, API route JSON 401 preserved, `next` honored, open-redirect rejected
- [ ] Manual: open `/dashboard` in a fresh browser → lands on `/login?next=%2Fdashboard`, sign in with email+password, end up on `/dashboard`
EOF
)"
```
Expected: PR URL printed. Do not merge — leave for user review.
---
## Self-Review
- [x] **Spec coverage:**
- Issue #14 acceptance: `chmod +x` + `bash -n` (Step 1.3), runs against good tag (PR test-plan manual), non-existent tag exits before VM (script logic lines), docs section (Step 1.4). All covered.
- Issue #10 acceptance: unauthenticated HTML redirects (Step 2.2/2.3/2.5), `?next=` round-trip after sign-in (Step 2.10), API JSON 401 preserved (Step 2.2 third case), `/dashboard` 302 test (Step 2.2). All covered.
- [x] **No placeholders:** every step has the concrete file path, code block, and command. No TBD/TODO.
- [x] **Type consistency:** `next_path` used consistently in router + template context; form field name `next` consistent between template hidden input (Step 2.8) and `password_login_web` signature (Step 2.9).

3143
docs/superpowers/plans/2026-04-21-user-mgmt-pat-cli.md Normal file
View file

File diff suppressed because it is too large Load diff

1411
docs/superpowers/plans/2026-04-22-cloudflare-access-auth.md Normal file
View file

File diff suppressed because it is too large Load diff

79
docs/superpowers/plans/2026-04-22-grpn-deploy-learnings.md Normal file
View file

@ -0,0 +1,79 @@
# GRPN deploy learnings — hackathon 2026-04-22
Running log of constraints encountered while deploying Agnes to GRPN's `prj-grp-foundryai-dev-7c37` on an existing VM (`foundryai-development`). Recorded during deploy; each entry captures the constraint, workaround, and what it implies for our Terraform flow.
## Constraints hit
### 1. No `projectIamAdmin` on human identity
- **Signal:** `bootstrap-gcp.sh` failed on `gcloud projects add-iam-policy-binding` with `[e_zsrotyr@groupon.com] does not have permission ... setIamPolicy`.
- **Root cause:** `roles/editor` intentionally excludes `resourcemanager.projects.setIamPolicy`.
- **Workaround (hackathon):** Skip `bootstrap-gcp.sh`. Deploy on existing VM with docker-compose; use VM's existing SA without adding any new IAM bindings.
- **Implication for TF flow:** For a proper per-customer deploy, the GRPN admin must grant `roles/resourcemanager.projectIamAdmin` to either the onboarding engineer or directly to `agnes-deploy` SA. Or onboarding becomes two-phase: engineer creates SA + bucket; admin grants roles.
### 2. Organization policy `iam.disableServiceAccountKeyCreation`
- **Signal:** `gcloud iam service-accounts keys create` returned `Key creation is not allowed on this service account`.
- **Root cause:** Org-level `constraints/iam.disableServiceAccountKeyCreation` applies to all projects in the organization. Intentional security posture — static SA JSON keys are the highest-risk credential type.
- **Workaround:** Can't produce a `GCP_SA_KEY` GitHub secret for CI/CD. Options:
- **WIF (Workload Identity Federation)**: GitHub Actions OIDC → GCP, no static keys. Requires bootstrap updates (create WIF pool + provider + binding on deploy SA).
- **Skip CI/CD for GRPN**: Run `terraform apply` only from developer laptops with user ADC (`gcloud auth application-default login`). Works for hackathon, does not scale.
- **Implication for TF flow:** Our current bootstrap + `apply.yml` assume SA JSON key. GRPN (and any org with this org policy) requires WIF path. Track as follow-up; for hackathon we skip CI entirely.
### 3. Resource-level `setIamPolicy` also blocked
- **Signal:** `gcloud secrets add-iam-policy-binding` returned `Permission 'secretmanager.secrets.setIamPolicy' denied`.
- **Root cause:** `editor` does not grant `setIamPolicy` on any resource, even secret-level. Stricter than standard GCP default; likely additional org policies.
- **Workaround:** Don't use Secret Manager for hackathon secrets. Store JWT + any tokens directly in `.env` on the VM with `chmod 600`.
- **Implication:** Our module's secret-based `.env` assembly from Secret Manager needs a fallback path when `setIamPolicy` is blocked. For now: document that customers who can't grant IAM must bake secrets into `.env` manually (still via `scp`, not git).
### 4. VM has no external IP (IAP tunnel only)
- **Signal:** `gcloud compute ssh` auto-falls-back to IAP tunnel; direct IP access from browser impossible.
- **Root cause:** GRPN VMs are created in a private VPC. Standard security posture. Our module default (`access_config { nat_ip = ... }`) is the opposite — external IP by default.
- **Workaround:** Browser access via IAP tunnel: `gcloud compute start-iap-tunnel foundryai-development 8000 --local-host-port=localhost:8000`. Then `http://localhost:8000`.
- **Implication:** Our module needs an `external_ip` variable (default `true`) that customers can disable. Plus docs for IAP tunnel access pattern.
### 5. VM's SA scopes include `cloud-platform` (default overkill)
- **Signal:** `grpn-sa-foundryai-execution@...` has `cloud-platform` scope — full GCP access.
- **Root cause:** GRPN's default compute SA configuration.
- **Workaround:** Use VM's existing SA; it already has enough (BigQuery datasets, Compute, etc.). No need to create a dedicated `agnes-vm` SA (and we couldn't anyway — would need `projectIamAdmin`).
- **Implication:** For hackathon OK. For production the SA is overprovisioned — different customer than us, our opinion doesn't apply.
### 6. Docker not pre-installed
- **Signal:** `docker: command not found` on fresh VM.
- **Root cause:** VM is generic Ubuntu, no opinions about Docker.
- **Workaround:** `curl -fsSL https://get.docker.com | sudo sh` + `sudo apt install docker-compose-plugin`. Took ~30 s.
- **Implication:** Any non-TF-managed VM will need this. Our module's startup script already does this; manual deploys need it inline or a small bootstrap script.
### 7. `/data` did not exist
- **Signal:** `df /data` → No such file or directory.
- **Root cause:** Fresh VM, no persistent disk attached for data.
- **Workaround:** `mkdir -p /data/{state,analytics,extracts}` on boot disk. Ephemeral — data lives with VM. Acceptable for hackathon.
- **Implication:** For production this would mean no data survives VM recreate. Module's persistent-disk + `host-mount` overlay is the right long-term answer. For hackathon, boot disk is fine.
## Derived follow-ups (post-hackathon)
- [ ] **Add WIF path to `bootstrap-gcp.sh`** — alternative to SA JSON key. Detect `iam.disableServiceAccountKeyCreation` constraint and switch automatically.
- [ ] **Make `external_ip` + `iap_only` optional in customer-instance module** — GRPN-style customers need VMs without NAT.
- [ ] **Document two-phase bootstrap flow** — engineer creates SA, admin grants roles. Or admin runs the script on behalf.
- [ ] **Fallback `.env` assembly** — when Secret Manager is blocked, allow operator to `scp` secrets.
- [ ] **Customer onboarding checklist addition** — verify required project IAM before onboarding starts:
- `resourcemanager.projects.setIamPolicy` (for adding binding to SA)
- `iam.serviceAccountKeys.create` — check org policy `iam.disableServiceAccountKeyCreation` → if true, mandate WIF
- `compute.firewalls.create` (for firewall rules)
- `compute.disks.create`, `compute.instances.create` (for VM)
- `secretmanager.*` (for secrets)
- `storage.buckets.create` (for tfstate bucket, if hosted in customer project)
## Hackathon deploy summary (live)
- VM: `foundryai-development` in `prj-grp-foundryai-dev-7c37`, zone `us-central1-a`, e2-medium, 30GB boot, IAP-only access
- Data source: `csv` (no external data ingest needed for hackathon)
- App directory: `/opt/agnes/`, docker-compose fetched from upstream `main`
- Data directory: `/data` on boot disk (ephemeral)
- Secrets: plain `.env` with chmod 600 (org policy blocks Secret Manager IAM bindings)
- Access: IAP tunnel on port 8000

1308
docs/superpowers/specs/2026-04-14-connector-kit-design.md Normal file
View file

File diff suppressed because it is too large Load diff

146
scripts/grpn/Makefile Normal file
View file

@ -0,0 +1,146 @@
# Makefile — Agnes on foundryai-development (GRPN hackathon deploy)
#
# This is a manual-deploy helper used while the full Terraform flow is
# blocked by GRPN org policies (iam.disableServiceAccountKeyCreation,
# no projectIamAdmin delegation). It targets the existing VM
# foundryai-development in prj-grp-foundryai-dev-7c37.
#
# Once WIF + Terraform is unblocked, this file moves to a private
# keboola/agnes-infra-grpn repo and most targets become obsolete.
#
# Usage:
# make -C scripts/grpn help
# make -C scripts/grpn deploy
# make -C scripts/grpn status
# make -C scripts/grpn tunnel
SHELL := /bin/bash
# -------- overridable config (safe defaults for GRPN foundryai-development) --------
PROJECT ?= prj-grp-foundryai-dev-7c37
ZONE ?= us-central1-a
VM ?= foundryai-development
APP_DIR ?= /opt/agnes
LOCAL_PORT ?= 8000
VM_PORT ?= 8000
IMAGE ?= ghcr.io/keboola/agnes-the-ai-analyst
ADMIN_EMAIL ?= e_zsrotyr@groupon.com
# compose files (note: host-mount overlay binds /data from host = boot-disk ephemeral for this VM)
COMPOSE_FILES = -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.host-mount.yml
COMPOSE = sudo docker compose $(COMPOSE_FILES)
SSH = gcloud compute ssh $(VM) --zone=$(ZONE) --project=$(PROJECT)
SCP = gcloud compute scp --zone=$(ZONE) --project=$(PROJECT)
# -------- help --------
.PHONY: help
help:
@echo "Agnes @ $(VM) (project: $(PROJECT), zone: $(ZONE))"
@echo ""
@echo " make deploy Pull latest :stable image, recreate containers (zero-downtime if healthy)"
@echo " make deploy-tag TAG=stable-2026.04.83 Pull a specific tag instead of floating :stable"
@echo " make status Health + version endpoint"
@echo " make logs Tail app logs (ctrl-c to exit)"
@echo " make logs-scheduler Tail scheduler logs"
@echo " make restart docker compose restart (keeps state)"
@echo " make stop docker compose stop (containers down, volumes preserved)"
@echo " make start docker compose up -d"
@echo " make recreate docker compose down + up -d (fresh containers, same data)"
@echo ""
@echo " make ssh Open interactive SSH session to the VM"
@echo " make tunnel Start IAP tunnel; open http://localhost:$(LOCAL_PORT) in browser"
@echo " make open Start tunnel AND open browser (macOS only)"
@echo ""
@echo " make bootstrap-admin PASSWORD=<new-pwd> Create admin (first-time only; 403 once any user has password)"
@echo " make set-data-source SOURCE=bigquery Edit .env DATA_SOURCE; restart app"
@echo ""
@echo " make install-cron Install auto-upgrade cron (pulls :stable every 5 min, restarts on digest change)"
@echo " make uninstall-cron Remove auto-upgrade cron"
@echo ""
@echo " make env Show .env keys (values NOT printed)"
@echo " make version What version/channel/commit is running now"
@echo " make ps docker ps on the VM"
# -------- deployment --------
.PHONY: deploy deploy-tag recreate restart start stop
deploy:
$(SSH) --command='cd $(APP_DIR) && $(COMPOSE) pull && $(COMPOSE) up -d'
@$(MAKE) --no-print-directory status
deploy-tag:
@test -n "$(TAG)" || (echo "Usage: make deploy-tag TAG=stable-2026.04.83" >&2; exit 2)
$(SSH) --command='cd $(APP_DIR) && sudo sed -i "s|^AGNES_TAG=.*|AGNES_TAG=$(TAG)|" .env && $(COMPOSE) pull && $(COMPOSE) up -d'
@$(MAKE) --no-print-directory status
recreate:
$(SSH) --command='cd $(APP_DIR) && $(COMPOSE) down && $(COMPOSE) up -d'
@$(MAKE) --no-print-directory status
restart:
$(SSH) --command='cd $(APP_DIR) && $(COMPOSE) restart'
start:
$(SSH) --command='cd $(APP_DIR) && $(COMPOSE) up -d'
stop:
$(SSH) --command='cd $(APP_DIR) && $(COMPOSE) down'
# -------- observability --------
.PHONY: status version ps env logs logs-scheduler
status:
@echo "=== health (via IAP tunnel on VM) ==="
@$(SSH) --command='curl -sf --max-time 10 http://localhost:$(VM_PORT)/api/health' 2>&1 | tail -1 | python3 -m json.tool 2>/dev/null | head -10 || echo "not healthy"
version:
@$(SSH) --command='curl -sf --max-time 10 http://localhost:$(VM_PORT)/api/version' 2>&1 | tail -1 | python3 -m json.tool 2>/dev/null | head -10 || echo "unreachable"
ps:
$(SSH) --command='sudo docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"'
env:
@echo "=== .env keys on VM (values not shown) ==="
$(SSH) --command='sudo cut -d= -f1 $(APP_DIR)/.env'
logs:
$(SSH) --command='sudo docker logs -f --tail 100 agnes-app-1'
logs-scheduler:
$(SSH) --command='sudo docker logs -f --tail 100 agnes-scheduler-1'
# -------- access --------
.PHONY: ssh tunnel open
ssh:
$(SSH)
tunnel:
@echo "Starting IAP tunnel — http://localhost:$(LOCAL_PORT) is now Agnes"
@echo "Leave this terminal open; Ctrl-C to stop."
gcloud compute start-iap-tunnel $(VM) $(VM_PORT) \
--local-host-port=localhost:$(LOCAL_PORT) \
--zone=$(ZONE) --project=$(PROJECT)
open:
@( gcloud compute start-iap-tunnel $(VM) $(VM_PORT) \
--local-host-port=localhost:$(LOCAL_PORT) \
--zone=$(ZONE) --project=$(PROJECT) & \
sleep 4 && open "http://localhost:$(LOCAL_PORT)/login" && wait )
# -------- one-off operations --------
.PHONY: bootstrap-admin set-data-source install-cron uninstall-cron
bootstrap-admin:
@test -n "$(PASSWORD)" || (echo "Usage: make bootstrap-admin PASSWORD=<secret>" >&2; exit 2)
@$(SSH) --command='curl -sS -X POST http://localhost:$(VM_PORT)/auth/bootstrap \
-H "Content-Type: application/json" \
-d "{\"email\":\"$(ADMIN_EMAIL)\",\"password\":\"$(PASSWORD)\"}"' 2>&1 | tail -1 | python3 -m json.tool 2>/dev/null | head -8
set-data-source:
@test -n "$(SOURCE)" || (echo "Usage: make set-data-source SOURCE=bigquery|csv|keboola" >&2; exit 2)
$(SSH) --command='sudo sed -i "s|^DATA_SOURCE=.*|DATA_SOURCE=$(SOURCE)|" $(APP_DIR)/.env && cd $(APP_DIR) && $(COMPOSE) up -d --force-recreate app'
@$(MAKE) --no-print-directory status
install-cron:
$(SCP) agnes-auto-upgrade.sh $(VM):/tmp/agnes-auto-upgrade.sh
$(SSH) --command='sudo install -m 755 /tmp/agnes-auto-upgrade.sh /usr/local/bin/agnes-auto-upgrade.sh && rm /tmp/agnes-auto-upgrade.sh && ( sudo crontab -l 2>/dev/null | grep -v agnes-auto-upgrade || true; echo "*/5 * * * * /usr/local/bin/agnes-auto-upgrade.sh >> /var/log/agnes-auto-upgrade.log 2>&1" ) | sudo crontab - && echo "cron installed"'
uninstall-cron:
$(SSH) --command='( sudo crontab -l 2>/dev/null | grep -v agnes-auto-upgrade ) | sudo crontab - && sudo rm -f /usr/local/bin/agnes-auto-upgrade.sh && echo "cron removed"'

179
scripts/grpn/README.md Normal file
View file

@ -0,0 +1,179 @@
# Manual deploy helper — Agnes on an existing VM (GRPN pattern)
A `make`-based helper for deploying and operating Agnes on an **existing** GCE VM when the full Terraform flow is blocked — typically by organization policies that forbid SA JSON key creation or by missing IAM delegation. This is the pattern we used on GRPN's `foundryai-development` during the 2026-04-22 hackathon.
It is **not** a replacement for the full Terraform module — only a stopgap while the proper flow is being unblocked. See [Migration path](#migration-path) below.
## When to use this
Use this helper when **all** are true:
- A target VM already exists in the customer's GCP project (we don't create it)
- You (or the deploy SA) do **not** have `roles/resourcemanager.projectIamAdmin` on that project, **or** the org has `constraints/iam.disableServiceAccountKeyCreation` enabled
- The customer is OK with a single-VM, single-node Agnes (no prod + dev split for now)
- Data persistence on the VM's boot disk is acceptable (no persistent disk attached → data loss on VM recreate)
Any of those false → go the Terraform route via [`docs/HACKATHON.md`](../../docs/HACKATHON.md) Part 1.
## What it does (and doesn't)
| Aspect | Manual helper (this) | Full Terraform flow |
|---|---|---|
| VM provisioning | Reuses existing VM | Creates a dedicated `agnes-prod` + optional `agnes-dev` VMs |
| Docker install | Inline `curl get.docker.com \| sh` on first deploy | Part of the module's startup script |
| Secrets | Plain `.env` on VM (`chmod 600`) | GCP Secret Manager, read by VM SA |
| Service account | Uses the VM's existing SA, whatever that is | Dedicated `agnes-<customer>-vm` with scoped `secretmanager.secretAccessor` only |
| Data persistence | Boot disk, ephemeral across VM recreate | Separate persistent disk (`/data` bind-mount), daily snapshot + 30-day retention |
| Auto-upgrade | `install-cron` target deploys the same cron script the module uses | Built into the startup script |
| Monitoring / alerts | None | Uptime check + alert policy per VM |
| Backup | None | Daily snapshot schedule |
| Branch-aware dev VMs | Not supported (single VM) | `dev_instances` list — one VM per branch/engineer |
| CI/CD | None — manual `make deploy` | GitHub Actions: PR → plan → apply (dev auto, prod gated) |
The helper covers the **runtime** aspects (pull image, restart, logs, access) but skips the infra-as-code posture.
## One-time setup
Done for GRPN during the 2026-04-22 hackathon. Re-useable template for any future customer in a similar constrained environment:
### 1. Verify access to the VM
```bash
gcloud compute ssh $VM --zone=$ZONE --project=$PROJECT --command='whoami'
```
If this works, you have SSH via OS Login or your own key. IAP tunnel auto-kicks in if the VM has no external IP. No further auth setup is needed.
### 2. Install Docker + compose plugin
```bash
gcloud compute ssh $VM --zone=$ZONE --project=$PROJECT --command="
curl -fsSL https://get.docker.com | sudo sh
sudo apt-get install -y -qq docker-compose-plugin
"
```
### 3. Prepare app directory and data root
```bash
gcloud compute ssh $VM --zone=$ZONE --project=$PROJECT --command="
sudo mkdir -p /opt/agnes /data/state /data/analytics /data/extracts
sudo chown -R \$USER:\$USER /opt/agnes
cd /opt/agnes
curl -fsSL https://raw.githubusercontent.com/keboola/agnes-the-ai-analyst/main/docker-compose.yml -o docker-compose.yml
curl -fsSL https://raw.githubusercontent.com/keboola/agnes-the-ai-analyst/main/docker-compose.prod.yml -o docker-compose.prod.yml
curl -fsSL https://raw.githubusercontent.com/keboola/agnes-the-ai-analyst/main/docker-compose.host-mount.yml -o docker-compose.host-mount.yml
"
```
### 4. Write `.env` (plain, chmod 600)
```bash
JWT=$(openssl rand -hex 32)
cat > /tmp/agnes-env <<EOF
JWT_SECRET_KEY=$JWT
DATA_DIR=/data
DATA_SOURCE=csv # or bigquery / keboola
SEED_ADMIN_EMAIL=<your@email>
LOG_LEVEL=info
AGNES_TAG=stable
EOF
gcloud compute scp /tmp/agnes-env $VM:/tmp/.env --zone=$ZONE --project=$PROJECT
gcloud compute ssh $VM --zone=$ZONE --project=$PROJECT --command="
sudo install -m 600 -o \$USER -g \$USER /tmp/.env /opt/agnes/.env
rm /tmp/.env
"
rm /tmp/agnes-env
```
If `DATA_SOURCE=keboola`, add `KEBOOLA_STORAGE_TOKEN=...` + `KEBOOLA_STACK_URL=...` lines. Same for any BQ / custom data source credentials — they all live in this one `.env`.
### 5. First boot
```bash
make deploy
make bootstrap-admin PASSWORD=<strong-initial>
```
`deploy` pulls the image + starts containers. `bootstrap-admin` hits `/auth/bootstrap` to activate the seed admin.
### 6. (Optional) Auto-upgrade
```bash
make install-cron
```
Installs the same 5-minute polling cron used by the Terraform module. After this, every new `:stable` image digest is picked up within ~5 min without any human action.
## Everyday operations
From the repo root (tested defaults target GRPN's `foundryai-development`):
```bash
make -C scripts/grpn help # list all targets
make -C scripts/grpn status # is it up?
make -C scripts/grpn version # what's deployed right now
make -C scripts/grpn logs # tail app logs
make -C scripts/grpn deploy # pull :stable + recreate
make -C scripts/grpn tunnel # IAP tunnel → http://localhost:8000
```
## Configuration
All targets read overridable variables at the top of `Makefile`. Defaults target GRPN's `foundryai-development`. For other VMs/projects:
```bash
# one-off override
make -C scripts/grpn status \
PROJECT=other-project \
ZONE=us-central1-a \
VM=other-vm
# or fork this Makefile into `scripts/<customer>/Makefile` with different defaults
```
| Variable | Default | Purpose |
|---|---|---|
| `PROJECT` | `prj-grp-foundryai-dev-7c37` | GCP project ID |
| `ZONE` | `us-central1-a` | VM zone |
| `VM` | `foundryai-development` | Instance name |
| `APP_DIR` | `/opt/agnes` | Where compose files + `.env` live on the VM |
| `LOCAL_PORT` | `8000` | Local port for `tunnel` target |
| `VM_PORT` | `8000` | Port the app listens on inside the VM |
| `IMAGE` | `ghcr.io/keboola/agnes-the-ai-analyst` | GHCR image repo |
| `ADMIN_EMAIL` | `e_zsrotyr@groupon.com` | Default bootstrap email |
## Files
```
scripts/grpn/
├── Makefile # the helper itself
├── agnes-auto-upgrade.sh # deployed by `make install-cron` to /usr/local/bin/
└── README.md # this file
```
Plus the deploy log: [`docs/superpowers/plans/2026-04-22-grpn-deploy-learnings.md`](../../docs/superpowers/plans/2026-04-22-grpn-deploy-learnings.md) — lists all the org-policy constraints encountered and their workarounds.
## Migration path
Once the blockers are lifted, move to the proper Terraform flow:
1. **Get `roles/resourcemanager.projectIamAdmin`** on the customer project (ask the GRPN admin to grant it).
2. **Create a WIF pool + provider** in the customer project (doesn't require SA JSON keys; bypasses `iam.disableServiceAccountKeyCreation`). Draft patch pending on [`bootstrap-gcp.sh`](../bootstrap-gcp.sh) — track via GitHub issue tagged `wif`.
3. **Migrate**: run the new `bootstrap-gcp.sh --wif`, create a private infra repo from [`keboola/agnes-infra-template`](https://github.com/keboola/agnes-infra-template), `terraform apply` → this creates a **new** Agnes VM alongside the existing `foundryai-development`.
4. **Optional** — move data from the manual VM to the TF VM with a `tar` snapshot through GCS (see the original migration in [`docs/superpowers/plans/2026-04-21-deployment-log.md`](../../docs/superpowers/plans/2026-04-21-deployment-log.md) "Data migration" section).
5. **Decommission** the manual deploy: `make stop` + delete `/opt/agnes/` on the VM.
## Caveats
- **Single VM, single point of failure.** No dev/prod split.
- **No automatic backups.** If someone deletes the VM, data is gone (30-day boot-disk retention from GCP default only).
- **Plain-text secrets in `.env`.** Acceptable for IAP-only internal VM; **not** acceptable if the VM ever gets an external IP.
- **No drift detection.** Anyone with SSH can hand-edit `.env` or compose files without leaving an audit trail. The Terraform flow's `ignore_changes` + `-replace` pattern is the correct version of this.
## See also
- [`docs/HACKATHON.md`](../../docs/HACKATHON.md) — the full TL;DR for deploy and develop (the TF path)
- [`docs/ONBOARDING.md`](../../docs/ONBOARDING.md) — detailed per-customer Terraform onboarding
- [`docs/DEPLOYMENT.md`](../../docs/DEPLOYMENT.md) — comparison of TF vs docker-compose deployment strategies
- [`infra/modules/customer-instance/`](../../infra/modules/customer-instance/) — the Terraform module this helper shadows

18
scripts/grpn/agnes-auto-upgrade.sh Executable file
View file

@ -0,0 +1,18 @@
#!/bin/bash
# Deployed to /usr/local/bin/agnes-auto-upgrade.sh on the VM.
# Cron fires it every 5 min; pulls latest image for the pinned AGNES_TAG
# and recreates containers only if the digest moved.
set -euo pipefail
cd /opt/agnes
# shellcheck disable=SC1091
set -a; . /opt/agnes/.env; set +a
IMAGE="ghcr.io/keboola/agnes-the-ai-analyst:${AGNES_TAG:-stable}"
COMPOSE_FILES="-f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.host-mount.yml"
BEFORE=$(docker images --no-trunc --format '{{.Digest}}' "$IMAGE" | head -1)
docker compose $COMPOSE_FILES pull >/dev/null 2>&1
AFTER=$(docker images --no-trunc --format '{{.Digest}}' "$IMAGE" | head -1)
if [ "$BEFORE" != "$AFTER" ]; then
echo "$(date): new digest for $IMAGE — recreating containers"
docker compose $COMPOSE_FILES up -d
docker image prune -f >/dev/null 2>&1
fi

59
src/db.py
View file

@ -16,7 +16,7 @@ logger = logging.getLogger(__name__)
_SAFE_IDENTIFIER = re.compile(r"^[a-zA-Z_][a-zA-Z0-9_]{0,63}$")
SCHEMA_VERSION = 4
SCHEMA_VERSION = 7
_SYSTEM_SCHEMA = """
CREATE TABLE IF NOT EXISTS schema_version (
@ -34,6 +34,9 @@ CREATE TABLE IF NOT EXISTS users (
setup_token_created TIMESTAMP,
reset_token VARCHAR,
reset_token_created TIMESTAMP,
active BOOLEAN NOT NULL DEFAULT TRUE,
deactivated_at TIMESTAMP,
deactivated_by VARCHAR,
created_at TIMESTAMP DEFAULT current_timestamp,
updated_at TIMESTAMP
);
@ -204,6 +207,20 @@ CREATE TABLE IF NOT EXISTS column_metadata (
updated_at TIMESTAMP DEFAULT current_timestamp,
PRIMARY KEY (table_id, column_name)
);
CREATE TABLE IF NOT EXISTS personal_access_tokens (
id VARCHAR PRIMARY KEY,
user_id VARCHAR NOT NULL,
name VARCHAR NOT NULL,
token_hash VARCHAR NOT NULL,
prefix VARCHAR NOT NULL,
scopes VARCHAR,
created_at TIMESTAMP NOT NULL DEFAULT current_timestamp,
expires_at TIMESTAMP,
last_used_at TIMESTAMP,
last_used_ip VARCHAR,
revoked_at TIMESTAMP
);
"""
@ -384,6 +401,37 @@ _V2_TO_V3_MIGRATIONS = [
"ALTER TABLE table_registry ADD COLUMN IF NOT EXISTS is_public BOOLEAN DEFAULT true",
]
_V4_TO_V5_MIGRATIONS = [
# DuckDB doesn't allow ALTER TABLE ADD COLUMN with NOT NULL constraint,
# so we add the column with a DEFAULT, backfill, then the app-level
# code enforces non-null semantics (never inserts NULL for `active`).
"ALTER TABLE users ADD COLUMN IF NOT EXISTS active BOOLEAN DEFAULT TRUE",
"UPDATE users SET active = TRUE WHERE active IS NULL",
"ALTER TABLE users ADD COLUMN IF NOT EXISTS deactivated_at TIMESTAMP",
"ALTER TABLE users ADD COLUMN IF NOT EXISTS deactivated_by VARCHAR",
]
_V5_TO_V6_MIGRATIONS = [
"""
CREATE TABLE IF NOT EXISTS personal_access_tokens (
id VARCHAR PRIMARY KEY,
user_id VARCHAR NOT NULL,
name VARCHAR NOT NULL,
token_hash VARCHAR NOT NULL,
prefix VARCHAR NOT NULL,
scopes VARCHAR,
created_at TIMESTAMP NOT NULL DEFAULT current_timestamp,
expires_at TIMESTAMP,
last_used_at TIMESTAMP,
revoked_at TIMESTAMP
)
""",
]
_V6_TO_V7_MIGRATIONS = [
"ALTER TABLE personal_access_tokens ADD COLUMN IF NOT EXISTS last_used_ip VARCHAR",
]
_V3_TO_V4_MIGRATIONS = [
"""
CREATE TABLE IF NOT EXISTS metric_definitions (
@ -465,6 +513,15 @@ def _ensure_schema(conn: duckdb.DuckDBPyConnection) -> None:
if current < 4:
for sql in _V3_TO_V4_MIGRATIONS:
conn.execute(sql)
if current < 5:
for sql in _V4_TO_V5_MIGRATIONS:
conn.execute(sql)
if current < 6:
for sql in _V5_TO_V6_MIGRATIONS:
conn.execute(sql)
if current < 7:
for sql in _V6_TO_V7_MIGRATIONS:
conn.execute(sql)
conn.execute(
"UPDATE schema_version SET version = ?, applied_at = current_timestamp",
[SCHEMA_VERSION],

96
src/repositories/access_tokens.py Normal file
View file

@ -0,0 +1,96 @@
"""Repository for personal access tokens (#12)."""
from datetime import datetime, timezone
from typing import Any, Optional, List, Dict
import duckdb
class AccessTokenRepository:
def __init__(self, conn: duckdb.DuckDBPyConnection):
self.conn = conn
def _row_to_dict(self, row) -> Optional[Dict[str, Any]]:
if not row:
return None
columns = [desc[0] for desc in self.conn.description]
return dict(zip(columns, row))
def create(
self,
id: str,
user_id: str,
name: str,
token_hash: str,
prefix: str,
expires_at: Optional[datetime] = None,
scopes: Optional[str] = None,
) -> None:
self.conn.execute(
"""INSERT INTO personal_access_tokens
(id, user_id, name, token_hash, prefix, scopes, created_at, expires_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?)""",
[id, user_id, name, token_hash, prefix, scopes,
datetime.now(timezone.utc), expires_at],
)
def get_by_id(self, token_id: str) -> Optional[Dict[str, Any]]:
result = self.conn.execute(
"SELECT * FROM personal_access_tokens WHERE id = ?", [token_id]
).fetchone()
return self._row_to_dict(result)
def list_for_user(self, user_id: str, include_revoked: bool = True) -> List[Dict[str, Any]]:
sql = "SELECT * FROM personal_access_tokens WHERE user_id = ?"
if not include_revoked:
sql += " AND revoked_at IS NULL"
sql += " ORDER BY created_at DESC"
rows = self.conn.execute(sql, [user_id]).fetchall()
if not rows:
return []
columns = [desc[0] for desc in self.conn.description]
return [dict(zip(columns, r)) for r in rows]
def list_all(self) -> List[Dict[str, Any]]:
rows = self.conn.execute(
"SELECT * FROM personal_access_tokens ORDER BY created_at DESC"
).fetchall()
if not rows:
return []
columns = [desc[0] for desc in self.conn.description]
return [dict(zip(columns, r)) for r in rows]
def list_all_with_user(self) -> List[Dict[str, Any]]:
"""Admin view: all tokens joined with the owning user's email.
Returns dict rows including every column of `personal_access_tokens`
plus a denormalized `user_email` key (may be NULL if the user row was
deleted — the token itself survives until an admin revokes it).
"""
rows = self.conn.execute(
"""
SELECT t.*, u.email AS user_email
FROM personal_access_tokens t
LEFT JOIN users u ON u.id = t.user_id
ORDER BY t.created_at DESC
"""
).fetchall()
if not rows:
return []
columns = [desc[0] for desc in self.conn.description]
return [dict(zip(columns, r)) for r in rows]
def revoke(self, token_id: str) -> None:
self.conn.execute(
"UPDATE personal_access_tokens SET revoked_at = ? WHERE id = ?",
[datetime.now(timezone.utc), token_id],
)
def delete(self, token_id: str) -> None:
self.conn.execute("DELETE FROM personal_access_tokens WHERE id = ?", [token_id])
def mark_used(self, token_id: str, ip: Optional[str] = None) -> None:
self.conn.execute(
"UPDATE personal_access_tokens SET last_used_at = ?, last_used_ip = ? WHERE id = ?",
[datetime.now(timezone.utc), ip, token_id],
)

14
src/repositories/users.py
View file

@ -47,8 +47,11 @@ class UserRepository:
)
def update(self, id: str, **kwargs) -> None:
allowed = {"email", "name", "role", "password_hash", "setup_token",
"setup_token_created", "reset_token", "reset_token_created"}
allowed = {
"email", "name", "role", "password_hash", "setup_token",
"setup_token_created", "reset_token", "reset_token_created",
"active", "deactivated_at", "deactivated_by",
}
updates = {k: v for k, v in kwargs.items() if k in allowed}
if not updates:
return
@ -57,5 +60,12 @@ class UserRepository:
values = list(updates.values()) + [id]
self.conn.execute(f"UPDATE users SET {set_clause} WHERE id = ?", values)
def count_admins(self, active_only: bool = True) -> int:
sql = "SELECT COUNT(*) FROM users WHERE role = 'admin'"
if active_only:
sql += " AND COALESCE(active, TRUE) = TRUE"
result = self.conn.execute(sql).fetchone()
return int(result[0]) if result else 0
def delete(self, user_id: str) -> None:
self.conn.execute("DELETE FROM users WHERE id = ?", [user_id])

420
tests/test_admin_tokens_ui.py Normal file
View file

@ -0,0 +1,420 @@
"""Tests for the split /tokens (own) and /admin/tokens (all) UI.
The two routes render distinct templates:
- /tokens → my_tokens.html (any signed-in user, own PATs, create modal)
- /admin/tokens → admin_tokens.html (admin-only, all users, stat strip,
owner search, sort-by-owner)
/profile 302-redirects to /tokens for back-compat.
"""
import hashlib
import tempfile
import uuid
from datetime import datetime, timezone, timedelta
import pytest
@pytest.fixture
def fresh_db(monkeypatch):
with tempfile.TemporaryDirectory() as tmp:
monkeypatch.setenv("DATA_DIR", tmp)
monkeypatch.setenv("TESTING", "1")
monkeypatch.setenv("JWT_SECRET_KEY", "test-jwt-secret-key-minimum-32-chars!!")
yield tmp
def _make_user_and_session(conn, email: str, role: str):
"""Create a user and return (uid, session_jwt)."""
from src.repositories.users import UserRepository
from app.auth.jwt import create_access_token
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email=email, name=email.split("@")[0], role=role)
token = create_access_token(user_id=uid, email=email, role=role)
return uid, token
def _make_pat_row(conn, user_id: str, name: str = "ci",
expires_in_days: int = 30, revoked: bool = False,
last_used_ip: str | None = None,
last_used_ago_days: int | None = None):
from src.repositories.access_tokens import AccessTokenRepository
repo = AccessTokenRepository(conn)
tid = str(uuid.uuid4())
raw = "r" * 40
exp = datetime.now(timezone.utc) + timedelta(days=expires_in_days) if expires_in_days is not None else None
repo.create(
id=tid, user_id=user_id, name=name,
token_hash=hashlib.sha256(raw.encode()).hexdigest(),
prefix=tid.replace("-", "")[:8],
expires_at=exp,
)
if last_used_ago_days is not None:
ts = datetime.now(timezone.utc) - timedelta(days=last_used_ago_days)
conn.execute(
"UPDATE personal_access_tokens SET last_used_at = ?, last_used_ip = ? WHERE id = ?",
[ts, last_used_ip, tid],
)
if revoked:
repo.revoke(tid)
return tid
# ── /tokens — "My tokens" (own PATs) — every signed-in user ────────────────
def test_non_admin_sees_my_tokens_page(fresh_db):
"""Non-admin GET /tokens: personal body, New-token CTA, create modal."""
from fastapi.testclient import TestClient
from src.db import get_system_db, close_system_db
from app.main import app
conn = get_system_db()
try:
_, sess = _make_user_and_session(conn, "user@t", "analyst")
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.get(
"/tokens",
headers={"Accept": "text/html"},
cookies={"access_token": sess},
)
assert resp.status_code == 200, resp.text
body = resp.text
# Non-admin title + eyebrow
assert "My tokens" in body
assert "Your account" in body
assert "Long-lived tokens for CLI" in body
# Role-awareness marker stays on the page root
assert 'data-is-admin="false"' in body
assert 'data-view="my"' in body
# New-token CTA + create modal are rendered
assert 'id="new-token-btn"' in body
assert 'id="create-modal"' in body
assert 'id="reveal-banner"' in body
# Admin-only stat strip is NOT rendered
assert 'id="tokens-counts"' not in body
assert 'id="count-active"' not in body
# Owner search (admin-only) is NOT rendered
assert 'placeholder="Search by owner email' not in body
# Admin title must not bleed in
assert "Access tokens" not in body
assert "Administration" not in body
def test_admin_sees_my_tokens_on_tokens_path(fresh_db):
"""Admin GET /tokens renders the SAME "My tokens" page as non-admins.
/tokens is always the personal view — admins use /admin/tokens for the
org-wide list."""
from fastapi.testclient import TestClient
from src.db import get_system_db, close_system_db
from app.main import app
conn = get_system_db()
try:
_, admin_sess = _make_user_and_session(conn, "admin@t", "admin")
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.get(
"/tokens",
headers={"Accept": "text/html"},
cookies={"access_token": admin_sess},
)
assert resp.status_code == 200, resp.text
body = resp.text
# Personal view markers (same as non-admin)
assert "My tokens" in body
assert "Your account" in body
assert 'id="new-token-btn"' in body
assert 'id="create-modal"' in body
assert 'data-is-admin="false"' in body
# Admin-only UI must NOT show on /tokens, even for an admin
assert 'id="tokens-counts"' not in body
assert "Access tokens" not in body # admin hero title
assert "Administration" not in body
def test_unauthenticated_redirects_from_tokens_page(fresh_db):
from fastapi.testclient import TestClient
from app.main import app
client = TestClient(app)
resp = client.get(
"/tokens",
headers={"Accept": "text/html"},
follow_redirects=False,
)
assert resp.status_code in (302, 303, 401), resp.text
# ── /admin/tokens — admin-only list of ALL tokens ──────────────────────────
def test_admin_can_render_admin_tokens_page(fresh_db):
"""Admin GET /admin/tokens: the org-wide list with stat strip + owner
search + sort-by-owner chip."""
from fastapi.testclient import TestClient
from src.db import get_system_db, close_system_db
from app.main import app
conn = get_system_db()
try:
_, admin_sess = _make_user_and_session(conn, "admin@t", "admin")
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.get(
"/admin/tokens",
headers={"Accept": "text/html"},
cookies={"access_token": admin_sess},
)
assert resp.status_code == 200, resp.text
body = resp.text
# Admin-specific title + eyebrow + subtitle
assert "Access tokens" in body
assert "Administration" in body
assert "incident response and offboarding" in body
# Role-awareness marker
assert 'data-is-admin="true"' in body
assert 'data-view="admin"' in body
# Filter controls
assert 'id="flt-status"' in body
assert 'id="flt-user"' in body
assert 'id="flt-last-used"' in body
# Stat strip (admin-only)
assert 'id="tokens-counts"' in body
assert 'id="count-active"' in body
assert 'id="count-expiring"' in body
# Sort-by-owner chip is only on admin page
assert 'data-sort-key="user_email"' in body
# Owner search input
assert 'placeholder="Search by owner email' in body
# Revoke hook is in JS template
assert "data-revoke" in body
# Admin page must NOT have the "New token" CTA or create modal
assert 'id="new-token-btn"' not in body
assert 'id="create-modal"' not in body
assert 'id="reveal-banner"' not in body
# Admin page must NOT use the "My tokens" title in its main content.
# (The shared user-menu in the header shows a "My tokens" link for
# every signed-in user — scope the check to the page body only.)
page_start = body.find('class="tokens-page"')
assert page_start != -1, "admin tokens page body marker not found"
assert "My tokens" not in body[page_start:]
def test_non_admin_cannot_access_admin_tokens_page(fresh_db):
"""Non-admin GET /admin/tokens: 403 (or redirect) — admin-only route."""
from fastapi.testclient import TestClient
from src.db import get_system_db, close_system_db
from app.main import app
conn = get_system_db()
try:
_, sess = _make_user_and_session(conn, "user@t", "analyst")
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.get(
"/admin/tokens",
headers={"Accept": "text/html"},
cookies={"access_token": sess},
follow_redirects=False,
)
# require_role(Role.ADMIN) denies with 403 for non-admin
assert resp.status_code in (302, 401, 403), resp.text
def test_admin_tokens_deeplink_preserves_user_query(fresh_db):
"""/admin/users deep-links with ?user=<email>; page should still render
and contain the owner search input (JS pre-fills it)."""
from fastapi.testclient import TestClient
from src.db import get_system_db, close_system_db
from app.main import app
conn = get_system_db()
try:
_, admin_sess = _make_user_and_session(conn, "admin@t", "admin")
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.get(
"/admin/tokens?user=alice%40example.com",
headers={"Accept": "text/html"},
cookies={"access_token": admin_sess},
)
assert resp.status_code == 200, resp.text
# Owner search input is present; JS reads ?user from window.location.
assert 'id="flt-user"' in resp.text
# ── Back-compat redirects ─────────────────────────────────────────────────
def test_profile_redirects_to_tokens(fresh_db):
"""/profile no longer renders — it 302-redirects to /tokens."""
from fastapi.testclient import TestClient
from app.main import app
client = TestClient(app)
resp = client.get("/profile", follow_redirects=False)
assert resp.status_code == 302
assert resp.headers["location"] == "/tokens"
# ── Admin list API — expanded fields ───────────────────────────────────────
def test_admin_list_includes_user_email_and_last_used_ip(fresh_db):
from fastapi.testclient import TestClient
from src.db import get_system_db, close_system_db
from app.main import app
conn = get_system_db()
try:
admin_uid, admin_sess = _make_user_and_session(conn, "admin@t", "admin")
other_uid, _ = _make_user_and_session(conn, "victim@t", "analyst")
_make_pat_row(conn, other_uid, name="laptop", last_used_ip="9.9.9.9",
last_used_ago_days=2)
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.get(
"/auth/admin/tokens",
headers={"Authorization": f"Bearer {admin_sess}"},
)
assert resp.status_code == 200, resp.text
items = resp.json()
assert len(items) >= 1
row = [r for r in items if r["name"] == "laptop"][0]
assert row["user_id"] == other_uid
assert row["user_email"] == "victim@t"
assert row["last_used_ip"] == "9.9.9.9"
assert row["last_used_at"] # not None
def test_non_admin_cannot_list_admin_tokens(fresh_db):
from fastapi.testclient import TestClient
from src.db import get_system_db, close_system_db
from app.main import app
conn = get_system_db()
try:
_, analyst_sess = _make_user_and_session(conn, "u@t", "analyst")
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.get(
"/auth/admin/tokens",
headers={"Authorization": f"Bearer {analyst_sess}"},
)
assert resp.status_code == 403
# ── Admin revoke ──────────────────────────────────────────────────────────
def test_admin_can_revoke_another_users_token(fresh_db):
from fastapi.testclient import TestClient
from src.db import get_system_db, close_system_db
from src.repositories.access_tokens import AccessTokenRepository
from app.main import app
conn = get_system_db()
try:
admin_uid, admin_sess = _make_user_and_session(conn, "admin@t", "admin")
other_uid, _ = _make_user_and_session(conn, "victim@t", "analyst")
tid = _make_pat_row(conn, other_uid, name="to-kill")
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.delete(
f"/auth/admin/tokens/{tid}",
headers={"Authorization": f"Bearer {admin_sess}"},
)
assert resp.status_code == 204
conn = get_system_db()
try:
row = AccessTokenRepository(conn).get_by_id(tid)
assert row is not None
assert row["revoked_at"] is not None
finally:
conn.close()
close_system_db()
def test_non_admin_can_create_pat_via_tokens_page_api(fresh_db):
"""The /tokens create-modal submits POST /auth/tokens (name + expires)."""
from fastapi.testclient import TestClient
from src.db import get_system_db, close_system_db
from src.repositories.access_tokens import AccessTokenRepository
from app.main import app
conn = get_system_db()
try:
uid, sess = _make_user_and_session(conn, "user@t", "analyst")
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.post(
"/auth/tokens",
headers={"Authorization": f"Bearer {sess}"},
json={"name": "laptop", "expires_in_days": 30},
)
assert resp.status_code == 201, resp.text
data = resp.json()
assert data["name"] == "laptop"
assert data["token"] # raw JWT returned exactly once
assert data["prefix"]
# It must be owned by the creator
conn = get_system_db()
try:
row = AccessTokenRepository(conn).get_by_id(data["id"])
finally:
conn.close()
close_system_db()
assert row is not None
assert row["user_id"] == uid
assert row["name"] == "laptop"
def test_non_admin_cannot_admin_revoke(fresh_db):
from fastapi.testclient import TestClient
from src.db import get_system_db, close_system_db
from app.main import app
conn = get_system_db()
try:
_, analyst_sess = _make_user_and_session(conn, "u@t", "analyst")
other_uid, _ = _make_user_and_session(conn, "other@t", "analyst")
tid = _make_pat_row(conn, other_uid, name="keep")
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.delete(
f"/auth/admin/tokens/{tid}",
headers={"Authorization": f"Bearer {analyst_sess}"},
)
assert resp.status_code == 403

30
tests/test_cli_admin.py
View file

@ -156,6 +156,36 @@ class TestMetadataShow:
assert result.exit_code == 1
def test_admin_set_role_invokes_patch(monkeypatch):
"""`da admin set-role` sends PATCH to /api/users/{id} with role."""
import httpx
from cli.commands.admin import admin_app
from typer.testing import CliRunner
captured = {}
def fake_patch(path, json=None, **kwargs):
captured["path"] = path
captured["json"] = json
return httpx.Response(200, json={
"id": "abc", "email": "x@y.z", "name": "X",
"role": json.get("role") if json else "viewer",
"active": True, "created_at": "", "deactivated_at": None,
})
from cli import client as cli_client
monkeypatch.setattr(cli_client, "api_patch", fake_patch, raising=False)
# patch admin.api_patch too since admin.py imports names
from cli.commands import admin as admin_mod
monkeypatch.setattr(admin_mod, "api_patch", fake_patch, raising=False)
runner = CliRunner()
result = runner.invoke(admin_app, ["set-role", "abc", "analyst"])
assert result.exit_code == 0
assert captured["path"] == "/api/users/abc"
assert captured["json"] == {"role": "analyst"}
class TestMetadataApply:
def test_metadata_apply_dry_run(self, tmp_path):
proposal = {

139
tests/test_cli_artifacts.py Normal file
View file

@ -0,0 +1,139 @@
"""Tests for #9 — CLI artifact + install script endpoints."""
import os
from pathlib import Path
import tempfile
def test_cli_install_script_bakes_server_url(monkeypatch):
from fastapi.testclient import TestClient
from app.main import app
client = TestClient(app, base_url="https://agnes.example.com")
resp = client.get("/cli/install.sh", headers={"host": "agnes.example.com"})
assert resp.status_code == 200
assert resp.headers["content-type"].startswith("text/")
body = resp.text
assert "https://agnes.example.com" in body or "agnes.example.com" in body
assert "pip install" in body or "uv tool install" in body
def test_cli_download_returns_wheel_or_404(monkeypatch):
from fastapi.testclient import TestClient
from app.main import app
client = TestClient(app)
resp = client.get("/cli/download")
# Either serve the wheel or return a clear 404 telling where to find it.
assert resp.status_code in (200, 404)
if resp.status_code == 200:
assert resp.headers["content-disposition"].startswith("attachment")
def test_cli_download_serves_wheel_when_present(monkeypatch, tmp_path):
"""Put a fake wheel and confirm the endpoint serves it."""
wheel = tmp_path / "agnes_fake-1.0-py3-none-any.whl"
wheel.write_bytes(b"PK\x03\x04fake-wheel-bytes")
monkeypatch.setenv("AGNES_CLI_DIST_DIR", str(tmp_path))
from fastapi.testclient import TestClient
from app.main import app
client = TestClient(app)
resp = client.get("/cli/download")
assert resp.status_code == 200
assert resp.content.startswith(b"PK")
def test_cli_agnes_whl_alias_serves_same_bytes_as_download(monkeypatch, tmp_path):
"""`/cli/agnes.whl` is a stable alias over `/cli/download` whose URL path
ends in `.whl`, which `uv tool install` requires to treat the resource as
a wheel. Both endpoints must serve identical bytes."""
wheel = tmp_path / "agnes_fake-1.0-py3-none-any.whl"
wheel.write_bytes(b"PK\x03\x04fake-wheel-bytes-agnes")
monkeypatch.setenv("AGNES_CLI_DIST_DIR", str(tmp_path))
from fastapi.testclient import TestClient
from app.main import app
client = TestClient(app)
resp_alias = client.get("/cli/agnes.whl")
assert resp_alias.status_code == 200
assert resp_alias.headers["content-type"] == "application/octet-stream"
assert resp_alias.content == wheel.read_bytes()
resp_download = client.get("/cli/download")
assert resp_download.status_code == 200
assert resp_alias.content == resp_download.content
def test_cli_agnes_whl_alias_404_when_no_wheel(monkeypatch, tmp_path):
"""Alias returns 404 with a helpful message when no wheel is present."""
monkeypatch.setenv("AGNES_CLI_DIST_DIR", str(tmp_path))
from fastapi.testclient import TestClient
from app.main import app
client = TestClient(app)
resp = client.get("/cli/agnes.whl")
assert resp.status_code == 404
def test_install_page_renders_with_server_url():
from fastapi.testclient import TestClient
from app.main import app
client = TestClient(app)
resp = client.get("/install", headers={"host": "agnes.test", "Accept": "text/html"})
assert resp.status_code == 200
assert "agnes.test" in resp.text
assert "da auth whoami" in resp.text
def test_safe_url_re_accepts_reverse_proxy_path_prefix():
"""Reverse-proxy deployments have request.base_url with a path segment
(e.g. https://host/agnes/). The regex must accept that; the install.sh
endpoint previously rejected it with 400."""
from app.api.cli_artifacts import _SAFE_URL_RE
# Path prefix (Agnes behind a reverse proxy with location /agnes/)
assert _SAFE_URL_RE.match("https://agnes.example.com/agnes")
assert _SAFE_URL_RE.match("https://agnes.example.com/agnes/")
# Underscores in Docker Compose hostnames
assert _SAFE_URL_RE.match("http://agnes_web:8000")
# IPv6 literal
assert _SAFE_URL_RE.match("http://[::1]:8000")
# Still rejects obvious bad shapes
assert not _SAFE_URL_RE.match("https://agnes.example.com/agnes;rm -rf /")
assert not _SAFE_URL_RE.match("ftp://agnes.example.com/")
assert not _SAFE_URL_RE.match("https://agnes.example.com/?x=$(id)")
def test_safe_url_re_rejects_trailing_newline_bypass():
"""Python's `$` matches immediately before a trailing `\\n`, so a naïve
allowlist with `^...$` would accept "good.example.com\\n$(rm -rf /)"
and allow shell-injection in the generated install.sh. Anchoring with
`\\Z` closes that bypass. Covers both allowlists."""
from app.api.cli_artifacts import _SAFE_URL_RE, _SAFE_VERSION_RE
# Trailing newline after an otherwise-valid URL must be rejected.
assert not _SAFE_URL_RE.match("https://good.example.com\n")
assert not _SAFE_URL_RE.match("https://good.example.com\n$(rm -rf /)")
assert not _SAFE_URL_RE.match("http://host:8000\nevil")
# Sanity: the clean form still matches.
assert _SAFE_URL_RE.match("https://good.example.com")
# Version allowlist — same class of bypass.
assert not _SAFE_VERSION_RE.match("1.2.3\n")
assert not _SAFE_VERSION_RE.match("1.2.3\nrm")
assert _SAFE_VERSION_RE.match("1.2.3")
def test_cli_install_sh_accepts_base_url_with_path_prefix(monkeypatch):
"""Reverse-proxy deployments (Caddy/Nginx routing /agnes/* to Agnes)
surface a request.base_url like 'https://host/agnes/'. The handler
previously 400'd on that. We call the handler directly with a stub
request so we don't need a mounted ASGI proxy in tests."""
import asyncio
from types import SimpleNamespace
from starlette.datastructures import URL
from app.api.cli_artifacts import cli_install_script
# Minimal Request stub — cli_install_script only needs .base_url.
stub = SimpleNamespace(base_url=URL("https://agnes.example.com/agnes/"))
result = asyncio.run(cli_install_script(stub)) # returns the script body
assert isinstance(result, str)
assert "https://agnes.example.com/agnes" in result

165
tests/test_cli_auth.py
View file

@ -37,7 +37,8 @@ class TestAuthLogin:
})
with patch("cli.commands.auth.api_post", return_value=mock_resp):
with patch("cli.commands.auth.save_token") as mock_save:
result = runner.invoke(app, ["auth", "login", "--email", "alice@example.com"])
# Empty password (simulates magic-link / OAuth account) — still 200 from server
result = runner.invoke(app, ["auth", "login", "--email", "alice@example.com"], input="\n")
assert result.exit_code == 0
assert "alice@example.com" in result.output
mock_save.assert_called_once_with("tok123", "alice@example.com", "analyst")
@ -46,14 +47,14 @@ class TestAuthLogin:
"""Login with bad credentials exits with error."""
mock_resp = _make_response(401, {"detail": "Invalid credentials"})
with patch("cli.commands.auth.api_post", return_value=mock_resp):
result = runner.invoke(app, ["auth", "login", "--email", "bad@example.com"])
result = runner.invoke(app, ["auth", "login", "--email", "bad@example.com"], input="\n")
assert result.exit_code == 1
assert "Login failed" in result.output
def test_login_connection_error(self):
"""Login propagates connection errors cleanly."""
with patch("cli.commands.auth.api_post", side_effect=Exception("Connection refused")):
result = runner.invoke(app, ["auth", "login", "--email", "alice@example.com"])
result = runner.invoke(app, ["auth", "login", "--email", "alice@example.com"], input="\n")
assert result.exit_code == 1
assert "Connection error" in result.output
@ -68,6 +69,112 @@ class TestAuthLogout:
mock_clear.assert_called_once()
class TestAuthImportToken:
def _make_jwt(self, email="alice@example.com", role="analyst", typ="pat"):
import jwt as pyjwt
return pyjwt.encode(
{"email": email, "role": role, "typ": typ, "sub": "u-1"},
"unused",
algorithm="HS256",
)
def _mock_verify(self, status_code=200, json_data=None):
"""Build a patcher for cli.commands.auth.httpx.Client that returns a canned response."""
resp = _make_response(status_code, json_data or {})
mock_client = MagicMock()
mock_client.__enter__.return_value = mock_client
mock_client.__exit__.return_value = False
mock_client.get.return_value = resp
return patch("cli.commands.auth.httpx.Client", return_value=mock_client)
def test_import_token_success_writes_canonical_format(self, tmp_path, monkeypatch):
"""Valid JWT + 200 from server -> canonical token.json on disk."""
monkeypatch.setenv("DA_SERVER", "http://example.test")
token = self._make_jwt(email="bob@example.com", role="admin")
with self._mock_verify(200):
result = runner.invoke(app, ["auth", "import-token", "--token", token])
assert result.exit_code == 0, result.output
assert "bob@example.com" in result.output
assert "admin" in result.output
token_file = tmp_path / "config" / "token.json"
assert token_file.exists()
data = json.loads(token_file.read_text())
assert data == {"access_token": token, "email": "bob@example.com", "role": "admin"}
def test_import_token_401_does_not_overwrite_existing(self, tmp_path, monkeypatch):
"""A 401 response aborts import and leaves the prior token file untouched."""
monkeypatch.setenv("DA_SERVER", "http://example.test")
existing = {"access_token": "keep-me", "email": "old@example.com", "role": "viewer"}
token_file = tmp_path / "config" / "token.json"
token_file.write_text(json.dumps(existing))
token = self._make_jwt()
with self._mock_verify(401, {"detail": "Token revoked"}):
result = runner.invoke(app, ["auth", "import-token", "--token", token])
assert result.exit_code == 1
assert "Token rejected by server" in result.output
assert "Token revoked" in result.output
# Existing file must be intact.
assert json.loads(token_file.read_text()) == existing
def test_import_token_with_server_flag_persists_server_to_config_yaml(
self, tmp_path, monkeypatch
):
"""Passing --server should write `server: URL` to ~/.config/da/config.yaml
so the user never has to configure the server in a separate step."""
# No DA_SERVER env var — rely entirely on the --server flag for persistence.
monkeypatch.delenv("DA_SERVER", raising=False)
token = self._make_jwt(email="dave@example.com", role="analyst")
with self._mock_verify(200):
result = runner.invoke(
app,
[
"auth", "import-token",
"--token", token,
"--server", "https://agnes.example.com",
],
)
assert result.exit_code == 0, result.output
config_file = tmp_path / "config" / "config.yaml"
assert config_file.exists(), "config.yaml must be written when --server is passed"
import yaml
cfg = yaml.safe_load(config_file.read_text())
assert cfg.get("server") == "https://agnes.example.com"
def test_import_token_claim_fallback_via_cli_overrides(self, tmp_path, monkeypatch):
"""Missing email/role claims -> refuse without overrides, accept with them."""
import jwt as pyjwt
monkeypatch.setenv("DA_SERVER", "http://example.test")
# JWT without email/role claims — simulates a malformed or minimal token.
token = pyjwt.encode({"sub": "u-1", "typ": "pat"}, "unused", algorithm="HS256")
with self._mock_verify(200):
fail_result = runner.invoke(app, ["auth", "import-token", "--token", token])
assert fail_result.exit_code == 1
assert "missing" in fail_result.output.lower()
with self._mock_verify(200):
ok_result = runner.invoke(
app,
[
"auth", "import-token",
"--token", token,
"--email", "carol@example.com",
"--role", "analyst",
],
)
assert ok_result.exit_code == 0, ok_result.output
token_file = tmp_path / "config" / "token.json"
data = json.loads(token_file.read_text())
assert data == {"access_token": token, "email": "carol@example.com", "role": "analyst"}
class TestAuthWhoami:
def test_whoami_no_token(self):
"""Whoami exits when no token is stored."""
@ -97,3 +204,55 @@ class TestAuthWhoami:
result = runner.invoke(app, ["auth", "whoami"])
# May succeed or fail depending on jwt decode — either way no traceback
assert result.exit_code in (0, 1)
def test_da_login_sends_password(monkeypatch):
import httpx
from typer.testing import CliRunner
from cli.commands import auth as auth_mod
captured = {}
def fake_post(path, json=None, **kwargs):
captured["path"] = path
captured["json"] = json
return httpx.Response(200, json={
"access_token": "tok", "email": "u@t", "role": "analyst",
"user_id": "u1", "token_type": "bearer",
})
monkeypatch.setattr(auth_mod, "api_post", fake_post, raising=False)
runner = CliRunner()
# Provide email and password via stdin (typer prompts)
result = runner.invoke(auth_mod.auth_app, ["login"], input="u@t\nhunter2\n")
assert result.exit_code == 0, result.output
assert captured["path"] == "/auth/token"
assert captured["json"] == {"email": "u@t", "password": "hunter2"}
def test_da_auth_token_create_calls_api(monkeypatch):
import httpx
from typer.testing import CliRunner
from cli.commands.auth import auth_app
from cli.commands import tokens as tok_mod
captured = {}
def fake_post(path, json=None, **kwargs):
captured["path"] = path
captured["json"] = json
return httpx.Response(201, json={
"id": "abc", "name": json["name"], "prefix": "XXXXXXXX",
"token": "raw-token-once",
"expires_at": None, "created_at": "2026-04-21T00:00:00+00:00",
})
monkeypatch.setattr(tok_mod, "api_post", fake_post, raising=False)
runner = CliRunner()
result = runner.invoke(auth_app, ["token", "create", "--name", "laptop", "--ttl", "30d"])
assert result.exit_code == 0, result.output
assert captured["path"] == "/auth/tokens"
assert captured["json"] == {"name": "laptop", "expires_in_days": 30}
assert "raw-token-once" in result.output

959
tests/test_connector_kit_poc.py Normal file
View file

@ -0,0 +1,959 @@
"""
Proof-of-concept: Connector Kit architecture validation.
Tests that the proposed Connector Protocol + Runtime model is:
1. Implementable in Python (Protocol, Cap flags, partial implementation)
2. Arrow RecordBatch iteration works with DuckDB (zero-copy)
3. ConnectorRuntime can build extract.duckdb from any connector
4. Schema evolution detection works via Arrow schema diff
5. A real connector can be written in ~50 lines
6. Incremental state tracking works
7. Manifest validation works
8. Discovery → read pipeline is end-to-end functional
"""
import asyncio
import os
import shutil
import tempfile
from dataclasses import dataclass, field
from enum import Flag, auto
from pathlib import Path
from typing import AsyncIterator, Iterator, Protocol, runtime_checkable
import duckdb
import pyarrow as pa
import pyarrow.parquet as pq
import pytest
import yaml
# ============================================================================
# Layer 2: Connector Protocol (the contract)
# ============================================================================
class Cap(Flag):
"""Connector capabilities — declare what you support."""
DISCOVER = auto()
READ = auto()
STREAM = auto()
REMOTE = auto()
WRITE = auto()
@dataclass
class TableInfo:
name: str
schema: pa.Schema
capabilities: Cap
primary_key: list[str] | None = None
description: str = ""
@dataclass
class ReadOptions:
columns: list[str] | None = None
filter: dict | None = None
incremental_key: str | None = None
incremental_value: str | None = None
batch_size: int = 10_000
@dataclass
class RemoteAttachInfo:
extension: str
url: str
token_env: str
@runtime_checkable
class Connector(Protocol):
@property
def capabilities(self) -> Cap: ...
def discover(self) -> list[TableInfo]: ...
def read(self, table: str, options: ReadOptions) -> Iterator[pa.RecordBatch]: ...
# ============================================================================
# Layer 3: Connector Runtime (the SDK — replaces manual boilerplate)
# ============================================================================
@dataclass
class ExtractStats:
tables_extracted: int = 0
tables_failed: int = 0
total_rows: int = 0
schema_changes: list[str] = field(default_factory=list)
errors: list[str] = field(default_factory=list)
class ConnectorRuntime:
"""Handles extract.duckdb lifecycle — what every connector does manually today."""
def __init__(self, output_dir: Path):
self.output_dir = output_dir
self.data_dir = output_dir / "data"
self.db_path = output_dir / "extract.duckdb"
self.state_path = output_dir / ".state.yaml"
self.data_dir.mkdir(parents=True, exist_ok=True)
def run(self, connector: Connector, tables: list[str] | None = None) -> ExtractStats:
stats = ExtractStats()
# 1. Discovery
available: list[TableInfo] = []
if Cap.DISCOVER in connector.capabilities:
available = connector.discover()
# If no tables specified, extract all discovered
if tables is None:
tables = [t.name for t in available if Cap.READ in t.capabilities]
# 2. Schema evolution check
for table_name in tables:
table_info = self._find_table(available, table_name)
if table_info:
change = self._check_schema_evolution(table_name, table_info.schema)
if change:
stats.schema_changes.append(change)
# 3. Extract via read()
if Cap.READ in connector.capabilities:
for table_name in tables:
try:
options = self._build_read_options(table_name, available)
rows = self._extract_table(connector, table_name, options)
stats.tables_extracted += 1
stats.total_rows += rows
except Exception as e:
stats.tables_failed += 1
stats.errors.append(f"{table_name}: {e}")
# 4. Remote attach (if supported)
if Cap.REMOTE in connector.capabilities:
try:
info = connector.remote() # type: ignore[attr-defined]
self._write_remote_attach(info)
except Exception as e:
stats.errors.append(f"remote_attach: {e}")
# 5. Build extract.duckdb (_meta + views)
self._build_extract_db(available, tables)
# 6. Save incremental state
self._save_state(tables)
return stats
def _extract_table(self, connector: Connector, table: str, options: ReadOptions) -> int:
"""Extract a table via Arrow RecordBatch iterator → Parquet."""
parquet_path = self.data_dir / f"{table}.parquet"
writer = None
total_rows = 0
for batch in connector.read(table, options):
if writer is None:
writer = pq.ParquetWriter(str(parquet_path), batch.schema)
writer.write_batch(batch)
total_rows += batch.num_rows
if writer:
writer.close()
return total_rows
def _build_extract_db(self, available: list[TableInfo], tables: list[str]):
"""Build extract.duckdb with _meta and views — atomic swap."""
tmp_db = self.output_dir / "extract.duckdb.tmp"
if tmp_db.exists():
tmp_db.unlink()
con = duckdb.connect(str(tmp_db))
try:
# _meta table
con.execute("""
CREATE TABLE _meta (
table_name VARCHAR NOT NULL,
description VARCHAR,
rows BIGINT,
size_bytes BIGINT,
extracted_at TIMESTAMP DEFAULT current_timestamp,
query_mode VARCHAR DEFAULT 'local',
schema_json VARCHAR
)
""")
for table_name in tables:
parquet_path = self.data_dir / f"{table_name}.parquet"
if parquet_path.exists():
# Create view pointing to parquet
con.execute(
f'CREATE VIEW "{table_name}" AS '
f"SELECT * FROM read_parquet('{parquet_path}')"
)
# Get row count + size
rows = con.execute(f'SELECT count(*) FROM "{table_name}"').fetchone()[0]
size = parquet_path.stat().st_size
# Find description and schema
info = self._find_table(available, table_name)
desc = info.description if info else ""
schema_json = info.schema.to_string() if info else ""
con.execute(
"INSERT INTO _meta (table_name, description, rows, size_bytes, schema_json) "
"VALUES (?, ?, ?, ?, ?)",
[table_name, desc, rows, size, schema_json],
)
finally:
con.close()
# Atomic swap
if self.db_path.exists():
self.db_path.unlink()
# Clean WAL if exists
wal = Path(str(tmp_db) + ".wal")
if wal.exists():
wal.unlink()
tmp_db.rename(self.db_path)
def _find_table(self, available: list[TableInfo], name: str) -> TableInfo | None:
return next((t for t in available if t.name == name), None)
def _check_schema_evolution(self, table: str, new_schema: pa.Schema) -> str | None:
"""Detect schema changes by comparing Arrow schemas."""
schema_file = self.output_dir / f".schema_{table}.arrow"
if schema_file.exists():
reader = pa.ipc.open_stream(schema_file.read_bytes())
old_schema = reader.schema
if old_schema != new_schema:
# Diff: added, removed, changed fields
old_names = set(old_schema.names)
new_names = set(new_schema.names)
added = new_names - old_names
removed = old_names - new_names
msg = f"{table}: "
if added:
msg += f"+{added} "
if removed:
msg += f"-{removed} "
# Check type changes for common fields
for name in old_names & new_names:
old_type = old_schema.field(name).type
new_type = new_schema.field(name).type
if old_type != new_type:
msg += f"{name}:{old_type}→{new_type} "
# Save new schema
self._save_schema(table, new_schema)
return msg.strip()
else:
self._save_schema(table, new_schema)
return None
def _save_schema(self, table: str, schema: pa.Schema):
"""Serialize Arrow schema via IPC stream (compatible with all PyArrow versions)."""
schema_file = self.output_dir / f".schema_{table}.arrow"
sink = pa.BufferOutputStream()
writer = pa.ipc.new_stream(sink, schema)
writer.close()
schema_file.write_bytes(sink.getvalue().to_pybytes())
def _build_read_options(self, table: str, available: list[TableInfo]) -> ReadOptions:
"""Build ReadOptions with incremental state if available."""
options = ReadOptions()
state = self._load_state()
if table in state:
options.incremental_key = state[table].get("incremental_key")
options.incremental_value = state[table].get("incremental_value")
return options
def _load_state(self) -> dict:
if self.state_path.exists():
return yaml.safe_load(self.state_path.read_text()) or {}
return {}
def _save_state(self, tables: list[str]):
state = self._load_state()
for table in tables:
if table not in state:
state[table] = {}
state[table]["last_extracted"] = str(duckdb.query("SELECT current_timestamp").fetchone()[0])
self.state_path.write_text(yaml.dump(state))
def _write_remote_attach(self, info: RemoteAttachInfo):
"""Write _remote_attach info for orchestrator."""
# This gets added to extract.duckdb in _build_extract_db
# For POC, store as yaml; real impl writes to DuckDB
ra_path = self.output_dir / ".remote_attach.yaml"
ra_path.write_text(
yaml.dump({"extension": info.extension, "url": info.url, "token_env": info.token_env})
)
# ============================================================================
# Example connectors (proving the contract works)
# ============================================================================
class SampleAPIConnector:
"""
A sample connector simulating an HTTP API source.
Proves: ~50 lines for a complete connector implementation.
"""
capabilities = Cap.DISCOVER | Cap.READ
ORDERS_SCHEMA = pa.schema(
[
pa.field("id", pa.int64()),
pa.field("customer", pa.string()),
pa.field("amount", pa.float64()),
pa.field("date", pa.string()),
]
)
USERS_SCHEMA = pa.schema(
[
pa.field("id", pa.int64()),
pa.field("name", pa.string()),
pa.field("email", pa.string()),
]
)
# Simulated API data
_data = {
"orders": [
{"id": 1, "customer": "Alice", "amount": 100.0, "date": "2026-01-15"},
{"id": 2, "customer": "Bob", "amount": 250.0, "date": "2026-02-01"},
{"id": 3, "customer": "Carol", "amount": 75.5, "date": "2026-03-10"},
],
"users": [
{"id": 1, "name": "Alice", "email": "alice@example.com"},
{"id": 2, "name": "Bob", "email": "bob@example.com"},
],
}
def discover(self) -> list[TableInfo]:
return [
TableInfo(
name="orders",
schema=self.ORDERS_SCHEMA,
capabilities=Cap.READ,
primary_key=["id"],
description="Sales orders",
),
TableInfo(
name="users",
schema=self.USERS_SCHEMA,
capabilities=Cap.READ,
primary_key=["id"],
description="Registered users",
),
]
def read(self, table: str, options: ReadOptions) -> Iterator[pa.RecordBatch]:
data = self._data.get(table, [])
schema = self.ORDERS_SCHEMA if table == "orders" else self.USERS_SCHEMA
# Simulate batched reading (batch_size controls chunking)
for i in range(0, len(data), options.batch_size):
chunk = data[i : i + options.batch_size]
arrays = [pa.array([row[col] for row in chunk], type=schema.field(col).type) for col in schema.names]
yield pa.RecordBatch.from_arrays(arrays, schema=schema)
class StreamingConnector:
"""Proves: async stream capability works."""
capabilities = Cap.DISCOVER | Cap.STREAM
EVENTS_SCHEMA = pa.schema(
[
pa.field("event_id", pa.string()),
pa.field("type", pa.string()),
pa.field("payload", pa.string()),
]
)
def discover(self) -> list[TableInfo]:
return [
TableInfo(
name="events",
schema=self.EVENTS_SCHEMA,
capabilities=Cap.STREAM,
description="Real-time events",
)
]
async def stream(self, table: str) -> AsyncIterator[pa.RecordBatch]:
"""Simulate webhook events arriving."""
events = [
{"event_id": "e1", "type": "created", "payload": '{"issue": "PROJ-1"}'},
{"event_id": "e2", "type": "updated", "payload": '{"issue": "PROJ-2"}'},
{"event_id": "e3", "type": "deleted", "payload": '{"issue": "PROJ-3"}'},
]
for event in events:
arrays = [pa.array([event[col]], type=self.EVENTS_SCHEMA.field(col).type) for col in self.EVENTS_SCHEMA.names]
yield pa.RecordBatch.from_arrays(arrays, schema=self.EVENTS_SCHEMA)
class RemoteOnlyConnector:
"""Proves: remote-only connector (like BigQuery) works."""
capabilities = Cap.DISCOVER | Cap.REMOTE
def discover(self) -> list[TableInfo]:
return [
TableInfo(
name="big_table",
schema=pa.schema([pa.field("id", pa.int64()), pa.field("value", pa.string())]),
capabilities=Cap.REMOTE,
description="Remote-only table, queries go to source",
)
]
def remote(self) -> RemoteAttachInfo:
return RemoteAttachInfo(
extension="bigquery",
url="project_id=my-project",
token_env="GOOGLE_APPLICATION_CREDENTIALS",
)
# ============================================================================
# Tests
# ============================================================================
class TestCapabilityFlags:
"""Test 1: Cap Flag enum works for declaration and checking."""
def test_flag_composition(self):
caps = Cap.DISCOVER | Cap.READ | Cap.REMOTE
assert Cap.DISCOVER in caps
assert Cap.READ in caps
assert Cap.REMOTE in caps
assert Cap.STREAM not in caps
assert Cap.WRITE not in caps
def test_per_table_capabilities(self):
info = TableInfo(
name="orders",
schema=pa.schema([pa.field("id", pa.int64())]),
capabilities=Cap.READ | Cap.STREAM,
)
assert Cap.READ in info.capabilities
assert Cap.STREAM in info.capabilities
assert Cap.WRITE not in info.capabilities
def test_flag_iteration(self):
"""Can iterate individual flags from a composite."""
caps = Cap.DISCOVER | Cap.READ | Cap.STREAM
individual = list(caps)
assert len(individual) == 3
assert Cap.DISCOVER in individual
class TestProtocolCompliance:
"""Test 2: Protocol type checking works at runtime."""
def test_sample_connector_is_connector(self):
c = SampleAPIConnector()
assert isinstance(c, Connector)
def test_streaming_connector_partial_protocol(self):
"""StreamingConnector doesn't implement read() — that's OK.
Protocol is structural, not enforced for methods you don't use."""
c = StreamingConnector()
assert hasattr(c, "capabilities")
assert hasattr(c, "discover")
assert Cap.STREAM in c.capabilities
def test_remote_connector_is_valid(self):
c = RemoteOnlyConnector()
assert hasattr(c, "discover")
assert hasattr(c, "remote")
assert Cap.REMOTE in c.capabilities
class TestArrowIntegration:
"""Test 3: Arrow RecordBatch → DuckDB zero-copy works."""
def test_record_batch_to_duckdb(self):
"""DuckDB can query Arrow RecordBatches directly."""
schema = pa.schema([pa.field("id", pa.int64()), pa.field("name", pa.string())])
batch = pa.RecordBatch.from_arrays(
[pa.array([1, 2, 3]), pa.array(["a", "b", "c"])],
schema=schema,
)
con = duckdb.connect()
result = con.execute("SELECT * FROM batch WHERE id > 1").fetchall()
assert len(result) == 2
assert result[0] == (2, "b")
def test_record_batch_iterator_to_duckdb(self):
"""DuckDB can consume an iterator of RecordBatches."""
schema = pa.schema([pa.field("value", pa.float64())])
def generate_batches():
for i in range(3):
yield pa.RecordBatch.from_arrays(
[pa.array([float(i * 10 + j) for j in range(5)])],
schema=schema,
)
reader = pa.RecordBatchReader.from_batches(schema, generate_batches())
con = duckdb.connect()
result = con.execute("SELECT count(*), sum(value) FROM reader").fetchone()
assert result[0] == 15 # 3 batches * 5 rows
assert result[1] == sum(float(i * 10 + j) for i in range(3) for j in range(5))
def test_arrow_to_parquet_roundtrip(self):
"""Arrow → Parquet → DuckDB roundtrip preserves data."""
schema = pa.schema(
[
pa.field("id", pa.int64()),
pa.field("amount", pa.float64()),
pa.field("label", pa.string()),
]
)
batch = pa.RecordBatch.from_arrays(
[pa.array([1, 2]), pa.array([99.9, 200.0]), pa.array(["x", "y"])],
schema=schema,
)
with tempfile.NamedTemporaryFile(suffix=".parquet", delete=False) as f:
pq.write_table(pa.Table.from_batches([batch]), f.name)
con = duckdb.connect()
result = con.execute(f"SELECT * FROM read_parquet('{f.name}')").fetchall()
assert result == [(1, 99.9, "x"), (2, 200.0, "y")]
os.unlink(f.name)
class TestConnectorRuntime:
"""Test 4: Full runtime pipeline — connector → extract.duckdb."""
@pytest.fixture
def output_dir(self, tmp_path):
return tmp_path / "extract_test"
def test_full_extract_pipeline(self, output_dir):
"""End-to-end: connector → runtime → extract.duckdb with _meta + views."""
connector = SampleAPIConnector()
runtime = ConnectorRuntime(output_dir)
stats = runtime.run(connector)
# Stats are correct
assert stats.tables_extracted == 2
assert stats.tables_failed == 0
assert stats.total_rows == 5 # 3 orders + 2 users
assert stats.errors == []
# extract.duckdb exists and is valid
db_path = output_dir / "extract.duckdb"
assert db_path.exists()
con = duckdb.connect(str(db_path), read_only=True)
# _meta table has both tables
meta = con.execute("SELECT table_name, rows, description FROM _meta ORDER BY table_name").fetchall()
assert len(meta) == 2
assert meta[0] == ("orders", 3, "Sales orders")
assert meta[1] == ("users", 2, "Registered users")
# Views work — can query data through extract.duckdb
orders = con.execute("SELECT * FROM orders ORDER BY id").fetchall()
assert len(orders) == 3
assert orders[0] == (1, "Alice", 100.0, "2026-01-15")
users = con.execute("SELECT * FROM users ORDER BY id").fetchall()
assert len(users) == 2
assert users[0][1] == "Alice"
# Cross-table query works
result = con.execute("""
SELECT u.name, SUM(o.amount) as total
FROM orders o JOIN users u ON o.customer = u.name
GROUP BY u.name ORDER BY total DESC
""").fetchall()
assert result[0] == ("Bob", 250.0)
assert result[1] == ("Alice", 100.0)
con.close()
def test_selective_table_extract(self, output_dir):
"""Can extract specific tables only."""
connector = SampleAPIConnector()
runtime = ConnectorRuntime(output_dir)
stats = runtime.run(connector, tables=["orders"])
assert stats.tables_extracted == 1
assert stats.total_rows == 3
con = duckdb.connect(str(output_dir / "extract.duckdb"), read_only=True)
tables = con.execute("SELECT table_name FROM _meta").fetchall()
assert tables == [("orders",)]
con.close()
def test_incremental_state_tracking(self, output_dir):
"""Runtime saves and loads incremental state between runs."""
connector = SampleAPIConnector()
runtime = ConnectorRuntime(output_dir)
# First run
runtime.run(connector, tables=["orders"])
# State file exists
state_path = output_dir / ".state.yaml"
assert state_path.exists()
state = yaml.safe_load(state_path.read_text())
assert "orders" in state
assert "last_extracted" in state["orders"]
# Second run — state persists
runtime2 = ConnectorRuntime(output_dir)
runtime2.run(connector, tables=["orders"])
state2 = yaml.safe_load(state_path.read_text())
assert "orders" in state2
def test_empty_table_handling(self, output_dir):
"""Connector that yields nothing for a table doesn't crash."""
class EmptyConnector:
capabilities = Cap.DISCOVER | Cap.READ
def discover(self) -> list[TableInfo]:
return [
TableInfo(
name="empty",
schema=pa.schema([pa.field("id", pa.int64())]),
capabilities=Cap.READ,
)
]
def read(self, table: str, options: ReadOptions) -> Iterator[pa.RecordBatch]:
return iter([]) # No data
runtime = ConnectorRuntime(output_dir)
stats = runtime.run(EmptyConnector())
# Extracted 0 rows, but no failure
assert stats.tables_extracted == 1
assert stats.total_rows == 0
assert stats.errors == []
def test_error_in_one_table_doesnt_stop_others(self, output_dir):
"""Partial failure: one table fails, others still extract."""
class PartialFailConnector:
capabilities = Cap.DISCOVER | Cap.READ
def discover(self) -> list[TableInfo]:
return [
TableInfo("good", pa.schema([pa.field("id", pa.int64())]), Cap.READ),
TableInfo("bad", pa.schema([pa.field("id", pa.int64())]), Cap.READ),
]
def read(self, table: str, options: ReadOptions) -> Iterator[pa.RecordBatch]:
if table == "bad":
raise ConnectionError("API timeout")
yield pa.RecordBatch.from_arrays(
[pa.array([1, 2, 3])],
schema=pa.schema([pa.field("id", pa.int64())]),
)
runtime = ConnectorRuntime(output_dir)
stats = runtime.run(PartialFailConnector())
assert stats.tables_extracted == 1
assert stats.tables_failed == 1
assert "bad: API timeout" in stats.errors[0]
class TestSchemaEvolution:
"""Test 5: Schema change detection via Arrow schema diff."""
def test_detect_added_column(self, tmp_path):
output_dir = tmp_path / "schema_test"
runtime = ConnectorRuntime(output_dir)
# V1 schema
schema_v1 = pa.schema([pa.field("id", pa.int64()), pa.field("name", pa.string())])
runtime._save_schema("orders", schema_v1)
# V2 schema — added column
schema_v2 = pa.schema(
[
pa.field("id", pa.int64()),
pa.field("name", pa.string()),
pa.field("email", pa.string()),
]
)
change = runtime._check_schema_evolution("orders", schema_v2)
assert change is not None
assert "email" in change
assert "+" in change
def test_detect_removed_column(self, tmp_path):
output_dir = tmp_path / "schema_test"
runtime = ConnectorRuntime(output_dir)
schema_v1 = pa.schema(
[
pa.field("id", pa.int64()),
pa.field("name", pa.string()),
pa.field("old_field", pa.string()),
]
)
runtime._save_schema("orders", schema_v1)
schema_v2 = pa.schema([pa.field("id", pa.int64()), pa.field("name", pa.string())])
change = runtime._check_schema_evolution("orders", schema_v2)
assert change is not None
assert "old_field" in change
assert "-" in change
def test_detect_type_change(self, tmp_path):
output_dir = tmp_path / "schema_test"
runtime = ConnectorRuntime(output_dir)
schema_v1 = pa.schema([pa.field("id", pa.int32()), pa.field("value", pa.string())])
runtime._save_schema("data", schema_v1)
schema_v2 = pa.schema([pa.field("id", pa.int64()), pa.field("value", pa.string())])
change = runtime._check_schema_evolution("data", schema_v2)
assert change is not None
assert "int32" in change
assert "int64" in change
def test_no_change_detected(self, tmp_path):
output_dir = tmp_path / "schema_test"
runtime = ConnectorRuntime(output_dir)
schema = pa.schema([pa.field("id", pa.int64())])
runtime._save_schema("stable", schema)
change = runtime._check_schema_evolution("stable", schema)
assert change is None
def test_first_run_no_previous_schema(self, tmp_path):
output_dir = tmp_path / "schema_test"
runtime = ConnectorRuntime(output_dir)
schema = pa.schema([pa.field("id", pa.int64())])
change = runtime._check_schema_evolution("new_table", schema)
assert change is None # First run, no previous schema to compare
class TestStreamingCapability:
"""Test 6: Async streaming connector works."""
def test_async_stream(self):
async def _run():
connector = StreamingConnector()
batches = []
async for batch in connector.stream("events"):
batches.append(batch)
return batches
batches = asyncio.run(_run())
assert len(batches) == 3
assert batches[0].num_rows == 1
assert batches[0].column("type")[0].as_py() == "created"
def test_stream_to_duckdb(self):
"""Stream batches can be consumed by DuckDB."""
async def _run():
connector = StreamingConnector()
all_batches = []
async for batch in connector.stream("events"):
all_batches.append(batch)
return all_batches
all_batches = asyncio.run(_run())
arrow_table = pa.Table.from_batches(all_batches)
con = duckdb.connect()
result = con.execute("SELECT count(*) FROM arrow_table").fetchone()
assert result[0] == 3
class TestRemoteOnlyConnector:
"""Test 7: Remote-only connector produces correct metadata."""
def test_remote_attach_info(self, tmp_path):
output_dir = tmp_path / "remote_test"
connector = RemoteOnlyConnector()
runtime = ConnectorRuntime(output_dir)
stats = runtime.run(connector)
# No tables extracted (remote only), but no errors
assert stats.tables_extracted == 0
assert stats.errors == []
# Remote attach info saved
ra_path = output_dir / ".remote_attach.yaml"
assert ra_path.exists()
ra = yaml.safe_load(ra_path.read_text())
assert ra["extension"] == "bigquery"
assert ra["token_env"] == "GOOGLE_APPLICATION_CREDENTIALS"
class TestManifestValidation:
"""Test 8: YAML manifest parsing and validation."""
SAMPLE_MANIFEST = """
name: sample_api
version: "1.0.0"
description: "Sample API connector"
entrypoint: connectors.sample.SampleAPIConnector
capabilities: [discover, read]
auth:
type: token
env_vars:
- name: SAMPLE_API_TOKEN
required: true
description: "API authentication token"
config:
base_url:
type: string
required: true
batch_size:
type: integer
required: false
default: 1000
health_check:
endpoint: "${base_url}/health"
method: GET
expect_status: 200
"""
def test_manifest_parses(self):
manifest = yaml.safe_load(self.SAMPLE_MANIFEST)
assert manifest["name"] == "sample_api"
assert manifest["version"] == "1.0.0"
assert "discover" in manifest["capabilities"]
assert "read" in manifest["capabilities"]
def test_manifest_capabilities_to_flags(self):
manifest = yaml.safe_load(self.SAMPLE_MANIFEST)
cap_map = {c.name.lower(): c for c in Cap}
flags = Cap(0)
for c in manifest["capabilities"]:
flags |= cap_map[c]
assert Cap.DISCOVER in flags
assert Cap.READ in flags
assert Cap.STREAM not in flags
def test_manifest_auth_config(self):
manifest = yaml.safe_load(self.SAMPLE_MANIFEST)
assert manifest["auth"]["type"] == "token"
assert manifest["auth"]["env_vars"][0]["name"] == "SAMPLE_API_TOKEN"
assert manifest["auth"]["env_vars"][0]["required"] is True
def test_manifest_config_schema(self):
manifest = yaml.safe_load(self.SAMPLE_MANIFEST)
assert manifest["config"]["base_url"]["required"] is True
assert manifest["config"]["batch_size"]["default"] == 1000
def test_manifest_health_check(self):
manifest = yaml.safe_load(self.SAMPLE_MANIFEST)
hc = manifest["health_check"]
assert "${base_url}" in hc["endpoint"]
assert hc["expect_status"] == 200
class TestDiscoveryToReadPipeline:
"""Test 9: Full discovery → read → query pipeline."""
def test_discover_then_read_all(self, tmp_path):
"""discover() → pick tables → read() → query in DuckDB."""
connector = SampleAPIConnector()
# Step 1: Discovery
tables = connector.discover()
assert len(tables) == 2
assert all(isinstance(t, TableInfo) for t in tables)
assert all(t.schema is not None for t in tables)
# Step 2: Read via runtime (auto-discovers all tables)
runtime = ConnectorRuntime(tmp_path / "full_pipeline")
stats = runtime.run(connector) # No tables= arg → discovers automatically
assert stats.tables_extracted == 2
# Step 3: Query
con = duckdb.connect(str(tmp_path / "full_pipeline" / "extract.duckdb"), read_only=True)
result = con.execute("""
SELECT table_name, rows, description
FROM _meta ORDER BY table_name
""").fetchall()
assert result[0][0] == "orders"
assert result[0][1] == 3
con.close()
class TestLargeDataBatching:
"""Test 10: Connector can handle large data via batched iteration."""
def test_batched_read_memory_constant(self, tmp_path):
"""Large dataset extracted in batches — memory doesn't explode."""
class LargeConnector:
capabilities = Cap.DISCOVER | Cap.READ
NUM_BATCHES = 100
BATCH_SIZE = 1000
def discover(self) -> list[TableInfo]:
return [
TableInfo(
"big_table",
pa.schema([pa.field("id", pa.int64()), pa.field("value", pa.float64())]),
Cap.READ,
)
]
def read(self, table: str, options: ReadOptions) -> Iterator[pa.RecordBatch]:
schema = pa.schema([pa.field("id", pa.int64()), pa.field("value", pa.float64())])
for batch_num in range(self.NUM_BATCHES):
start = batch_num * self.BATCH_SIZE
yield pa.RecordBatch.from_arrays(
[
pa.array(range(start, start + self.BATCH_SIZE), type=pa.int64()),
pa.array(
[float(i) * 0.1 for i in range(start, start + self.BATCH_SIZE)],
type=pa.float64(),
),
],
schema=schema,
)
runtime = ConnectorRuntime(tmp_path / "large_test")
stats = runtime.run(LargeConnector())
assert stats.total_rows == 100_000
assert stats.tables_extracted == 1
# Verify DuckDB can read it
con = duckdb.connect(str(tmp_path / "large_test" / "extract.duckdb"), read_only=True)
count = con.execute("SELECT count(*) FROM big_table").fetchone()[0]
assert count == 100_000
con.close()

710
tests/test_pat.py Normal file
View file

@ -0,0 +1,710 @@
"""Tests for #12 — personal access tokens (PAT)."""
import os
import tempfile
import pytest
@pytest.fixture
def fresh_db(monkeypatch):
with tempfile.TemporaryDirectory() as tmp:
monkeypatch.setenv("DATA_DIR", tmp)
monkeypatch.setenv("TESTING", "1")
monkeypatch.setenv("JWT_SECRET_KEY", "test-jwt-secret-key-minimum-32-chars!!")
yield tmp
def test_schema_v6_creates_pat_table(fresh_db):
from src.db import get_system_db, get_schema_version, close_system_db
conn = get_system_db()
try:
cols = conn.execute("PRAGMA table_info(personal_access_tokens)").fetchall()
col_names = [c[1] for c in cols]
for expected in ("id", "user_id", "name", "token_hash", "prefix",
"scopes", "created_at", "expires_at", "last_used_at", "revoked_at"):
assert expected in col_names
assert get_schema_version(conn) >= 6
finally:
conn.close()
close_system_db()
def test_schema_v7_adds_last_used_ip_column(fresh_db):
"""Schema v7: personal_access_tokens has last_used_ip column."""
from src.db import get_system_db, get_schema_version, close_system_db
conn = get_system_db()
try:
cols = conn.execute("PRAGMA table_info(personal_access_tokens)").fetchall()
col_names = [c[1] for c in cols]
assert "last_used_ip" in col_names
assert get_schema_version(conn) >= 7
finally:
conn.close()
close_system_db()
def test_access_token_repo_create_and_lookup(fresh_db):
import hashlib, uuid
from datetime import datetime, timezone, timedelta
from src.db import get_system_db, close_system_db
from src.repositories.access_tokens import AccessTokenRepository
conn = get_system_db()
try:
repo = AccessTokenRepository(conn)
token_id = str(uuid.uuid4())
raw = "abcdefgh" + "x" * 32
repo.create(
id=token_id,
user_id="u1",
name="laptop",
token_hash=hashlib.sha256(raw.encode()).hexdigest(),
prefix=raw[:8],
expires_at=datetime.now(timezone.utc) + timedelta(days=90),
)
row = repo.get_by_id(token_id)
assert row is not None
assert row["name"] == "laptop"
assert row["prefix"] == "abcdefgh"
assert row["revoked_at"] is None
rows = repo.list_for_user("u1")
assert len(rows) == 1
repo.revoke(token_id)
assert repo.get_by_id(token_id)["revoked_at"] is not None
finally:
conn.close()
close_system_db()
def test_access_token_repo_mark_used(fresh_db):
import hashlib, uuid
from datetime import datetime, timezone
from src.db import get_system_db, close_system_db
from src.repositories.access_tokens import AccessTokenRepository
conn = get_system_db()
try:
repo = AccessTokenRepository(conn)
tid = str(uuid.uuid4())
repo.create(id=tid, user_id="u1", name="x",
token_hash=hashlib.sha256(b"r").hexdigest(), prefix="rrrrrrrr")
assert repo.get_by_id(tid)["last_used_at"] is None
repo.mark_used(tid)
assert repo.get_by_id(tid)["last_used_at"] is not None
finally:
conn.close()
close_system_db()
def test_pat_token_carries_typ_claim(fresh_db):
from app.auth.jwt import create_access_token, verify_token
token = create_access_token(
user_id="u1", email="u@test", role="analyst",
token_id="deadbeef-1234", typ="pat",
)
payload = verify_token(token)
assert payload["typ"] == "pat"
assert payload["jti"] == "deadbeef-1234"
def test_session_token_defaults_typ(fresh_db):
from app.auth.jwt import create_access_token, verify_token
token = create_access_token(user_id="u1", email="u@test", role="analyst")
payload = verify_token(token)
# Default typ is "session".
assert payload.get("typ") == "session"
def test_revoked_pat_is_rejected(fresh_db, monkeypatch):
from fastapi.testclient import TestClient
import hashlib, uuid
from datetime import datetime, timezone, timedelta
from src.db import get_system_db, close_system_db
from src.repositories.users import UserRepository
from src.repositories.access_tokens import AccessTokenRepository
from app.auth.jwt import create_access_token
from app.main import app
conn = get_system_db()
try:
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email="u@t", name="U", role="admin")
token_id = str(uuid.uuid4())
raw = "secretXX" + "a" * 32
AccessTokenRepository(conn).create(
id=token_id, user_id=uid, name="ci",
token_hash=hashlib.sha256(raw.encode()).hexdigest(),
prefix=raw[:8],
expires_at=datetime.now(timezone.utc) + timedelta(days=30),
)
jwt_token = create_access_token(
user_id=uid, email="u@t", role="admin", token_id=token_id, typ="pat",
)
# Revoke
AccessTokenRepository(conn).revoke(token_id)
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.get(
"/api/users",
headers={"Authorization": f"Bearer {jwt_token}", "Accept": "application/json"},
)
assert resp.status_code == 401
def test_expired_pat_is_rejected_from_db(fresh_db):
"""A PAT with a past expires_at in DB is rejected even if JWT exp is in future."""
from fastapi.testclient import TestClient
import hashlib, uuid
from datetime import datetime, timezone, timedelta
from src.db import get_system_db, close_system_db
from src.repositories.users import UserRepository
from src.repositories.access_tokens import AccessTokenRepository
from app.auth.jwt import create_access_token
from app.main import app
conn = get_system_db()
try:
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email="u@t", name="U", role="admin")
tid = str(uuid.uuid4())
# Past-dated expiry in DB
AccessTokenRepository(conn).create(
id=tid, user_id=uid, name="stale",
token_hash=hashlib.sha256(b"whatever").hexdigest(), prefix=tid.replace("-","")[:8],
expires_at=datetime.now(timezone.utc) - timedelta(days=1),
)
# JWT with much longer TTL so signature-level `exp` would pass
pat = create_access_token(
user_id=uid, email="u@t", role="admin",
token_id=tid, typ="pat",
expires_delta=timedelta(days=365),
)
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.get(
"/api/users",
headers={"Authorization": f"Bearer {pat}", "Accept": "application/json"},
)
assert resp.status_code == 401
def test_create_pat_returns_raw_once(fresh_db):
from fastapi.testclient import TestClient
import uuid
from src.db import get_system_db, close_system_db
from src.repositories.users import UserRepository
from app.auth.jwt import create_access_token
from app.main import app
conn = get_system_db()
try:
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email="u@t", name="U", role="admin")
sess_token = create_access_token(user_id=uid, email="u@t", role="admin") # typ=session
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.post(
"/auth/tokens",
headers={"Authorization": f"Bearer {sess_token}"},
json={"name": "laptop", "expires_in_days": 30},
)
assert resp.status_code == 201
data = resp.json()
assert data["name"] == "laptop"
assert "token" in data and data["token"] # raw token returned exactly once
# Listing returns prefix, never raw.
# Prefix is derived from the token id (jti), not the JWT string, to avoid
# all tokens having the useless "eyJhbGci" JWT-header prefix.
list_resp = client.get(
"/auth/tokens", headers={"Authorization": f"Bearer {sess_token}"},
)
assert list_resp.status_code == 200
rows = list_resp.json()
assert len(rows) == 1
assert "token" not in rows[0]
assert rows[0]["prefix"] == data["prefix"]
assert len(rows[0]["prefix"]) == 8
assert not data["prefix"].startswith("eyJ") # regression: not the JWT header
def test_pat_cannot_create_pat(fresh_db):
from fastapi.testclient import TestClient
import hashlib, uuid
from datetime import datetime, timezone, timedelta
from src.db import get_system_db, close_system_db
from src.repositories.users import UserRepository
from src.repositories.access_tokens import AccessTokenRepository
from app.auth.jwt import create_access_token
from app.main import app
conn = get_system_db()
try:
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email="u@t", name="U", role="admin")
tid = str(uuid.uuid4())
# Create the JWT first so we can store its sha256 as token_hash (otherwise
# the defense-in-depth check in get_current_user would reject it with 401
# before require_session_token ever runs).
pat = create_access_token(user_id=uid, email="u@t", role="admin", token_id=tid, typ="pat")
AccessTokenRepository(conn).create(
id=tid, user_id=uid, name="x",
token_hash=hashlib.sha256(pat.encode()).hexdigest(),
prefix=tid.replace("-", "")[:8],
expires_at=datetime.now(timezone.utc) + timedelta(days=90),
)
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.post(
"/auth/tokens",
headers={"Authorization": f"Bearer {pat}"},
json={"name": "bad", "expires_in_days": 30},
)
assert resp.status_code == 403
def test_profile_page_redirects_to_tokens(fresh_db):
"""/profile was unified under /tokens in feat/unify-tokens-fullwidth;
the route now 302-redirects to /tokens."""
from fastapi.testclient import TestClient
import uuid
from src.db import get_system_db, close_system_db
from src.repositories.users import UserRepository
from app.auth.jwt import create_access_token
from app.main import app
conn = get_system_db()
try:
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email="u@t", name="U", role="analyst")
token = create_access_token(user_id=uid, email="u@t", role="analyst")
finally:
conn.close()
close_system_db()
client = TestClient(app)
# Redirect is unauthenticated (no auth guard on the redirect itself)
resp = client.get("/profile", follow_redirects=False)
assert resp.status_code == 302
assert resp.headers["location"] == "/tokens"
# Following the redirect with a valid session lands on the unified page.
resp = client.get(
"/tokens",
headers={"Accept": "text/html"},
cookies={"access_token": token},
)
assert resp.status_code == 200
assert "My tokens" in resp.text # non-admin title
assert 'id="new-token-btn"' in resp.text # non-admin CTA
def test_pat_first_use_from_new_ip_audits(fresh_db):
"""Using a PAT from a different IP than last time emits an audit entry."""
from fastapi.testclient import TestClient
import hashlib, uuid
from datetime import datetime, timezone, timedelta
from src.db import get_system_db, close_system_db
from src.repositories.users import UserRepository
from src.repositories.access_tokens import AccessTokenRepository
from app.auth.jwt import create_access_token
from app.main import app
conn = get_system_db()
try:
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email="u@t", name="U", role="admin")
tid = str(uuid.uuid4())
pat = create_access_token(
user_id=uid, email="u@t", role="admin", token_id=tid, typ="pat",
expires_delta=timedelta(days=90),
)
repo = AccessTokenRepository(conn)
repo.create(
id=tid, user_id=uid, name="ci",
token_hash=hashlib.sha256(pat.encode()).hexdigest(),
prefix=tid.replace("-", "")[:8],
expires_at=datetime.now(timezone.utc) + timedelta(days=90),
)
# Simulate a prior use from 1.1.1.1 so the upcoming call is a "new IP".
repo.mark_used(tid, ip="1.1.1.1")
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.get(
"/api/users",
headers={
"Authorization": f"Bearer {pat}",
"Accept": "application/json",
"X-Forwarded-For": "2.2.2.2",
},
)
assert resp.status_code == 200, resp.text
conn = get_system_db()
try:
rows = conn.execute(
"SELECT params FROM audit_log WHERE action = 'token.first_use_new_ip' AND user_id = ?",
[uid],
).fetchall()
assert len(rows) == 1, f"expected 1 audit row, got {len(rows)}"
params = rows[0][0]
# params is stored as JSON text; check the IP appears
assert "2.2.2.2" in str(params)
finally:
conn.close()
close_system_db()
def test_pat_same_ip_does_not_audit(fresh_db):
"""Using a PAT from the same IP as last time does NOT emit an audit entry."""
from fastapi.testclient import TestClient
import hashlib, uuid
from datetime import datetime, timezone, timedelta
from src.db import get_system_db, close_system_db
from src.repositories.users import UserRepository
from src.repositories.access_tokens import AccessTokenRepository
from app.auth.jwt import create_access_token
from app.main import app
conn = get_system_db()
try:
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email="u@t", name="U", role="admin")
tid = str(uuid.uuid4())
pat = create_access_token(
user_id=uid, email="u@t", role="admin", token_id=tid, typ="pat",
expires_delta=timedelta(days=90),
)
repo = AccessTokenRepository(conn)
repo.create(
id=tid, user_id=uid, name="ci",
token_hash=hashlib.sha256(pat.encode()).hexdigest(),
prefix=tid.replace("-", "")[:8],
expires_at=datetime.now(timezone.utc) + timedelta(days=90),
)
repo.mark_used(tid, ip="3.3.3.3")
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.get(
"/api/users",
headers={
"Authorization": f"Bearer {pat}",
"Accept": "application/json",
"X-Forwarded-For": "3.3.3.3",
},
)
assert resp.status_code == 200, resp.text
conn = get_system_db()
try:
count = conn.execute(
"SELECT COUNT(*) FROM audit_log WHERE action = 'token.first_use_new_ip' AND user_id = ?",
[uid],
).fetchone()[0]
assert count == 0
finally:
conn.close()
close_system_db()
def test_pat_can_list_own_tokens(fresh_db):
"""A PAT must be allowed to list its owner's tokens — `da auth token list`
CLI flow. Previously this returned 403 because require_session_token
blocked all PATs uniformly."""
from fastapi.testclient import TestClient
import hashlib, uuid
from datetime import datetime, timezone, timedelta
from src.db import get_system_db, close_system_db
from src.repositories.users import UserRepository
from src.repositories.access_tokens import AccessTokenRepository
from app.auth.jwt import create_access_token
from app.main import app
conn = get_system_db()
try:
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email="u@t", name="U", role="analyst")
tid = str(uuid.uuid4())
pat = create_access_token(
user_id=uid, email="u@t", role="analyst", token_id=tid, typ="pat",
expires_delta=timedelta(days=90),
)
AccessTokenRepository(conn).create(
id=tid, user_id=uid, name="laptop",
token_hash=hashlib.sha256(pat.encode()).hexdigest(),
prefix=tid.replace("-", "")[:8],
expires_at=datetime.now(timezone.utc) + timedelta(days=90),
)
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.get(
"/auth/tokens",
headers={"Authorization": f"Bearer {pat}"},
)
assert resp.status_code == 200, resp.text
rows = resp.json()
assert any(r["id"] == tid for r in rows)
def test_pat_can_revoke_own_token(fresh_db):
"""A PAT must be allowed to revoke its owner's own tokens —
`da auth token revoke` CLI flow."""
from fastapi.testclient import TestClient
import hashlib, uuid
from datetime import datetime, timezone, timedelta
from src.db import get_system_db, close_system_db
from src.repositories.users import UserRepository
from src.repositories.access_tokens import AccessTokenRepository
from app.auth.jwt import create_access_token
from app.main import app
conn = get_system_db()
try:
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email="u@t", name="U", role="analyst")
# Token A — the PAT used to authenticate this call.
tid_a = str(uuid.uuid4())
pat_a = create_access_token(
user_id=uid, email="u@t", role="analyst", token_id=tid_a, typ="pat",
expires_delta=timedelta(days=90),
)
AccessTokenRepository(conn).create(
id=tid_a, user_id=uid, name="primary",
token_hash=hashlib.sha256(pat_a.encode()).hexdigest(),
prefix=tid_a.replace("-", "")[:8],
expires_at=datetime.now(timezone.utc) + timedelta(days=90),
)
# Token B — the one we'll revoke with A.
tid_b = str(uuid.uuid4())
AccessTokenRepository(conn).create(
id=tid_b, user_id=uid, name="old-ci",
token_hash=hashlib.sha256(b"whatever").hexdigest(),
prefix=tid_b.replace("-", "")[:8],
expires_at=datetime.now(timezone.utc) + timedelta(days=90),
)
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.delete(
f"/auth/tokens/{tid_b}",
headers={"Authorization": f"Bearer {pat_a}"},
)
assert resp.status_code == 204, resp.text
# Confirm B is now revoked.
conn = get_system_db()
try:
row = AccessTokenRepository(conn).get_by_id(tid_b)
assert row["revoked_at"] is not None
finally:
conn.close()
close_system_db()
def test_create_token_rejects_expires_in_days_above_cap(fresh_db):
"""expires_in_days > 3650 must return 400 (not 500 via datetime overflow)."""
from fastapi.testclient import TestClient
import uuid
from src.db import get_system_db, close_system_db
from src.repositories.users import UserRepository
from app.auth.jwt import create_access_token
from app.main import app
conn = get_system_db()
try:
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email="u@t", name="U", role="admin")
sess_token = create_access_token(user_id=uid, email="u@t", role="admin")
finally:
conn.close()
close_system_db()
client = TestClient(app)
# Just above the cap — must be 400, not 500.
resp = client.post(
"/auth/tokens",
headers={"Authorization": f"Bearer {sess_token}"},
json={"name": "laptop", "expires_in_days": 3651},
)
assert resp.status_code == 400, resp.text
assert "3650" in resp.text
# Huge value that would previously overflow datetime.max — still 400.
resp = client.post(
"/auth/tokens",
headers={"Authorization": f"Bearer {sess_token}"},
json={"name": "laptop", "expires_in_days": 10_000_000_000},
)
assert resp.status_code == 400, resp.text
def test_pat_first_ever_use_does_not_audit(fresh_db):
"""The first-ever use of a PAT (no prior last_used_at) does NOT emit an audit entry."""
from fastapi.testclient import TestClient
import hashlib, uuid
from datetime import datetime, timezone, timedelta
from src.db import get_system_db, close_system_db
from src.repositories.users import UserRepository
from src.repositories.access_tokens import AccessTokenRepository
from app.auth.jwt import create_access_token
from app.main import app
conn = get_system_db()
try:
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email="u@t", name="U", role="admin")
tid = str(uuid.uuid4())
pat = create_access_token(
user_id=uid, email="u@t", role="admin", token_id=tid, typ="pat",
expires_delta=timedelta(days=90),
)
AccessTokenRepository(conn).create(
id=tid, user_id=uid, name="ci",
token_hash=hashlib.sha256(pat.encode()).hexdigest(),
prefix=tid.replace("-", "")[:8],
expires_at=datetime.now(timezone.utc) + timedelta(days=90),
)
# No mark_used call → first-ever use
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.get(
"/api/users",
headers={
"Authorization": f"Bearer {pat}",
"Accept": "application/json",
"X-Forwarded-For": "4.4.4.4",
},
)
assert resp.status_code == 200, resp.text
conn = get_system_db()
try:
count = conn.execute(
"SELECT COUNT(*) FROM audit_log WHERE action = 'token.first_use_new_ip' AND user_id = ?",
[uid],
).fetchone()[0]
assert count == 0
finally:
conn.close()
close_system_db()
def test_pat_null_expiry_jwt_has_no_exp_claim(fresh_db):
"""PAT with `expires_in_days=null` (user-requested "never") must not
carry an `exp` claim at all — the DB `expires_at=NULL` is the source
of truth. The previous ~100y `exp` claim was a misleading silent expiry."""
from fastapi.testclient import TestClient
import uuid
import jwt as pyjwt
from src.db import get_system_db, close_system_db
from src.repositories.users import UserRepository
from app.auth.jwt import create_access_token
from app.main import app
conn = get_system_db()
try:
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email="u@t", name="U", role="admin")
sess_token = create_access_token(user_id=uid, email="u@t", role="admin")
finally:
conn.close()
close_system_db()
client = TestClient(app)
resp = client.post(
"/auth/tokens",
headers={"Authorization": f"Bearer {sess_token}"},
json={"name": "forever", "expires_in_days": None},
)
assert resp.status_code == 201, resp.text
raw_pat = resp.json()["token"]
# Decode without signature verification — we're inspecting claims only.
claims = pyjwt.decode(raw_pat, options={"verify_signature": False})
assert "exp" not in claims, f"expected no exp claim, got: {claims.get('exp')}"
# But the other PAT claims are still present.
assert claims.get("typ") == "pat"
assert claims.get("sub") == uid
assert "jti" in claims
# DB row mirrors this: expires_at is NULL.
assert resp.json()["expires_at"] is None
def test_pat_with_null_expiry_is_accepted_by_verify_token(fresh_db):
"""A claim-less JWT (no `exp`) must round-trip through verify_token without
raising ExpiredSignatureError and without falling back to a wall-clock
cap. The DB-level expiry check in dependencies.py remains authoritative."""
from app.auth.jwt import create_access_token, verify_token
raw = create_access_token(
user_id="u-1", email="u@t", role="admin",
token_id="tid-1", typ="pat", omit_exp=True,
)
payload = verify_token(raw)
assert payload is not None
assert "exp" not in payload
assert payload["typ"] == "pat"
assert payload["jti"] == "tid-1"
def test_pat_null_expiry_end_to_end_allows_authenticated_request(fresh_db):
"""Create a PAT with `expires_in_days=null`, then use it to call an
authenticated endpoint. Previously relied on the 36500-day `exp`;
now relies on the DB row. Regression guard for the switch."""
from fastapi.testclient import TestClient
import uuid
from src.db import get_system_db, close_system_db
from src.repositories.users import UserRepository
from app.auth.jwt import create_access_token
from app.main import app
conn = get_system_db()
try:
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email="u@t", name="U", role="admin")
sess_token = create_access_token(user_id=uid, email="u@t", role="admin")
finally:
conn.close()
close_system_db()
client = TestClient(app)
created = client.post(
"/auth/tokens",
headers={"Authorization": f"Bearer {sess_token}"},
json={"name": "forever", "expires_in_days": None},
)
assert created.status_code == 201, created.text
pat = created.json()["token"]
# Use the PAT to list tokens (any authenticated endpoint).
listed = client.get("/auth/tokens", headers={"Authorization": f"Bearer {pat}"})
assert listed.status_code == 200, listed.text
assert any(row["name"] == "forever" for row in listed.json())

298
tests/test_user_management.py Normal file
View file

@ -0,0 +1,298 @@
"""Tests for #11 — user management (active flag, safeguards, endpoints)."""
import os
import tempfile
import pytest
import duckdb
from src.db import _ensure_schema, get_schema_version
@pytest.fixture
def fresh_db(monkeypatch):
with tempfile.TemporaryDirectory() as tmp:
monkeypatch.setenv("DATA_DIR", tmp)
# Reset cached system DB so we open a brand-new instance in tmp
from src.db import close_system_db
close_system_db()
yield tmp
close_system_db()
def test_schema_v5_adds_active_column(fresh_db):
from src.db import get_system_db, close_system_db
conn = get_system_db()
try:
cols = conn.execute("PRAGMA table_info(users)").fetchall()
col_names = [c[1] for c in cols]
assert "active" in col_names
assert "deactivated_at" in col_names
assert "deactivated_by" in col_names
assert get_schema_version(conn) >= 5
finally:
conn.close()
close_system_db()
def test_schema_v5_backfill_keeps_existing_users_active(fresh_db):
"""Simulate upgrading from v4: insert a user pre-migration, verify active=TRUE afterwards."""
import uuid
import duckdb as _duckdb
from pathlib import Path
# 1. Create a v4-era DB by hand.
db_dir = Path(fresh_db) / "state"
db_dir.mkdir(parents=True, exist_ok=True)
db_path = db_dir / "system.duckdb"
conn = _duckdb.connect(str(db_path))
try:
conn.execute("CREATE TABLE schema_version (version INTEGER NOT NULL, applied_at TIMESTAMP DEFAULT current_timestamp)")
conn.execute("INSERT INTO schema_version (version) VALUES (4)")
conn.execute("""CREATE TABLE users (
id VARCHAR PRIMARY KEY, email VARCHAR UNIQUE NOT NULL,
name VARCHAR, role VARCHAR DEFAULT 'analyst',
password_hash VARCHAR, setup_token VARCHAR,
setup_token_created TIMESTAMP, reset_token VARCHAR,
reset_token_created TIMESTAMP,
created_at TIMESTAMP DEFAULT current_timestamp, updated_at TIMESTAMP)""")
uid = str(uuid.uuid4())
conn.execute("INSERT INTO users (id, email, name, role) VALUES (?, 'pre@v4', 'Pre', 'admin')", [uid])
finally:
conn.close()
# 2. Now let the app open it — schema should migrate to v5 and backfill active=TRUE.
from src.db import get_system_db, close_system_db, get_schema_version
close_system_db()
conn = get_system_db()
try:
assert get_schema_version(conn) >= 5
row = conn.execute("SELECT email, active FROM users WHERE email = 'pre@v4'").fetchone()
assert row is not None
assert row[1] is True
finally:
conn.close()
close_system_db()
def test_repository_update_accepts_active(fresh_db):
import uuid
from src.db import get_system_db, close_system_db
from src.repositories.users import UserRepository
conn = get_system_db()
try:
repo = UserRepository(conn)
uid = str(uuid.uuid4())
repo.create(id=uid, email="a@b.c", name="A", role="analyst")
repo.update(id=uid, active=False, deactivated_by="admin-uuid")
row = repo.get_by_id(uid)
assert row["active"] is False
assert row["deactivated_by"] == "admin-uuid"
finally:
conn.close()
close_system_db()
def test_repository_count_admins(fresh_db):
import uuid
from src.db import get_system_db, close_system_db
from src.repositories.users import UserRepository
conn = get_system_db()
try:
repo = UserRepository(conn)
assert repo.count_admins() == 0
repo.create(id=str(uuid.uuid4()), email="a@b.c", name="A", role="admin")
repo.create(id=str(uuid.uuid4()), email="b@b.c", name="B", role="analyst")
assert repo.count_admins() == 1
finally:
conn.close()
close_system_db()
from fastapi.testclient import TestClient
@pytest.fixture
def app_client(fresh_db, monkeypatch):
monkeypatch.setenv("TESTING", "1")
monkeypatch.setenv("JWT_SECRET_KEY", "test-jwt-secret-key-minimum-32-chars!!")
from app.main import app
return TestClient(app)
def _seed_admin(fresh_db):
"""Create an admin user and return (id, bearer_token)."""
import uuid
from src.db import get_system_db
from src.repositories.users import UserRepository
from app.auth.jwt import create_access_token
conn = get_system_db()
try:
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email="admin@test", name="Admin", role="admin")
token = create_access_token(user_id=uid, email="admin@test", role="admin")
return uid, token
finally:
conn.close()
def test_patch_user_updates_role(app_client, fresh_db):
import uuid
from src.db import get_system_db
from src.repositories.users import UserRepository
admin_id, token = _seed_admin(fresh_db)
target_id = str(uuid.uuid4())
conn = get_system_db()
try:
UserRepository(conn).create(id=target_id, email="x@test", name="X", role="viewer")
finally:
conn.close()
resp = app_client.patch(
f"/api/users/{target_id}",
headers={"Authorization": f"Bearer {token}"},
json={"role": "analyst", "name": "X2"},
)
assert resp.status_code == 200
data = resp.json()
assert data["role"] == "analyst"
assert data["name"] == "X2"
def test_cannot_self_deactivate(app_client, fresh_db):
admin_id, token = _seed_admin(fresh_db)
resp = app_client.patch(
f"/api/users/{admin_id}",
headers={"Authorization": f"Bearer {token}"},
json={"active": False},
)
assert resp.status_code == 409
assert "yourself" in resp.json()["detail"].lower()
def test_cannot_delete_last_admin(app_client, fresh_db):
"""Deleting the sole active admin must 409.
Note: the endpoint checks self-delete first, which also triggers 409 here,
so we accept either "yourself" or "last" wording — the point is the
safeguard blocks deletion of the only admin."""
admin_id, token = _seed_admin(fresh_db)
# Create a non-admin so we have ≥2 users, but admin is still the only admin.
resp = app_client.post(
"/api/users",
headers={"Authorization": f"Bearer {token}"},
json={"email": "x@test", "name": "X", "role": "viewer"},
)
x_id = resp.json()["id"]
# Try deleting the admin.
resp = app_client.delete(
f"/api/users/{admin_id}",
headers={"Authorization": f"Bearer {token}"},
)
assert resp.status_code == 409
detail = resp.json()["detail"].lower()
assert "last" in detail or "yourself" in detail
def test_deactivated_user_cannot_authenticate(app_client, fresh_db):
"""A deactivated user's old JWT must be rejected."""
import uuid
from src.db import get_system_db
from src.repositories.users import UserRepository
from app.auth.jwt import create_access_token
conn = get_system_db()
try:
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email="u@test", name="U", role="analyst")
token = create_access_token(user_id=uid, email="u@test", role="analyst")
UserRepository(conn).update(id=uid, active=False)
finally:
conn.close()
resp = app_client.get(
"/api/users", # any authenticated endpoint
headers={"Authorization": f"Bearer {token}", "Accept": "application/json"},
)
# Deactivated — must not succeed.
assert resp.status_code in (401, 403)
def test_admin_users_page_renders_for_admin(app_client, fresh_db):
admin_id, token = _seed_admin(fresh_db)
resp = app_client.get(
"/admin/users",
headers={"Accept": "text/html"},
cookies={"access_token": token},
)
assert resp.status_code == 200
assert 'class="users-title">Users' in resp.text
def test_admin_users_page_denies_non_admin(app_client, fresh_db):
import uuid
from src.db import get_system_db
from src.repositories.users import UserRepository
from app.auth.jwt import create_access_token
conn = get_system_db()
try:
uid = str(uuid.uuid4())
UserRepository(conn).create(id=uid, email="a@test", name="A", role="analyst")
token = create_access_token(user_id=uid, email="a@test", role="analyst")
finally:
conn.close()
resp = app_client.get(
"/admin/users",
headers={"Accept": "text/html"},
cookies={"access_token": token},
follow_redirects=False,
)
# HTML request to admin-only page → 302 (to /login) for non-admin per Phase 0, or 403.
# Phase 0 is out of scope here so we accept 403 (current behaviour) or 302.
assert resp.status_code in (302, 403)
def test_deactivated_admin_rejected_by_active_check(app_client, fresh_db):
"""Deactivating an admin must cause their token to be rejected as 401 (not succeed)."""
import uuid
from src.db import get_system_db
from src.repositories.users import UserRepository
from app.auth.jwt import create_access_token
# Seed two admins so we can deactivate one without tripping the last-admin rule.
admin_id, admin_token = _seed_admin(fresh_db)
conn = get_system_db()
try:
other_uid = str(uuid.uuid4())
UserRepository(conn).create(id=other_uid, email="other@test", name="Other", role="admin")
other_token = create_access_token(user_id=other_uid, email="other@test", role="admin")
# Directly deactivate the "other" admin via repository (bypass safeguard
# because we already have 2 admins; this is just a state setup).
UserRepository(conn).update(id=other_uid, active=False)
finally:
conn.close()
resp = app_client.get(
"/api/users",
headers={"Authorization": f"Bearer {other_token}", "Accept": "application/json"},
)
assert resp.status_code == 401
assert "deactivated" in resp.json().get("detail", "").lower()
def test_cannot_deactivate_last_admin(app_client, fresh_db):
admin_id, token = _seed_admin(fresh_db)
# Create a second user and try to demote the current admin via PATCH.
resp = app_client.post(
"/api/users",
headers={"Authorization": f"Bearer {token}"},
json={"email": "y@test", "name": "Y", "role": "viewer"},
)
y_id = resp.json()["id"]
# Try to demote self (admin → viewer) while only admin — should fail.
resp = app_client.patch(
f"/api/users/{admin_id}",
headers={"Authorization": f"Bearer {token}"},
json={"role": "viewer"},
)
assert resp.status_code == 409
assert "admin" in resp.json()["detail"].lower()

254
tests/test_web_ui.py
View file

@ -92,6 +92,101 @@ class TestWebUISmoke:
pytest.skip("Route /admin/permissions does not exist")
assert resp.status_code == 200
def test_admin_users_renders_modern_ui(self, web_client, admin_cookie):
resp = web_client.get("/admin/users", cookies=admin_cookie)
assert resp.status_code == 200
body = resp.text
# New shared header chrome
assert "app-header" in body
# Nav after split: "Tokens" (own) for every signed-in user +
# admin-only "All tokens" link pointing at /admin/tokens.
assert 'href="/tokens"' in body
assert 'href="/admin/tokens"' in body
assert 'href="/profile"' not in body
assert 'href="/admin/users"' in body
# New modern UI markers
assert 'class="users-page"' in body
assert 'role-pill' in body
assert 'class="toggle"' in body
assert 'id="confirm-modal"' in body
def test_nav_shows_tokens_link_for_non_admin(self, web_client, analyst_cookie):
"""Non-admins see the 'My tokens' user-menu link — no 'All tokens' link, no /profile."""
resp = web_client.get("/dashboard", cookies=analyst_cookie)
assert resp.status_code in (200, 302)
if resp.status_code == 302:
# Dashboard may redirect in some flows; follow it for nav check.
resp = web_client.get(resp.headers["location"], cookies=analyst_cookie)
body = resp.text
assert 'href="/tokens"' in body
assert 'href="/profile"' not in body
assert ">My tokens<" in body
assert ">Profile<" not in body
# Non-admins must NOT see the admin "All tokens" link.
assert 'href="/admin/tokens"' not in body
assert ">All tokens<" not in body
def test_nav_shows_all_tokens_link_for_admin(self, web_client, admin_cookie):
"""Admins see the 'My tokens' user-menu link and the 'All tokens' nav link."""
resp = web_client.get("/dashboard", cookies=admin_cookie)
assert resp.status_code in (200, 302)
if resp.status_code == 302:
resp = web_client.get(resp.headers["location"], cookies=admin_cookie)
body = resp.text
assert 'href="/tokens"' in body
assert 'href="/admin/tokens"' in body
assert ">My tokens<" in body
assert ">All tokens<" in body
def test_profile_redirects_to_tokens(self, web_client, admin_cookie):
"""Back-compat: /profile 302-redirects to /tokens."""
resp = web_client.get("/profile", cookies=admin_cookie, follow_redirects=False)
assert resp.status_code == 302
assert resp.headers["location"] == "/tokens"
class TestClaudeSetupPreview:
"""/install and /dashboard render a visible, read-only preview of the
'Setup a new Claude Code' clipboard payload. The real token is never
rendered into the HTML — only a styled placeholder is.
"""
def test_install_preview_visible_for_signed_in_user(self, web_client, admin_cookie):
resp = web_client.get("/install", cookies=admin_cookie)
assert resp.status_code == 200
body = resp.text
# Preview card + placeholder token render
assert "setup-preview-pre" in body
assert "What Claude Code will receive" in body
assert "&lt;will be generated on click&gt;" in body
assert 'class="placeholder-token"' in body
# Setup payload text substituted with real server URL
assert "/cli/agnes.whl" in body
# New numbered headers + da diagnose step
assert "1) Install the CLI" in body
assert "4) Run diagnostics" in body
assert "da diagnose" in body
assert "da auth whoami" in body
def test_dashboard_preview_visible(self, web_client, admin_cookie):
resp = web_client.get("/dashboard", cookies=admin_cookie)
assert resp.status_code == 200
body = resp.text
assert "env-setup-cta" in body
assert "setup-preview-pre" in body
assert "What Claude Code will receive" in body
assert "&lt;will be generated on click&gt;" in body
def test_install_mcp_card_removed(self, web_client):
"""The stale 'Use with Claude Code / MCP' card on /install has been
removed — there is no Agnes MCP server today.
"""
resp = web_client.get("/install")
assert resp.status_code == 200
body = resp.text
assert "Use with Claude Code / MCP" not in body
assert "MCP" not in body
class TestAdminRoleGuards:
def test_analyst_cannot_access_admin_tables(self, web_client, admin_cookie, analyst_cookie):
@ -113,3 +208,162 @@ class TestAdminRoleGuards:
def test_analyst_cannot_access_corporate_memory_admin(self, web_client, admin_cookie, analyst_cookie):
resp = web_client.get("/corporate-memory/admin", cookies=analyst_cookie)
assert resp.status_code == 403
class TestUnauthenticatedHtmlRedirects:
def test_dashboard_unauthenticated_redirects_to_login(self, web_client):
resp = web_client.get("/dashboard", follow_redirects=False)
assert resp.status_code == 302
assert resp.headers["location"].startswith("/login")
assert "next=%2Fdashboard" in resp.headers["location"]
def test_catalog_unauthenticated_redirects_to_login(self, web_client):
resp = web_client.get("/catalog", follow_redirects=False)
assert resp.status_code == 302
assert resp.headers["location"].startswith("/login")
assert "next=%2Fcatalog" in resp.headers["location"]
def test_api_route_still_returns_json_401(self, web_client):
# /api/sync/manifest requires auth; must keep JSON 401 (no redirect).
resp = web_client.get("/api/sync/manifest", follow_redirects=False)
assert resp.status_code == 401
assert resp.headers["content-type"].startswith("application/json")
def test_password_login_honors_next(self, web_client, tmp_path):
from argon2 import PasswordHasher
from src.db import get_system_db
from src.repositories.users import UserRepository
password = "TestPass1!"
conn = get_system_db()
UserRepository(conn).create(
id="u1", email="u1@test.com", name="U1", role="admin",
password_hash=PasswordHasher().hash(password),
)
conn.close()
resp = web_client.post(
"/auth/password/login/web",
data={"email": "u1@test.com", "password": password, "next": "/catalog"},
follow_redirects=False,
)
assert resp.status_code == 302
assert resp.headers["location"] == "/catalog"
def test_password_login_rejects_open_redirect(self, web_client, tmp_path):
from argon2 import PasswordHasher
from src.db import get_system_db
from src.repositories.users import UserRepository
password = "TestPass1!"
conn = get_system_db()
UserRepository(conn).create(
id="u2", email="u2@test.com", name="U2", role="admin",
password_hash=PasswordHasher().hash(password),
)
conn.close()
resp = web_client.post(
"/auth/password/login/web",
data={"email": "u2@test.com", "password": password, "next": "//evil.example/"},
follow_redirects=False,
)
assert resp.status_code == 302
assert resp.headers["location"] == "/dashboard"
@pytest.mark.parametrize("hostile_next,expected_location", [
("javascript:alert(1)", "/dashboard"),
("http://evil.example/", "/dashboard"),
("//evil.example/", "/dashboard"),
("dashboard", "/dashboard"), # missing leading slash
("/foo?bar=baz", "/foo?bar=baz"), # valid same-origin with query
])
def test_password_login_sanitizes_next(self, web_client, tmp_path, hostile_next, expected_location):
from argon2 import PasswordHasher
from src.db import get_system_db
from src.repositories.users import UserRepository
import uuid
password = "TestPass1!"
uid = f"u-{uuid.uuid4().hex[:8]}"
conn = get_system_db()
UserRepository(conn).create(
id=uid, email=f"{uid}@test.com", name=uid, role="admin",
password_hash=PasswordHasher().hash(password),
)
conn.close()
resp = web_client.post(
"/auth/password/login/web",
data={"email": f"{uid}@test.com", "password": password, "next": hostile_next},
follow_redirects=False,
)
assert resp.status_code == 302
assert resp.headers["location"] == expected_location
def test_non_api_post_still_returns_json_401(self, web_client):
# POST to a JSON auth endpoint that lives outside /api/ — must NOT be redirected.
resp = web_client.post("/auth/token", json={"email": "nope@x.com", "password": "wrong"},
follow_redirects=False)
assert resp.status_code == 401
assert resp.headers["content-type"].startswith("application/json")
def test_auth_json_get_still_returns_json_401(self, web_client):
# GET to a JSON endpoint under /auth/* (e.g. PAT CRUD) — must NOT be redirected,
# so CLI clients calling api_get("/auth/tokens") get JSON they can parse.
resp = web_client.get("/auth/tokens", follow_redirects=False)
assert resp.status_code == 401
assert resp.headers["content-type"].startswith("application/json")
def test_login_page_propagates_next_to_password_button(self, web_client):
resp = web_client.get("/login?next=/catalog")
assert resp.status_code == 200
body = resp.text
# Password button URL should carry next.
assert "/login/password?next=%2Fcatalog" in body, \
f"Expected /login/password?next=%2Fcatalog in login page HTML; got snippet: {body[:500]}"
def test_login_page_propagates_next_to_google_button(self, web_client, monkeypatch):
"""The Google OAuth button URL must also carry the ?next param so the
post-login redirect honors the requested destination."""
# Force Google provider to appear available so the button is rendered.
monkeypatch.setattr(
"app.auth.providers.google.is_available", lambda: True,
)
resp = web_client.get("/login?next=/catalog")
assert resp.status_code == 200
body = resp.text
assert "/auth/google/login?next=%2Fcatalog" in body, \
f"Expected google login URL with ?next in login page; snippet: {body[:800]}"
def test_login_email_page_extracts_and_renders_next(self, web_client):
"""/login/email (magic link) must extract ?next from the URL and
emit it into the hidden form field so it round-trips to the POST."""
resp = web_client.get("/login/email?next=/catalog")
assert resp.status_code == 200
body = resp.text
# The template renders <input type="hidden" name="next" value="/catalog">
assert 'name="next" value="/catalog"' in body, \
f"Expected /catalog in next hidden field; snippet: {body[:800]}"
def test_login_email_page_rejects_open_redirect_in_next(self, web_client):
"""Hostile ?next values (e.g. //evil) must be sanitized away before
the hidden field is rendered."""
resp = web_client.get("/login/email?next=//evil.example/")
assert resp.status_code == 200
body = resp.text
assert "evil.example" not in body
# Empty string is the sanitized default.
assert 'name="next" value=""' in body
def test_google_login_stashes_safe_next_in_session(self, web_client, monkeypatch):
"""google_login() must stash the sanitized next_path in the session.
We can't exercise the full OAuth flow without a Google mock, but we
can verify the helper applies the sanitizer correctly."""
from app.auth._common import safe_next_path
# Valid same-origin paths pass through.
assert safe_next_path("/catalog") == "/catalog"
assert safe_next_path("/foo?bar=baz") == "/foo?bar=baz"
# Open-redirect shapes get defaulted.
assert safe_next_path("//evil.example/") == "/dashboard"
assert safe_next_path("http://evil.example/") == "/dashboard"
assert safe_next_path("javascript:alert(1)") == "/dashboard"
assert safe_next_path("") == "/dashboard"
assert safe_next_path(None) == "/dashboard"
# Empty-default variant (used when computing query string).
assert safe_next_path(None, default="") == ""