* docs(spec): admin observability spec + Activity Center MVP plan
Parent spec (480 lines) + executable plan (2295 lines, 14 TDD tasks).
Covers Activity Center rebuild (/admin/activity), with /admin/sessions
and /admin/feedback deferred to follow-up plans.
Already incorporates reviewer-pass revisions across three angles
(security, production resilience, code architecture):
- _get_db import path corrected to app.auth.dependencies
- Test fixtures aligned with seeded_app / admin_user / get_system_db
- All new audit writes wrapped in try/except + logger.exception
- Filename sanitization on session uploads
- DuckDB DESC index behavior documented; upgrade window flagged
- Migration idempotency + evolved-DB test cases
- reveal_raw + shared-cache multi-worker explicitly deferred
Targets schema v40 (audit_log gains params_before, client_ip,
client_kind, correlation_id + 3 indices).
* feat(db): schema v40 — audit_log gains params_before, client_ip, client_kind, correlation_id + 3 indices
* chore(test): clean up Task 1 — drop unused import, rename stale test
* feat(audit): AuditRepository.log() accepts params_before/client_ip/client_kind/correlation_id
* test(audit): strengthen params_before assertion to round-trip JSON content
* feat(audit): AuditRepository.query() rich filters + keyset cursor pagination
* feat(sync): SyncStateRepository.list_recent() cross-table feed
* feat(audit): POST /api/sync/trigger writes audit_log row
* feat(audit): POST /api/scripts/run-due writes audit_log row
* feat(audit): POST /api/upload/sessions writes audit_log row + sanitizes filename
* feat(audit): GET /api/data/{table_id}/download writes audit_log row
* feat(activity): /api/admin/activity timeline + /health + /sync endpoints
* feat(ui): /admin/activity rebuilt — health pulse, timeline, sync grid; /activity-center → 308 redirect
BREAKING: removed demo executive-pulse / maturity-roadmap content from activity_center.html.
The page now reflects real audit_log + sync_history data.
* feat(ui): admin nav + dashboard widget point at /admin/activity
* feat(activity): recursive-audit suppression for AC read endpoints (60s window per actor+filter)
* feat(activity): emit PostHog events when integration enabled (no-op default)
* fix(audit): move v40 indices out of _SYSTEM_SCHEMA + update test_repositories to unpack query() tuple
_SYSTEM_SCHEMA CREATE INDEX on audit_log(timestamp) failed when migration
tests hand-roll a bare audit_log (id, action) without the timestamp column.
Fix: remove indices from _SYSTEM_SCHEMA; add ADD COLUMN IF NOT EXISTS guards
for timestamp and other pre-v40 columns in _v39_to_v40() so the upgrade path
is safe on any hand-rolled schema; call _v39_to_v40 explicitly in the
fresh-install (current==0) path to restore index creation there.
Also unpack the (rows, next_cursor) tuple from AuditRepository.query() in
the three TestAuditRepository tests that still treated it as a list.
* docs: CHANGELOG entry for Activity Center MVP
* chore: refresh stale module docstring in app/api/activity.py
* feat(cli): agnes admin activity — terminal access to Activity Center (timeline + health + sync)
* fix(db): _v39_to_v40 — add IF NOT EXISTS guard for 'action' column
The v39→v40 ladder step adds defensive ADD COLUMN IF NOT EXISTS for
every audit_log column so a hand-rolled bare audit_log (id only) is
safe through the ladder. 'action' was missing from the guard list,
causing CREATE INDEX idx_audit_action_time to fail on tests that
stub audit_log with only an id column (tests/test_e2e_extract.py::
TestSchemaMigration::test_migration_preserves_and_extends).
Local 6/6 schema tests + the previously-failing CI test pass.
* docs(spec): platform telemetry epic — Boss directive + Activity Monitoring plan rebased onto v40 (stacked on zs/spec-activity-center)
* feat(db): schema v41 — 7 usage_* tables for telemetry (events, summary, rollups, attribution)
* chore(db): tighten v41 — usage_session_summary.session_id NOT NULL + upgrade test asserts all 7 tables
* feat(usage): UsageAttributionRepository — replace/delete/lookup over usage_attribution_* tables
* refactor(marketplace): extract list_inner_skills/agents/commands to src/marketplace_listing.py for reuse
* feat(usage): explode plugin attribution on marketplace sync + store entity write; backfill script
* refactor(marketplace): finish src/marketplace_listing.py extraction — drop duplicate _list_inner_* + _parse_frontmatter from app/api/marketplace.py
* feat(usage): promote attribution helpers to src/usage_attribution_helpers.py; hook update_entity rename + bundle-swap; clarify best-effort semantics
* feat(usage): UsageProcessor real extraction + rollup rebuild + 10 fixture-driven tests
* fix(usage): include tool_id in event hash + executemany + rollup transaction (critical multi-tool-turn drop fix)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(marketplace): popularity stats — invocations_30d + trend + sort=most_used|trending + Most Popular section
* feat(admin): /admin/users/<id> Sessions section — list + single-file + bulk-zip downloads (audit-logged)
* feat(usage): admin export endpoint + CLI — csv/json/parquet streaming, filters, audit-logged
* feat(usage): agnes admin ask — LLM Text-to-SQL over usage_events with SELECT-only validator (audit-logged)
* feat(usage): reprocess + prune endpoints + scheduler daily prune job + CLI
* docs: PLATFORM_SETUP.md operator playbook + HOWTO/ cookbook (5 guides + index)
Adds docs/PLATFORM_SETUP.md as a consolidated operator playbook covering
bootstrap, TLS, marketplaces (curated + flea), scheduler env vars, telemetry
extraction/export/ask/prune, privacy posture, and daily routine.
Adds docs/HOWTO/ with 5 analyst cookbook guides: first query, snapshots for
remote tables, private sessions, feedback + admin ask, and customizing skills.
Existing setup docs (QUICKSTART, DEPLOYMENT, ONBOARDING, HEADLESS_USAGE)
get a one-line cross-reference at the top pointing to PLATFORM_SETUP.md.
* docs(changelog): platform telemetry epic — usage_* foundation + surfaces + admin access + docs
Comprehensive [Unreleased] entry covering: usage_events/session_summary/
tool_daily/plugin_daily tables (v41), attribution lookup tables, backfill
script, marketplace Most Popular + invocation chips + sort, admin Sessions
section, export/ask/reprocess/prune endpoints + CLI mirrors, Activity Center
(v40), PLATFORM_SETUP.md + HOWTO/ docs, and operations notes for v41 upgrade.
* fix(security): block DuckDB read_*/http_*/glob functions in usage_ask validator + symlink escape guard in session zip + clarify mark-private semantics
* fix(admin): parquet export tempfile cleanup on COPY failure + correct processed-first sort on /admin/users/<id>/sessions
* feat(audit): close 8 production audit gaps — query (local/remote/hybrid), catalog/schema/sample, snapshot estimate/create, check-access
* feat(ui): /admin/usage summary dashboard + per-user activity tab on /admin/users/<id>
* fix(audit): cap error messages at 200 chars + audit user_activity reads + recursion guard on usage.summary
* fix(audit): catalog.list audits on error path + clean up deferred json import
* fix(ux): client_kind=cli for PAT auth + timeline empty state + email-instead-of-uuid + nav reorder + help text + loading indicators + ask doc
* feat(observability): unify /admin/activity into single page with saved views
- KPI cards (events, users, error rate, p95) clickable as quick-filters
- Faceted filter dropdowns populated from audit_log in the current window
- Sortable audit table, cursor pagination, per-row JSON side panel
- Saved views (schema v43: user_observability_views) — per-user state
- Top bar: window selector + 30s Live toggle + saved views dropdown
- /admin/scheduler-runs → 308 redirect (source=scheduler filter)
- New endpoints: /api/admin/observability/{facets,kpis,views}
* test: update activity + scheduler-runs tests for unified page
- test_admin_activity_page_renders asserts new structural anchors
- test_admin_scheduler_runs_page_admin_only asserts 308 redirect
* fix(observability): respect [hidden] on modal + side panel
CSS `display: flex` on .obs-modal beat the [hidden] attribute's UA
display:none, so the save-view modal rendered on page load and Cancel
clicks couldn't dismiss it. Gate the modal's flex layout on
:not([hidden]); add the same display:none guard prophylactically to
.obs-panel and .obs-views-panel.
* feat(observability): user enrichment in audit + interactive /admin/usage
Activity:
- /api/admin/activity now joins users for user_email + user_name per row
- User column renders "name (id-prefix)" or "email (id-prefix)" instead
of an opaque truncated UUID; falls back to id when the user record is
missing
Usage:
- /admin/usage rewritten as the same filter/group-by/search pattern as
/admin/activity. Faceted dropdowns (User / Tool / Source / Event type)
populated from usage_events; debounced free-text search across
tool_name / skill_name / subagent_type / command_name
- New endpoints /api/admin/usage/{facets,kpis,query}; the query endpoint
supports group_by in {day, username, tool_name, source, ref_id} with
sort + offset pagination, plus an ungrouped raw-events mode
- 4 KPI cards (events, distinct users, distinct tools, error rate) are
clickable quick-filters; clicking a grouped row applies the bucket as
a filter
- Old static `?window=7d|30d|all` server preload removed; all state is
client-side via since_minutes + group_by + filters in the URL
* fix(observability): clearer labels, all-column sort, drop saved views UI
- Rename page titles: "Activity" → "Server activity", "Usage" → "Tool usage"
with a one-line subtitle on each explaining what the page covers and
linking the other one. The two pages source different data (audit_log
vs usage_events) and the previous labels conflated them.
- Drop the saved-views dropdown + save modal from /admin/activity. The
modal pop-open bug was the trigger; the value wasn't there yet. The
/api/admin/observability/views CRUD + DuckDB table stay in place.
- Rename "Live (30s)" to "Auto-refresh (30s)" with a tooltip clarifying
that it's the re-fetch rate, not the time range. Time range now
labeled "Time range" instead of "Window".
- All audit-table columns are sortable (User, Source, Action, Resource,
Result added); sort is page-local with a Jinja comment explaining the
trade-off. Same for raw usage rows.
- Fix duplicate sort-arrow bug — the literal "▼" in the Time th HTML was
rendering alongside the CSS ::before arrow. Removed the literal; CSS
is the single source of truth.
* feat(observability): global Sessions browser + transcript viewer + CLI
Web:
- /admin/sessions — list every collected session JSONL across all users
with time-range, user, model, errors-only and free-text filters. Default
sort surfaces error-heavy sessions first. KPI cards (sessions, distinct
users, sessions w/ errors, tool error rate) clickable as quick-filters.
- /admin/sessions/<username>/<file> — transcript viewer rendering the
JSONL chronologically: user prompts, assistant text, tool calls (with
JSON input) and tool results (with flattened output). Errors get a red
border + chip and a "Next error" navigation button at the top.
- Admin dropdown gains a "Sessions" link.
API:
- GET /api/admin/sessions/{list,kpis,facets} — filtered cross-user reads
off usage_session_summary
- GET /api/admin/sessions/{username}/{file}/transcript — parses JSONL via
the existing services.session_pipeline.lib, returns chronological events
- GET /api/admin/sessions/{username}/{file}/download — JSONL stream, same
path-safety guards as the per-user endpoint, audit-logged
CLI:
- `agnes admin sessions list [--user X] [--errors] [--since 7d]` — table
output with `!` prefix on rows that hit a tool error
- `agnes admin sessions show <username> <file>` — transcript dump, with
`--errors` to print only the failed tool_result blocks
- `agnes admin sessions download <username> <file> [-o path]`
- `agnes admin sessions kpis` — top-level numbers
* feat(internal): expose telemetry tables to agnes query with row-level RBAC
Three new registered tables backed by system.duckdb, queryable through
the same /api/query plumbing analysts use for Keboola / BigQuery /
local sources:
agnes_sessions → usage_session_summary (filter: username)
agnes_usage → usage_events (filter: username)
agnes_audit → audit_log (filter: user_id)
RBAC is per-row, not per-table: admins see every user's rows; non-admins
see only their own. The filter is built server-side from the auth user
dict; non-admin filter values are regex-validated before SQL interpolation.
Implementation:
- new connector connectors/internal/ with access (filter+exec) + registry
(idempotent table_registry seed at startup)
- /api/query detects internal table refs and short-circuits to a CTE
wrapper that prepends "WITH agnes_x AS (SELECT * FROM <src> WHERE …),
…" then "SELECT * FROM (<user_sql>) AS _q". DuckDB cursor on the
shared system.duckdb handle — opening parallel handles / ATTACH on the
same file is blocked process-wide.
- mixing internal + BQ / registered local tables in one SELECT is
rejected (v1 limitation)
- src.rbac.can_access_table waves internal tables through for all
authenticated users; row scoping is the actual security control
- /api/v2/schema and /api/v2/sample gained internal branches; sample
intentionally skips its cache because rows are RBAC-scoped per caller
- audit row written as action='query.internal' with is_admin flag
Tests: connectors/internal/access — RBAC, filter clause, schema, CTE
wrapper coexistence with user-supplied aggregations, unsafe-username
rejection. 16/16 passing.
Motivating queries this enables:
SELECT tool_name, COUNT(*) FROM agnes_usage
WHERE is_error GROUP BY 1 ORDER BY 2 DESC
-- analyst self-introspection: which tools fail for me?
SELECT user_id, COUNT(*) FROM agnes_audit
WHERE action = 'session.transcript_view' GROUP BY 1
-- admin: who's been looking at whose session transcripts?
* feat(admin): group dropdown into 5 named sections + internal tables in /catalog
Admin dropdown gains section headers so admins can land on the right
page without re-reading the full menu:
Activity Center Server activity / Tool usage / Sessions
Users & Access Users / Groups / Resource access / Tokens
Data Tables
Agent Experience Curated Marketplaces / Flea Submissions /
Agent Setup Prompt / Agent Workspace Prompt
Server Server config
"Agent Experience" frames the curated content + prompts as one cluster
— it's all admin-controlled material that shapes what an analyst's AI
agent encounters. "Configuration" → "Server" since only one item lives
there now.
Renamed the section's first two items:
"Activity" → "Server activity" (matches page H1)
"Usage" → "Tool usage"
Also fixes /catalog visibility of the internal tables (agnes_sessions /
_usage / _audit) for non-admin users: ``app.auth.access.can_access``
short-circuits to True for resource_type='table' + an internal-table id.
Without this, non-admins saw the tables in /api/v2/catalog (which uses
the same RBAC bypass) but not on the /catalog HTML page (which calls
can_access directly, requiring a resource_grants row internal tables
don't have).
CSS for `.app-nav-menu-section`: small caps, muted, non-clickable; first
section trims top padding so the panel doesn't open with an awkward gap.
* refactor(admin): move corporate memory into Admin > Agent Experience
Memory link was the only admin-only entry in the primary nav (gated by
session.user.is_admin). Moves it into the Admin dropdown under Agent
Experience, alongside Curated Marketplaces / Flea Submissions / Prompts
— all admin-curated content that shapes what an analyst's AI agent
encounters.
Renamed the nav label to "Shared Knowledge" to match what the page
actually is (admin-curated organisational knowledge from session
verification, surfaced to agents). URL stays at /corporate-memory; the
route still gates on require_admin per the existing comment.
Side effect: primary nav (Home / Marketplace / Data Packages) is now
uniform for every authenticated user — no conditional admin-only entry.
* ui: rename admin entries to Curated Knowledge / Init Prompt / Workspace Prompt
- "Shared Knowledge" → "Curated Knowledge" (parallel with "Curated
Marketplaces" in the same Agent Experience section; "curated" tells
the admin what they do there — review + approve)
- "Agent Setup Prompt" → "Init Prompt" (matches the `agnes init` flow
it actually drives)
- "Agent Workspace Prompt" → "Workspace Prompt" (the "Agent" prefix
was redundant — every item in the section is agent-facing)
Renames page titles + H1s on /admin/agent-prompt and
/admin/workspace-prompt to match.
* refactor: rename Usage → Telemetry across user-facing surfaces
External surfaces all switch; internal Python module / file names and the
physical DB tables (usage_events, usage_session_summary, usage_tool_daily,
usage_plugin_daily) stay — renaming them would force a schema migration
+ a redo of the LLM Text-to-SQL prompt for no analyst-visible win.
Changes:
- Admin dropdown: "Tool usage" → "Telemetry"
- Page H1 / <title>: same
- URL: /admin/usage → /admin/telemetry; old URL 308-redirects
- API prefix: /api/admin/usage/* → /api/admin/telemetry/*
- CLI: primary command `agnes admin telemetry …`; `agnes admin usage` kept
as a deprecated alias so existing operator scripts keep working
- Internal data-source table id: agnes_usage → agnes_telemetry. The
registry seed now evicts any stale internal-source row whose id no
longer matches INTERNAL_TABLES, so the old `agnes_usage` row is
removed from table_registry on next app boot
- All tests + JS endpoint paths updated
* test(rbac): include auto-appended internal tables in expectations
get_accessible_tables now appends agnes_sessions / agnes_telemetry /
agnes_audit to every authenticated user's accessible-tables list so the
internal data source shows up in /catalog. The two existing rbac tests
asserted hardcoded list shapes that pre-dated the change.
Rewritten to assert "granted tables + the canonical internal-table set"
instead of literal lists, so the test stays correct if the internal
table roster changes again later.
* ui: visual dividers between admin-dropdown sections
Adds a 1px top border + 6px top margin to every section header except
the first, so the five named groups (Activity Center, Users & Access,
Data, Agent Experience, Server) read as visually separated clusters.
The header itself stays small-caps + muted as before — the border is
additive.
* ui(memory): match obs-topbar visual on /corporate-memory
The Curated Knowledge page (linked from the admin dropdown's Agent
Experience section) opened straight into the stats bar — no title,
no subtitle, no shared chrome with the other admin pages. Adds an
obs-topbar-style header at the top of .container-memory:
- H1 "Curated Knowledge"
- subtitle explaining what the page is + how AI agents pull from it
The `.ck-*` class set duplicates the inline obs-* styles from
/admin/activity etc. for this one page; promoting the obs-* class set
to style-custom.css for shared reuse is the obvious next step (4 pages
already inline the same CSS), tracked as a follow-up.
Page <title> also renamed from "Corporate Memory" → "Curated Knowledge".
* ui(tables): list Agnes internal tables in /admin/tables + group in /catalog
/admin/tables previously rendered three per-source-type listings
(BQ / Keboola / Jira) and dropped any row whose source_type didn't
match — so the agnes_sessions / agnes_telemetry / agnes_audit rows
seeded into table_registry were invisible. Adds a fourth read-only
section "Agnes internal tables" that filters source_type === 'internal'
and renders the same registry-table layout the other sections use,
with two changes:
- no Register button (these rows are seeded on every app boot from
connectors/internal/registry.py)
- Edit + Delete actions hidden (any change would be reverted on the
next start). Manage access stays so admins can still inspect.
Mode badge picks up a new mode-internal CSS class (teal accent) so the
display doesn't lie and call it "local".
In /catalog, internal tables now group under an "agnes" accordion
section (bucket="agnes" on seed) instead of falling into the catch-all
"default". Single source of truth for which tables exist; admins find
them where they expect.
* ui(tables): Agnes internal as a 4th tab next to BQ/Keboola/Jira
Previous iteration mounted the internal-table listing as a separate
standalone card under the tab strip. Reshapes it to a proper
tab-content section so admins switch between data sources via one
consistent nav (BigQuery / Keboola / Jira / Agnes internal).
- New tab button "Agnes internal" in the tab-nav.
- The listing card becomes <section id="tab-content-internal"
class="tab-content">; switchTab() already routes by id so no JS
change beyond extending the hash allowlist for direct #internal
links.
- Tab content keeps the read-only treatment from the previous commit
(no Register button, no Edit / Delete in renderRegistryListing).
* ui: rename Curated Knowledge → Curated Memory
Settles the naming back on "Curated Memory" — parallel structure with
"Curated Marketplaces" in the same Agent Experience section, and zero
rename ripple: URL (/corporate-memory), API (/api/memory/*), CLI
(agnes admin memory), and Python modules all stay on "memory" so the
admin label finally lines up with the underlying surfaces.
The "Curated" prefix still tells admins what they do on the page
(review pending → approve / mandate / reject) and reads as a sibling
of "Curated Marketplaces" right next to it in the dropdown.
Touches: admin dropdown label, page <title>, page H1. DB tables stay
on knowledge_* (already the canonical naming for the data shape).
* ui: rename "Server activity" → "Audit log"
"Audit log" is what the page actually is — server-side audit_log table
rendered with KPI cards + filter bar + sortable table. The "Server
activity" label confused the term with Claude Code session telemetry
(Telemetry page) and didn't make the source/concept clear.
Touches:
- Admin dropdown nav label
- /admin/activity page H1 + subtitle
- /admin/telemetry subtitle cross-link
- test_activity_api page-renders assertion
URL (/admin/activity) and API (/api/admin/activity/*) stay — the
"activity" name has stuck at the route layer for a year; rerouting
those would churn dashboards/bookmarks for zero analyst-visible win.
* ui(admin-nav): gray band on each section header for clearer separation
Previous iteration used a 1px top border between section labels — the
labels still blended into the items above/below at a glance. Switches
to a light gray background band per section header, extended edge-to-
edge inside the panel via negative horizontal margins. Bolder
font-weight (700) reinforces the separation; bumping the font color
isn't needed because the band itself does the work.
First section's header tucks into the panel's top border-radius so the
band reaches the corners without a gap.
* ui(catalog): rename internal-table category to "Agnes Internal"
`bucket` is what /catalog renders as the accordion category header
verbatim — "agnes" lowercase didn't read as a real category name and
got confused with a system identifier. Bumps to "Agnes Internal".
Seed re-applies on every app boot so existing rows pick up the new
bucket value via `ON CONFLICT (id) DO UPDATE`.
* ui(catalog): split Agnes Internal into its own card on /catalog
Previously the three internal tables landed inside the "Core Business
Data" card under an "Agnes Internal" accordion alongside Keboola / BQ
buckets — readers conflated system telemetry with business datasets,
and the data_stats header counter ("3 tables · ~X rows total") only
ever counted synced rows so internal tables looked invisible.
Split the catalog page into two cards:
- Core Business Data: only non-internal source_types (Keboola, BQ,
Jira). Accordions group by bucket as before. Stats counter reflects
this card's tables.
- Agnes Internal: a dedicated card with its own visual treatment
(teal accent matching the mode-internal badge in /admin/tables).
Flat list (no accordion — only 3 rows, never grows here), each
row carries the canonical `agnes query` snippet. Read-only — no
profiler click, no In-stack toggle, no sync metadata.
Route adds `internal_card` context object; template renders the new
card only when it's non-None.
* fix(rbac): hide internal tables from /admin/access + drop "my" framing
Two related cleanups for the Agnes-internal tables:
1. /admin/access (resource grants) no longer lists them. The
`can_access` check has a hardcoded internal-table bypass — security
is row-level (per-request view filter), so a table-grain
`resource_grants` row would do nothing. Surfacing them in the UI
let admins set up grants that silently no-op. Filter at the
`_table_blocks` projection so the UI tree never sees them.
2. Display names drop the analyst-perspective "my" framing:
"Agnes — my sessions" → "Agnes sessions"
"Agnes — my telemetry events" → "Agnes telemetry events"
"Agnes — my audit log" → "Agnes audit log"
The "my" only makes sense from the querying analyst's seat
(`SELECT … FROM agnes_sessions` returns *their* rows); on /admin/*
pages where admin sees / configures them across users, the
pronoun was misleading. Description text now spells out the
row-level RBAC contract explicitly.
Display names update via TableRegistryRepository.register's ON CONFLICT
UPDATE on next app boot; no manual cleanup needed.
* ui: subtitle notes about agnes_* tables on each Activity Center page
The recursive observability story — Agnes serves its own audit /
telemetry / session data through the same `agnes query` plumbing
analysts use for business data — wasn't surfaced anywhere on the
admin pages that show that data. Three pages get a one-liner with
the canonical `agnes query` snippet + the RBAC contract (analysts
see their own rows, admin sees all):
- /admin/activity (Audit log) → agnes_audit
- /admin/telemetry (Tool usage) → agnes_telemetry
- /admin/sessions → agnes_sessions
Sets up the discovery moment for admins: they're reading the page,
they see "you can query this from Claude Code", they remember it
when an analyst asks "how do I find my own failed tool calls?".
* ui(tables): explain "Show log" empty-state on /admin/tables
Cache warmup log <pre> renders with a dark background and is only
populated by the SSE stream during a Re-warm all run. Opening the
page cold + clicking Show log just revealed a black bar with no
context — admins couldn't tell what they were looking at.
Adds an inline paragraph above the <pre> explaining what the log is,
the row format, when it fills in, and where to find the historical
audit trail (/admin/activity). The actual <pre> stays empty until
SSE events arrive, but the surrounding copy carries the meaning.
* ui(tables): auto-open cache-warmup log on Re-warm all click
A Re-warm all run takes ~24s per remote BQ row. With the <details>
collapsed by default, operators saw the button disable, watched a
quiet ~24s pass, and assumed nothing had happened — the streaming
log was hidden behind a closed disclosure.
Two small JS tweaks:
- cacheWarmupRun() opens the details on click, so streamed lines
appear without an extra interaction
- cacheWarmupOnStart() hides the inline hint paragraph the moment
real log content lands, so the dark log block isn't competing
with redundant context
Hint paragraph also clarifies that only `query_mode='remote'` BQ
rows are warmed — operators with only materialized/internal tables
would see total=0 and the page would "do nothing" by spec.
* ui: trim Agnes internal copy across surfaces
Descriptions had grown to explain the extraction pipeline ("parsed
out of session JSONLs"), the underlying table ("Backed by
usage_session_summary"), the RBAC mechanic ("row-level RBAC at query
time — analysts see their own; admin sees all"), and the SQL snippet.
Every implementation detail meant another rewrite on the next iter.
Strips to one stable line per surface: what the data is, plus
"Also available locally for analysis". Mechanics live in code +
docs; the page copy says what the user needs to know.
Touched:
- connectors/internal/access.py: INTERNAL_TABLES descriptions
- activity_center.html / admin_usage.html / admin_sessions.html
subtitles
- catalog.html Agnes Internal card description + row strip
- admin_tables.html "Agnes internal" tab hint
* fix(internal): is_user_admin arity bugs + + saved-view payload cap
Round-1 code review (PR #278) caught two blocking bugs and three nits.
Blocking — both `is_user_admin(user)` (single dict arg) calls raised
TypeError. is_user_admin signature is `(user_id, conn)`. Affected:
- app/api/query.py:_run_internal_query — every POST /api/query that
references agnes_sessions / agnes_telemetry / agnes_audit blew up
with a 500. The headline analyst-facing feature of this PR was
unusable through the API.
- app/api/v2_sample.py — same shape; `GET /api/v2/sample/agnes_*`
returned 500.
Both fixed to call `is_user_admin(user.get("id"), conn)`. Added two
FastAPI-level tests in test_internal_data_source.py that go through
the TestClient — the existing unit tests on `execute_internal_query`
and `build_filter_clause` skipped the request-handler layer where the
bugs lived, which is why this landed.
Nits also closed:
- connectors/internal/access.py: `+` allowed in _USERNAME_RE /
_USER_ID_RE so RFC 5321 email local-parts (alice+test@x) resolve
correctly without hitting InternalAccessError.
- app/api/observability.py: saved-view payload capped at 64 KiB to
prevent an admin from bloating system.duckdb with a malformed save.
* fix(security): close non-admin data-leak via underlying-table refs
PR #278 R2 review surfaced a non-admin-exploitable bypass: SQL whose
string literal contains 'agnes_sessions' routed into the privileged
internal-query path, then queried the underlying physical table
(usage_session_summary / usage_events / audit_log) directly, escaping
the CTE wrapper's row filter. Two reinforcing defenses:
1. find_internal_refs() now strips single-quoted string literals
before scanning for alias names — a literal alone no longer
routes the request into the privileged code path.
2. execute_internal_query() rejects non-admin SQL that references
the underlying physical tables (usage_*, audit_log). The CTE
wrapper only scopes the agnes_* aliases; a direct FROM on the
base table — or a shadowing inner WITH that still has to read
the base table — bypasses RBAC. Block before execution with an
actionable error pointing to the agnes_* alias. Admins are
unaffected (god-mode short-circuit on the filter clause).
3. tests/test_internal_data_source.py — three new negative tests
covering literal-only matches, direct-table refs, and CTE
shadow attempts.
Also tightens usage_ask.py's SELECT-only validator: pragma_table_info,
pragma_storage_info, pragma_database_*, and duckdb_tables / columns /
views / indexes / schemas are reflection functions that leak metadata
the analyst question shouldn't reach. \bPRAGMA\b in _FORBIDDEN never
matched the function-call form (word-boundary between `A` and `_`).
* fix(security): dynamic denylist for non-admin internal queries
R3 review (PR #278) caught a wider data-leak than R2: the underlying-
physical-table guard listed only the 7 usage_* + audit_log tables,
but system.duckdb has 30+ other sensitive tables — users (emails +
ids), personal_access_tokens, resource_grants, user_groups,
user_observability_views, store_*, marketplace_*, knowledge_*, etc.
A non-admin SQL like
SELECT * FROM agnes_sessions
UNION ALL SELECT email, id, … FROM users LIMIT 1
would leak every user's row.
Replaces the hardcoded denylist with a **dynamic allowlist** —
non-admin SQL may reference ONLY the registered agnes_* aliases.
Every other table in `information_schema.tables` (main schema) is
rejected. Future migrations that add a new sensitive table are
automatically covered without re-editing this module.
Also strips SQL comments (`/* */` and `--`) before the identifier
scan so a comment-wrapped table name (`/**/users/**/`) can't slip
past the regex.
Four new negative tests pin: `users`, `personal_access_tokens`,
block-comment wrap, line-comment wrap.
Plus: per-user view-count cap (100) on /api/admin/observability/views
so an admin can't fill system.duckdb with thousands of saved views.
* release: 0.54.0 — Activity Center + Telemetry + Sessions + internal datasource
Cuts the work shipped across this PR (Activity Center build, recursive
internal data source) into a versioned release. Bumps pyproject.toml
to 0.54.0; renames the top of CHANGELOG.md from [Unreleased] to
[0.54.0] — 2026-05-12 with a header summary; opens a fresh
[Unreleased] section for the next round.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
86 KiB
Activity Center MVP — Implementation Plan
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: Rebuild /activity-center as /admin/activity — a health pulse + chronological timeline view fed by audit_log + sync_history. Close 4 audit-coverage gaps so the timeline shows what actually happens on the server.
Architecture: Single new admin handler at /admin/activity reads from audit_log and sync_history via expanded repositories. Schema migration v39→v40 adds four columns and three indices on audit_log. Four endpoints that today bypass the audit log start writing to it. Template rebuilt from scratch — the old executive-pulse / maturity-roadmap demo content is deleted. Adheres to parent spec 2026-05-11-admin-observability-spec.md.
Tech Stack: FastAPI, DuckDB, Jinja2, pytest, Typer (CLI). PostHog optional (no-op when POSTHOG_API_KEY unset).
File structure
Files to CREATE
app/api/activity.py— read endpoints for AC (/api/admin/activity,/api/admin/activity/health,/api/admin/activity/sync)tests/test_activity_api.py— endpoint teststests/test_schema_v40_migration.py— migration round-trip testtests/test_audit_repository_query.py— repository filter / cursor teststests/test_sync_history_recent.py— sync history aggregation tests
Files to MODIFY
src/db.py— bumpSCHEMA_VERSIONto 40, add_v39_to_v40migration function, add indicessrc/repositories/audit.py— rewritequery()with rich filters + cursor pagination; addparams_before/client_ip/client_kind/correlation_idkwargs tolog()src/repositories/sync_state.py— addlist_recent(since: datetime, limit: int)method (cross-table)app/api/sync.py— addAuditRepository.log()call toPOST /api/sync/triggerapp/api/scripts.py— add audit toPOST /api/scripts/run-dueapp/api/upload.py— add audit toPOST /api/upload/sessionsapp/api/data.py— add audit toGET /api/data/{table_id}/downloadapp/web/router.py— replace/activity-centerhandler with redirect to/admin/activity; add new/admin/activityhandler underrequire_adminapp/web/templates/activity_center.html— DELETE all demo content, replace with new admin-activity template (or rename + slim down)app/web/templates/_app_header.html— add admin nav link to/admin/activityapp/web/templates/dashboard.html— update widget link from/activity-centerto/admin/activityapp/main.py— registerapp/api/activity.pyrouterCHANGELOG.md—[Unreleased]entry with BREAKING marker
Files to DELETE
(none — activity_center.html is rewritten, not deleted)
Conventions (verified against origin/main; reviewer-corrected)
Imports — use these exact paths:
from app.auth.dependencies import _get_db # NOT app.dependencies
from app.auth.access import require_admin
from src.db import get_system_db
from src.repositories.audit import AuditRepository
from src.repositories.sync_state import SyncStateRepository
Test fixtures (tests/conftest.py:193):
def test_x(seeded_app, admin_user):
c = seeded_app["client"] # the FastAPI TestClient
resp = c.get("/api/foo", headers=admin_user) # admin_user is a dict like {"Authorization": "Bearer …"}
# For DB introspection in tests, open a fresh connection:
from src.db import get_system_db
conn = get_system_db()
n = conn.execute("SELECT COUNT(*) FROM audit_log").fetchone()[0]
conn.close()
There is no admin_client, authenticated_client, or get_system_conn fixture. Use the pattern above in every test in this plan.
Resilience rules (apply to every audit write added by this plan):
- Wrap
AuditRepository(conn).log(...)calls intry/exceptso a DB-locked / disk-full event doesn't 5xx the underlying business request. Log the exception vialogger.exceptionand continue. - Cap any user-controlled string before storing in
params:value[:256](and append"…"if truncated). - Sanitize filenames stored in audit: keep only
[A-Za-z0-9._\-]characters; reject other chars with a 400.
Audit suppression scope (Task 12):
_RECENT_AUDITS is a per-process dict. This means recursive-audit suppression is per uvicorn worker. For MVP we assume single-worker uvicorn (existing Agnes default in compose). If multi-worker is later enabled, this needs to move to a shared store. Comment must say so in code.
DuckDB index notes:
DuckDB's CREATE INDEX doesn't honor DESC; index direction is implementation-defined. The migration creates plain (non-DESC) indices; the planner picks them up either direction. Index creation on a populated audit_log (100k+ rows) is single-threaded and can take 30–60s — document upgrade window in CHANGELOG.
Task 1: Schema v40 migration
Adds four columns + three indices to audit_log.
Files:
-
Modify:
src/db.py:43(SCHEMA_VERSION) -
Modify:
src/db.py(add_v39_to_v40function and migration step in the ladder) -
Test:
tests/test_schema_v40_migration.py -
Step 1.1: Write the failing migration test
Create tests/test_schema_v40_migration.py:
"""v39 → v40 migration: add params_before, client_ip, client_kind,
correlation_id columns to audit_log + three indices."""
import duckdb
import pytest
from src.db import init_database, SCHEMA_VERSION
def test_schema_version_is_40():
assert SCHEMA_VERSION == 40
def test_v40_columns_exist_after_init(tmp_path):
db_path = tmp_path / "test.duckdb"
conn = duckdb.connect(str(db_path))
init_database(conn)
cols = {row[1] for row in conn.execute("PRAGMA table_info(audit_log)").fetchall()}
assert "params_before" in cols
assert "client_ip" in cols
assert "client_kind" in cols
assert "correlation_id" in cols
conn.close()
def test_v40_indices_exist(tmp_path):
db_path = tmp_path / "test.duckdb"
conn = duckdb.connect(str(db_path))
init_database(conn)
# DuckDB exposes indices via duckdb_indexes()
idx_names = {row[0] for row in conn.execute(
"SELECT index_name FROM duckdb_indexes WHERE table_name='audit_log'"
).fetchall()}
assert "idx_audit_timestamp_desc" in idx_names
assert "idx_audit_user_time" in idx_names
assert "idx_audit_action_time" in idx_names
conn.close()
def test_v39_to_v40_is_idempotent(tmp_path):
"""Running the migration twice in a row is a no-op the second time."""
db_path = tmp_path / "twice.duckdb"
conn = duckdb.connect(str(db_path))
init_database(conn)
# Second open + init must not raise (IF NOT EXISTS guards do the work)
conn.close()
conn = duckdb.connect(str(db_path))
init_database(conn)
version = conn.execute("SELECT MAX(version) FROM schema_version").fetchone()[0]
assert version == 40
conn.close()
def test_v30_db_ladders_all_the_way_up(tmp_path):
"""Representative evolved-DB test: an instance hand-rolled at v30 must
ladder through to v40 without data loss, mirroring a customer who's
been upgrading regularly since older releases. If this fails, ANY
intermediate migration is broken — surface the offending step."""
db_path = tmp_path / "v30.duckdb"
conn = duckdb.connect(str(db_path))
# Minimal v30 baseline (only the table needed to assert preservation).
conn.execute("""
CREATE TABLE audit_log (
id VARCHAR PRIMARY KEY,
timestamp TIMESTAMP NOT NULL DEFAULT current_timestamp,
user_id VARCHAR,
action VARCHAR NOT NULL,
resource VARCHAR,
params JSON,
result VARCHAR,
duration_ms INTEGER
)
""")
conn.execute("CREATE TABLE schema_version (version INTEGER)")
conn.execute("INSERT INTO schema_version VALUES (30)")
conn.execute("INSERT INTO audit_log (id, action) VALUES ('vintage', 'test.x')")
conn.close()
conn = duckdb.connect(str(db_path))
init_database(conn)
version = conn.execute("SELECT MAX(version) FROM schema_version").fetchone()[0]
assert version == 40
assert conn.execute("SELECT COUNT(*) FROM audit_log WHERE id='vintage'").fetchone()[0] == 1
conn.close()
def test_v39_db_upgrades_cleanly(tmp_path):
"""A DB hand-rolled at v39 (audit_log without the four new columns)
must upgrade to v40 without data loss."""
db_path = tmp_path / "v39.duckdb"
conn = duckdb.connect(str(db_path))
# Hand-roll the v39 audit_log shape
conn.execute("""
CREATE TABLE audit_log (
id VARCHAR PRIMARY KEY,
timestamp TIMESTAMP NOT NULL DEFAULT current_timestamp,
user_id VARCHAR,
action VARCHAR NOT NULL,
resource VARCHAR,
params JSON,
result VARCHAR,
duration_ms INTEGER
)
""")
conn.execute("CREATE TABLE schema_version (version INTEGER)")
conn.execute("INSERT INTO schema_version VALUES (39)")
conn.execute("INSERT INTO audit_log (id, action) VALUES ('row1', 'test.action')")
conn.close()
# Reopen and run init — should ladder v39 → v40
conn = duckdb.connect(str(db_path))
init_database(conn)
version = conn.execute("SELECT MAX(version) FROM schema_version").fetchone()[0]
assert version == 40
# Row preserved
cnt = conn.execute("SELECT COUNT(*) FROM audit_log WHERE id='row1'").fetchone()[0]
assert cnt == 1
# New columns nullable
row = conn.execute(
"SELECT params_before, client_ip, client_kind, correlation_id FROM audit_log WHERE id='row1'"
).fetchone()
assert row == (None, None, None, None)
conn.close()
- Step 1.2: Run test — expect FAIL
Run: pytest tests/test_schema_v40_migration.py -v
Expected: 4 failures — SCHEMA_VERSION == 39 (not 40), columns missing.
- Step 1.3: Implement migration in
src/db.py
In src/db.py, bump SCHEMA_VERSION:
SCHEMA_VERSION = 40
Add a new migration function (mirror the existing _v38_to_v39 / _v37_to_v38 pattern — search for them in the file). Below all existing _vN_to_vN_plus_1 functions, add:
def _v39_to_v40(conn: duckdb.DuckDBPyConnection) -> None:
"""v40: audit_log gains params_before (JSON, prior state for diff/rollback),
client_ip (VARCHAR, promoted from params for indexability), client_kind
(VARCHAR, 'cli'|'web'|'agent'|'scheduler'|'external'), and correlation_id
(VARCHAR, groups multi-step operations).
Three indices added on (timestamp DESC), (user_id, timestamp),
(action, timestamp) to keep Activity Center timeline queries under
100ms even at 100k+ rows.
"""
# Add columns idempotently (re-run safety on partial migrations)
conn.execute("ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS params_before JSON")
conn.execute("ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS client_ip VARCHAR")
conn.execute("ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS client_kind VARCHAR")
conn.execute("ALTER TABLE audit_log ADD COLUMN IF NOT EXISTS correlation_id VARCHAR")
# Indices for AC query patterns.
# NOTE: DuckDB does not honor DESC in CREATE INDEX; the planner is free to
# scan either direction. Names retain `_desc` for readability — the order
# is enforced by the ORDER BY clause in AuditRepository.query().
# On a populated audit_log (~100k+ rows), each CREATE INDEX is single-
# threaded and may take 10–30s. Cumulative cold-start cost on upgrade is
# documented in CHANGELOG as a 30–120s upgrade window.
conn.execute("CREATE INDEX IF NOT EXISTS idx_audit_timestamp_desc ON audit_log(timestamp)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_audit_user_time ON audit_log(user_id, timestamp)")
conn.execute("CREATE INDEX IF NOT EXISTS idx_audit_action_time ON audit_log(action, timestamp)")
Then add the step to the migration ladder. Locate the _run_migrations function (or equivalent — read the file to find the existing pattern; in current code it's a series of if current_version < N blocks). Add:
if current_version < 40:
_v39_to_v40(conn)
_set_schema_version(conn, 40)
current_version = 40
- Step 1.4: Run test — expect PASS
Run: pytest tests/test_schema_v40_migration.py -v
Expected: 4 passed.
Run full schema test suite to catch regressions:
pytest tests/test_db_schema_version.py -v
Expected: all pass (existing tests assert ladder steps; new v40 step should slot in cleanly).
- Step 1.5: Commit
cd "/Users/zdeneksrotyr/Library/Mobile Documents/com~apple~CloudDocs/Sources/VsCode/component_factory/tmp_oss-activity-spec"
git add src/db.py tests/test_schema_v40_migration.py
git commit -m "feat(db): schema v40 — audit_log gains params_before, client_ip, client_kind, correlation_id + 3 indices"
Task 2: Extend AuditRepository.log() with new kwargs
Existing callers must keep working unchanged. New kwargs default to None.
Files:
-
Modify:
src/repositories/audit.py -
Test:
tests/test_audit_repository_query.py -
Step 2.1: Write the failing test for new kwargs
Append to tests/test_audit_repository_query.py:
"""AuditRepository v40 — new kwargs (params_before, client_ip, client_kind,
correlation_id) round-trip; legacy callers compile-time-unbroken."""
import duckdb
import pytest
from src.db import init_database
from src.repositories.audit import AuditRepository
@pytest.fixture
def conn(tmp_path):
db_path = tmp_path / "test.duckdb"
c = duckdb.connect(str(db_path))
init_database(c)
yield c
c.close()
def test_log_accepts_new_kwargs(conn):
repo = AuditRepository(conn)
entry_id = repo.log(
user_id="u1",
action="registry.update",
resource="table:web_sessions",
params={"after": {"cron": "*/15 * * * *"}},
params_before={"cron": "0 */1 * * *"},
client_ip="10.0.0.42",
client_kind="web",
correlation_id="corr-123",
)
row = conn.execute("SELECT params_before, client_ip, client_kind, correlation_id FROM audit_log WHERE id=?", [entry_id]).fetchone()
assert row[0] is not None # JSON
assert row[1] == "10.0.0.42"
assert row[2] == "web"
assert row[3] == "corr-123"
def test_log_legacy_signature_still_works(conn):
"""The original kwargs-only call site (used by 30+ existing endpoints)
must keep working unchanged."""
repo = AuditRepository(conn)
entry_id = repo.log(user_id="u1", action="auth.login")
row = conn.execute("SELECT user_id, action, params_before FROM audit_log WHERE id=?", [entry_id]).fetchone()
assert row == ("u1", "auth.login", None)
- Step 2.2: Run test — expect FAIL
Run: pytest tests/test_audit_repository_query.py::test_log_accepts_new_kwargs tests/test_audit_repository_query.py::test_log_legacy_signature_still_works -v
Expected: FAIL — log() doesn't accept the new kwargs.
- Step 2.3: Extend
AuditRepository.log()insrc/repositories/audit.py
Open src/repositories/audit.py and replace the log() method body. The current signature is:
def log(
self,
user_id: Optional[str] = None,
action: str = "",
resource: Optional[str] = None,
params: Optional[dict] = None,
result: Optional[str] = None,
duration_ms: Optional[int] = None,
) -> str:
Replace with:
def log(
self,
user_id: Optional[str] = None,
action: str = "",
resource: Optional[str] = None,
params: Optional[dict] = None,
result: Optional[str] = None,
duration_ms: Optional[int] = None,
*,
params_before: Optional[dict] = None,
client_ip: Optional[str] = None,
client_kind: Optional[str] = None,
correlation_id: Optional[str] = None,
) -> str:
"""Insert one audit_log row. Returns the new row id.
The four kwargs after `*` are v40 additions; legacy callers using
positional args or the original kwargs are unaffected. `params_before`
is only used for mutating actions where rollback / diff is meaningful;
leave None for reads, ticks, queries.
"""
entry_id = str(uuid.uuid4())
now = datetime.now(timezone.utc)
self.conn.execute(
"""INSERT INTO audit_log
(id, timestamp, user_id, action, resource, params, result, duration_ms,
params_before, client_ip, client_kind, correlation_id)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)""",
[
entry_id, now, user_id, action, resource,
json.dumps(params) if params else None,
result, duration_ms,
json.dumps(params_before) if params_before else None,
client_ip, client_kind, correlation_id,
],
)
return entry_id
- Step 2.4: Run test — expect PASS
Run: pytest tests/test_audit_repository_query.py -v
Expected: 2 passed.
Then run the full test suite to confirm no regression in the 30+ existing call sites:
pytest tests/ -x -q --no-header 2>&1 | tail -30
Expected: no failures in audit-touching tests.
- Step 2.5: Commit
git add src/repositories/audit.py tests/test_audit_repository_query.py
git commit -m "feat(audit): AuditRepository.log() accepts params_before/client_ip/client_kind/correlation_id"
Task 3: Rewrite AuditRepository.query() with rich filters + cursor pagination
This is the data engine for the Timeline tab.
Files:
-
Modify:
src/repositories/audit.py -
Test:
tests/test_audit_repository_query.py -
Step 3.1: Write failing tests for filter combinations
Append to tests/test_audit_repository_query.py:
from datetime import datetime, timezone, timedelta
def _seed(conn, rows: list[dict]):
"""Insert audit_log rows with explicit timestamps."""
repo = AuditRepository(conn)
ids = []
for r in rows:
entry_id = repo.log(
user_id=r.get("user_id"),
action=r.get("action", "test.x"),
resource=r.get("resource"),
params=r.get("params"),
result=r.get("result"),
)
if "ts" in r:
conn.execute("UPDATE audit_log SET timestamp=? WHERE id=?", [r["ts"], entry_id])
ids.append(entry_id)
return ids
def test_query_filter_by_time_range(conn):
repo = AuditRepository(conn)
now = datetime(2026, 5, 11, 12, 0, 0, tzinfo=timezone.utc)
_seed(conn, [
{"action": "a.1", "ts": now - timedelta(hours=2)},
{"action": "a.2", "ts": now - timedelta(minutes=30)},
{"action": "a.3", "ts": now - timedelta(minutes=5)},
])
rows, _ = repo.query(since=now - timedelta(hours=1), until=now)
assert {r["action"] for r in rows} == {"a.2", "a.3"}
def test_query_filter_by_action_prefix(conn):
repo = AuditRepository(conn)
_seed(conn, [
{"action": "sync.trigger"},
{"action": "sync.complete"},
{"action": "auth.login"},
])
rows, _ = repo.query(action_prefix="sync.")
assert {r["action"] for r in rows} == {"sync.trigger", "sync.complete"}
def test_query_filter_by_action_in(conn):
repo = AuditRepository(conn)
_seed(conn, [
{"action": "a"}, {"action": "b"}, {"action": "c"},
])
rows, _ = repo.query(action_in=["a", "c"])
assert {r["action"] for r in rows} == {"a", "c"}
def test_query_filter_by_user(conn):
repo = AuditRepository(conn)
_seed(conn, [
{"user_id": "u1", "action": "x"},
{"user_id": "u2", "action": "x"},
])
rows, _ = repo.query(user_id="u1")
assert len(rows) == 1
assert rows[0]["user_id"] == "u1"
def test_query_filter_by_resource(conn):
repo = AuditRepository(conn)
_seed(conn, [
{"action": "x", "resource": "table:a"},
{"action": "x", "resource": "table:b"},
])
rows, _ = repo.query(resource="table:a")
assert len(rows) == 1
def test_query_filter_by_result_pattern(conn):
repo = AuditRepository(conn)
_seed(conn, [
{"action": "x", "result": "success"},
{"action": "x", "result": "error.timeout"},
{"action": "x", "result": "error.permission"},
])
rows, _ = repo.query(result_pattern="error.%")
assert {r["result"] for r in rows} == {"error.timeout", "error.permission"}
def test_query_full_text_q(conn):
repo = AuditRepository(conn)
_seed(conn, [
{"action": "x", "params": {"sql": "SELECT * FROM finance"}},
{"action": "x", "params": {"sql": "SELECT * FROM marketing"}},
])
rows, _ = repo.query(q="finance")
assert len(rows) == 1
def test_query_cursor_pagination(conn):
repo = AuditRepository(conn)
now = datetime(2026, 5, 11, 12, 0, 0, tzinfo=timezone.utc)
for i in range(5):
_seed(conn, [{"action": f"a.{i}", "ts": now - timedelta(minutes=i)}])
page1, cursor1 = repo.query(limit=2)
assert len(page1) == 2
assert cursor1 is not None
page2, cursor2 = repo.query(limit=2, cursor=cursor1)
assert len(page2) == 2
page3, cursor3 = repo.query(limit=2, cursor=cursor2)
assert len(page3) == 1
assert cursor3 is None
# Pages don't overlap
all_ids = {r["id"] for r in page1 + page2 + page3}
assert len(all_ids) == 5
def test_query_ordering_newest_first(conn):
repo = AuditRepository(conn)
now = datetime(2026, 5, 11, 12, 0, 0, tzinfo=timezone.utc)
_seed(conn, [
{"action": "old", "ts": now - timedelta(hours=2)},
{"action": "new", "ts": now - timedelta(minutes=1)},
])
rows, _ = repo.query()
assert rows[0]["action"] == "new"
assert rows[1]["action"] == "old"
- Step 3.2: Run tests — expect FAIL
Run: pytest tests/test_audit_repository_query.py -v -k 'test_query_'
Expected: 8 failures — the current query() only supports user_id, action, limit.
- Step 3.3: Rewrite
query()insrc/repositories/audit.py
First, ensure imports at the top of src/repositories/audit.py include:
from datetime import datetime, timezone, timedelta
from typing import Any, Dict, List, Optional
timedelta is the new addition (needed for the q-without-since safeguard below).
Replace the existing query() method with:
def query(
self,
*,
since: Optional[datetime] = None,
until: Optional[datetime] = None,
user_id: Optional[str] = None,
action: Optional[str] = None, # legacy single-action filter
action_prefix: Optional[str] = None,
action_in: Optional[List[str]] = None,
resource: Optional[str] = None,
result_pattern: Optional[str] = None,
correlation_id: Optional[str] = None,
q: Optional[str] = None,
cursor: Optional[tuple] = None, # (timestamp, id) — keyset pagination
limit: int = 100,
) -> tuple[List[Dict[str, Any]], Optional[tuple]]:
"""Query audit_log with rich filters; returns (rows, next_cursor).
Cursor encodes (timestamp, id) so pagination is stable under
same-second writes. Pass the returned cursor back as `cursor=` for
the next page. `None` cursor on input = newest page; `None` cursor
in return = last page reached.
"""
where = []
params: List[Any] = []
if since is not None:
where.append("timestamp >= ?"); params.append(since)
if until is not None:
where.append("timestamp < ?"); params.append(until)
if user_id is not None:
where.append("user_id = ?"); params.append(user_id)
if action is not None:
where.append("action = ?"); params.append(action)
if action_prefix is not None:
where.append("action LIKE ?"); params.append(action_prefix + "%")
if action_in:
placeholders = ",".join("?" for _ in action_in)
where.append(f"action IN ({placeholders})")
params.extend(action_in)
if resource is not None:
where.append("resource = ?"); params.append(resource)
if result_pattern is not None:
where.append("result LIKE ?"); params.append(result_pattern)
if correlation_id is not None:
where.append("correlation_id = ?"); params.append(correlation_id)
if q:
# Full-text search is a table scan on `params` JSON cast to text.
# Safeguard: if caller passes `q` without a `since` filter, force a
# 7-day cap so we don't scan the entire audit_log. Proper FTS lands
# in Phase B/C (see parent spec §5.5).
if since is None:
since = datetime.now(timezone.utc) - timedelta(days=7)
where.append("timestamp >= ?"); params.append(since)
where.append("CAST(params AS VARCHAR) LIKE ?"); params.append(f"%{q}%")
if cursor is not None:
ts, cid = cursor
# Keyset: rows strictly older than the cursor, breaking ties by id desc
where.append("(timestamp, id) < (?, ?)")
params.extend([ts, cid])
sql = "SELECT * FROM audit_log"
if where:
sql += " WHERE " + " AND ".join(where)
# Fetch limit+1 to determine whether there's a next page
sql += " ORDER BY timestamp DESC, id DESC LIMIT ?"
params.append(limit + 1)
rows = self.conn.execute(sql, params).fetchall()
if not rows:
return [], None
columns = [desc[0] for desc in self.conn.description]
out = [dict(zip(columns, r)) for r in rows]
next_cursor: Optional[tuple] = None
if len(out) > limit:
# The (limit+1)th row tells us "more exists"; drop it from response.
last_shown = out[limit - 1]
next_cursor = (last_shown["timestamp"], last_shown["id"])
out = out[:limit]
return out, next_cursor
- Step 3.4: Run tests — expect PASS
Run: pytest tests/test_audit_repository_query.py -v
Expected: all 10 tests pass.
Then sweep for legacy .query( callers that may pass positional args:
grep -rn 'AuditRepository.*\.query(' app/ src/ services/ | grep -v test_
Expected: any caller passes kwargs only (the new signature requires kwargs after the leading *).
If a caller is found using positional args, fix it: convert to kwargs.
- Step 3.5: Commit
git add src/repositories/audit.py tests/test_audit_repository_query.py
git commit -m "feat(audit): AuditRepository.query() rich filters + keyset cursor pagination"
Task 4: Add SyncHistoryRepository.list_recent()
The Sync tab needs cross-table sync events.
Files:
-
Modify:
src/repositories/sync_state.py -
Test:
tests/test_sync_history_recent.py -
Step 4.1: Write failing test
Create tests/test_sync_history_recent.py:
"""SyncHistoryRepository.list_recent() — cross-table chronological feed."""
import duckdb
import pytest
from datetime import datetime, timezone, timedelta
from src.db import init_database
from src.repositories.sync_state import SyncStateRepository
@pytest.fixture
def conn(tmp_path):
db_path = tmp_path / "test.duckdb"
c = duckdb.connect(str(db_path))
init_database(c)
yield c
c.close()
def _record(conn, table_id: str, synced_at: datetime, status: str = "ok", rows: int = 100):
import uuid
conn.execute(
"INSERT INTO sync_history (id, table_id, synced_at, rows, duration_ms, status, error) VALUES (?, ?, ?, ?, ?, ?, ?)",
[str(uuid.uuid4()), table_id, synced_at, rows, 1234, status, None]
)
def test_list_recent_returns_all_tables_newest_first(conn):
repo = SyncStateRepository(conn)
now = datetime(2026, 5, 11, 12, 0, 0, tzinfo=timezone.utc)
_record(conn, "orders", now - timedelta(hours=1))
_record(conn, "customers", now - timedelta(minutes=30))
_record(conn, "products", now - timedelta(minutes=5))
rows = repo.list_recent(since=now - timedelta(hours=2), limit=50)
assert [r["table_id"] for r in rows] == ["products", "customers", "orders"]
def test_list_recent_respects_since(conn):
repo = SyncStateRepository(conn)
now = datetime(2026, 5, 11, 12, 0, 0, tzinfo=timezone.utc)
_record(conn, "old", now - timedelta(days=3))
_record(conn, "new", now - timedelta(minutes=10))
rows = repo.list_recent(since=now - timedelta(hours=1), limit=50)
assert [r["table_id"] for r in rows] == ["new"]
def test_list_recent_respects_limit(conn):
repo = SyncStateRepository(conn)
now = datetime(2026, 5, 11, 12, 0, 0, tzinfo=timezone.utc)
for i in range(20):
_record(conn, f"t{i}", now - timedelta(minutes=i))
rows = repo.list_recent(since=now - timedelta(hours=1), limit=5)
assert len(rows) == 5
def test_list_recent_includes_failures(conn):
repo = SyncStateRepository(conn)
now = datetime(2026, 5, 11, 12, 0, 0, tzinfo=timezone.utc)
_record(conn, "t1", now, status="ok")
_record(conn, "t2", now, status="error")
rows = repo.list_recent(since=now - timedelta(hours=1), limit=10)
statuses = {r["table_id"]: r["status"] for r in rows}
assert statuses["t1"] == "ok"
assert statuses["t2"] == "error"
- Step 4.2: Run test — expect FAIL
Run: pytest tests/test_sync_history_recent.py -v
Expected: 4 failures — method doesn't exist.
- Step 4.3: Add
list_recent()toSyncStateRepository
Open src/repositories/sync_state.py. After the existing get_sync_history() method, add:
def list_recent(
self,
*,
since: datetime,
limit: int = 100,
status: Optional[str] = None,
) -> List[Dict[str, Any]]:
"""Return cross-table sync events newer than `since`, newest first.
Used by Activity Center's Sync tab to render a unified feed across
all registered tables. Per-table history stays available via
`get_sync_history(table_id, limit)`.
"""
sql = "SELECT * FROM sync_history WHERE synced_at >= ?"
params: List[Any] = [since]
if status is not None:
sql += " AND status = ?"
params.append(status)
sql += " ORDER BY synced_at DESC LIMIT ?"
params.append(limit)
rows = self.conn.execute(sql, params).fetchall()
if not rows:
return []
columns = [d[0] for d in self.conn.description]
return [dict(zip(columns, r)) for r in rows]
Make sure datetime and Optional are imported at the top of the file. They likely are; if not:
from datetime import datetime
from typing import Any, Dict, List, Optional
- Step 4.4: Run test — expect PASS
Run: pytest tests/test_sync_history_recent.py -v
Expected: 4 passed.
- Step 4.5: Commit
git add src/repositories/sync_state.py tests/test_sync_history_recent.py
git commit -m "feat(sync): SyncStateRepository.list_recent() cross-table feed"
Task 5: Close audit gap 1/4 — POST /api/sync/trigger
Files:
-
Modify:
app/api/sync.py -
Test:
tests/test_audit_gap_sync_trigger.py -
Step 5.1: Write failing test
Create tests/test_audit_gap_sync_trigger.py:
"""POST /api/sync/trigger must write to audit_log (closes coverage gap).
Uses canonical fixtures (Conventions section): seeded_app["client"] + admin_user
headers + get_system_db() for direct DB access.
"""
import pytest
from src.db import get_system_db
def test_sync_trigger_writes_audit_log(seeded_app, admin_user):
c = seeded_app["client"]
conn = get_system_db()
before = conn.execute(
"SELECT COUNT(*) FROM audit_log WHERE action='sync.trigger'"
).fetchone()[0]
conn.close()
resp = c.post("/api/sync/trigger", headers=admin_user)
assert resp.status_code in (200, 202)
conn = get_system_db()
after = conn.execute(
"SELECT COUNT(*) FROM audit_log WHERE action='sync.trigger'"
).fetchone()[0]
assert after == before + 1
row = conn.execute(
"SELECT user_id, action, result FROM audit_log WHERE action='sync.trigger' ORDER BY timestamp DESC LIMIT 1"
).fetchone()
conn.close()
assert row[0] is not None # user_id captured
assert row[1] == "sync.trigger"
assert row[2] in ("success", "error.in_progress", "error.locked")
def test_sync_trigger_does_not_5xx_if_audit_write_fails(seeded_app, admin_user, monkeypatch):
"""Resilience rule (Conventions): a failed audit write must NOT crash
the wrapped business request — log + swallow + continue."""
from src.repositories.audit import AuditRepository
def boom(*args, **kwargs):
raise duckdb.IOException("simulated DB-locked")
monkeypatch.setattr(AuditRepository, "log", boom)
c = seeded_app["client"]
resp = c.post("/api/sync/trigger", headers=admin_user)
# The sync trigger itself must still respond — audit failure is invisible.
assert resp.status_code in (200, 202, 409)
- Step 5.2: Run test — expect FAIL
Run: pytest tests/test_audit_gap_sync_trigger.py -v
Expected: FAIL — assert after == before + 1 fails because no row written.
- Step 5.3: Add audit call in
app/api/sync.py
Open app/api/sync.py around line 772 (the POST /api/sync/trigger handler). The current body looks like:
@router.post("/sync/trigger")
async def trigger_sync(...):
# ... existing logic ...
return {"status": "ok", ...}
Add the audit call. Locate the imports at the top of the file. Add (if not already present):
from src.repositories.audit import AuditRepository
Inside the handler, after determining user_id and the eventual result/status, add a try/except-wrapped audit call (per Conventions, resilience rule #1):
try:
AuditRepository(conn).log(
user_id=user_id,
action="sync.trigger",
resource=(table_id or "all_tables")[:256],
params={"requested_at": datetime.now(timezone.utc).isoformat()},
result=result_status, # 'success' | 'error.locked' | 'error.in_progress' | …
duration_ms=int((time.monotonic() - t0) * 1000) if t0 else None,
client_kind="scheduler" if is_scheduler_caller else "web",
)
except Exception:
logger.exception("audit_log write failed for sync.trigger; continuing")
Notes:
-
The exact variable names depend on the current handler body — read it before this step.
-
Look for an existing pattern in
app/api/admin.py:1120for reference style. -
is_scheduler_callercan be derived from whetherSCHEDULER_API_TOKENmatched (the handler likely has this check already). -
Ensure
loggeris in scope at the top of the file (most Agnes API modules already have one). -
Step 5.4: Run test — expect PASS
Run: pytest tests/test_audit_gap_sync_trigger.py -v
Expected: 1 passed.
Run regression: pytest tests/test_sync*.py -v to ensure existing sync tests still pass.
- Step 5.5: Commit
git add app/api/sync.py tests/test_audit_gap_sync_trigger.py
git commit -m "feat(audit): POST /api/sync/trigger writes audit_log row"
Task 6: Close audit gap 2/4 — POST /api/scripts/run-due
Files:
-
Modify:
app/api/scripts.py -
Test:
tests/test_audit_gap_scripts_run_due.py -
Step 6.1: Write failing test
Create tests/test_audit_gap_scripts_run_due.py:
"""POST /api/scripts/run-due must write to audit_log."""
from src.db import get_system_db
def test_scripts_run_due_writes_audit_log(seeded_app, admin_user):
c = seeded_app["client"]
conn = get_system_db()
before = conn.execute(
"SELECT COUNT(*) FROM audit_log WHERE action='script_runner.tick'"
).fetchone()[0]
conn.close()
resp = c.post("/api/scripts/run-due", headers=admin_user)
assert resp.status_code in (200, 202)
conn = get_system_db()
after = conn.execute(
"SELECT COUNT(*) FROM audit_log WHERE action='script_runner.tick'"
).fetchone()[0]
conn.close()
assert after == before + 1
- Step 6.2: Run test — expect FAIL
Run: pytest tests/test_audit_gap_scripts_run_due.py -v
Expected: FAIL.
- Step 6.3: Add audit call in
app/api/scripts.py
Open app/api/scripts.py. Locate the run_due handler around line 138. Add at the end of the handler body (before the return statement), wrapped per Conventions:
try:
AuditRepository(conn).log(
user_id=user_id,
action="script_runner.tick",
params={"scripts_run": scripts_run_count, "scripts_failed": scripts_failed_count},
result="success" if scripts_failed_count == 0 else f"error.{scripts_failed_count}_failed",
client_kind="scheduler",
)
except Exception:
logger.exception("audit_log write failed for script_runner.tick; continuing")
Adjust variable names to match the actual handler's locals.
- Step 6.4: Run test — expect PASS
Run: pytest tests/test_audit_gap_scripts_run_due.py -v
Expected: 1 passed.
- Step 6.5: Commit
git add app/api/scripts.py tests/test_audit_gap_scripts_run_due.py
git commit -m "feat(audit): POST /api/scripts/run-due writes audit_log row"
Task 7: Close audit gap 3/4 — POST /api/upload/sessions
Files:
-
Modify:
app/api/upload.py -
Test:
tests/test_audit_gap_upload_sessions.py -
Step 7.1: Write failing test
Create tests/test_audit_gap_upload_sessions.py:
"""POST /api/upload/sessions must write to audit_log; filename is sanitized."""
import io
import json
from src.db import get_system_db
def test_upload_sessions_writes_audit_log(seeded_app, analyst_user):
c = seeded_app["client"]
conn = get_system_db()
before = conn.execute(
"SELECT COUNT(*) FROM audit_log WHERE action='session.upload'"
).fetchone()[0]
conn.close()
jsonl = b'{"role":"user","content":"hello"}\n{"role":"assistant","content":"hi"}\n'
files = {"file": ("sess-test.jsonl", io.BytesIO(jsonl), "application/x-ndjson")}
resp = c.post("/api/upload/sessions", files=files, headers=analyst_user)
assert resp.status_code == 200
conn = get_system_db()
after = conn.execute(
"SELECT COUNT(*) FROM audit_log WHERE action='session.upload'"
).fetchone()[0]
row = conn.execute(
"SELECT params FROM audit_log WHERE action='session.upload' ORDER BY timestamp DESC LIMIT 1"
).fetchone()
conn.close()
assert after == before + 1
params = json.loads(row[0]) if row[0] else {}
assert "filename" in params
assert "bytes" in params
def test_upload_sessions_rejects_dangerous_filename(seeded_app, analyst_user):
"""Conventions sanitization rule #3 — filename limited to [A-Za-z0-9._-]."""
c = seeded_app["client"]
jsonl = b'{"role":"user","content":"x"}\n'
files = {"file": ("<script>alert(1)</script>.jsonl", io.BytesIO(jsonl), "application/x-ndjson")}
resp = c.post("/api/upload/sessions", files=files, headers=analyst_user)
assert resp.status_code == 400
assert "filename" in resp.text.lower()
- Step 7.2: Run test — expect FAIL
Run: pytest tests/test_audit_gap_upload_sessions.py -v
Expected: FAIL.
- Step 7.3: Add filename sanitization + audit call in
app/api/upload.py
Open app/api/upload.py. Locate the upload handler around line 55. First, add sanitization at the top of the handler (after parsing the multipart):
import re
_FILENAME_RE = re.compile(r"^[A-Za-z0-9._\-]{1,200}$")
# At the top of the handler:
if not _FILENAME_RE.match(file.filename or ""):
raise HTTPException(
status_code=400,
detail="filename must match [A-Za-z0-9._-]{1,200}"
)
Then, after the file has been written to disk and the handler is about to return, add the audit call wrapped in try/except per Conventions:
try:
AuditRepository(conn).log(
user_id=user_id,
action="session.upload",
params={"filename": stored_filename[:256], "bytes": file_size_bytes},
result="success",
client_kind="cli",
)
except Exception:
logger.exception("audit_log write failed for session.upload; continuing")
- Step 7.4: Run test — expect PASS
Run: pytest tests/test_audit_gap_upload_sessions.py -v
Expected: 1 passed.
- Step 7.5: Commit
git add app/api/upload.py tests/test_audit_gap_upload_sessions.py
git commit -m "feat(audit): POST /api/upload/sessions writes audit_log row"
Task 8: Close audit gap 4/4 — GET /api/data/{table_id}/download
Files:
-
Modify:
app/api/data.py -
Test:
tests/test_audit_gap_data_download.py -
Step 8.1: Write failing test
Create tests/test_audit_gap_data_download.py:
"""GET /api/data/{table_id}/download must write to audit_log."""
from src.db import get_system_db
from tests.conftest import create_mock_extract
def test_data_download_writes_audit_log(seeded_app, analyst_user, mock_extract_factory):
"""mock_extract_factory (conftest.py:244) creates extract.duckdb + parquet
on disk and registers it via the standard extract path."""
mock_extract_factory("test_src", [
{"name": "test_tbl", "data": [{"a": "1", "b": "2"}], "query_mode": "local"},
])
# NOTE: register the table via /api/admin/register-table before downloading.
# Pattern from existing tests, e.g. tests/test_journey_*.py.
c = seeded_app["client"]
# ... register table (read tests/test_journey_*.py for an example) ...
conn = get_system_db()
before = conn.execute(
"SELECT COUNT(*) FROM audit_log WHERE action='data.download'"
).fetchone()[0]
conn.close()
resp = c.get("/api/data/test_tbl/download", headers=analyst_user)
assert resp.status_code == 200
conn = get_system_db()
after = conn.execute(
"SELECT COUNT(*) FROM audit_log WHERE action='data.download'"
).fetchone()[0]
conn.close()
assert after == before + 1
NOTE: this test requires a registered table with an on-disk parquet. The
mock_extract_factory fixture provides the on-disk part; registration goes
via POST /api/admin/register-table (existing endpoint). If the table-
registration prelude is non-trivial, factor it into a helper inside this
test file rather than depending on a non-existent seeded_table fixture.
- Step 8.2: Run test — expect FAIL
Run: pytest tests/test_audit_gap_data_download.py -v
Expected: FAIL.
- Step 8.3: Add audit call in
app/api/data.py
Open app/api/data.py:45. The download handler returns a FileResponse. Add the audit call BEFORE the return, wrapped per Conventions:
try:
AuditRepository(conn).log(
user_id=user_id,
action="data.download",
resource=f"table:{table_id}"[:256],
params={"bytes": file_size, "format": "parquet"},
result="success",
client_kind="cli",
)
except Exception:
logger.exception("audit_log write failed for data.download; continuing")
- Step 8.4: Run test — expect PASS
Run: pytest tests/test_audit_gap_data_download.py -v
Expected: 1 passed.
- Step 8.5: Commit
git add app/api/data.py tests/test_audit_gap_data_download.py
git commit -m "feat(audit): GET /api/data/{table_id}/download writes audit_log row"
Task 9: New API module app/api/activity.py
Three read endpoints: timeline, health, sync.
Files:
-
Create:
app/api/activity.py -
Modify:
app/main.py(register router) -
Test:
tests/test_activity_api.py -
Step 9.1: Write failing test for
/api/admin/activitytimeline
Create tests/test_activity_api.py:
"""Activity Center read API."""
from datetime import datetime, timezone, timedelta
from fastapi.testclient import TestClient
def test_activity_timeline_requires_admin(seeded_app, analyst_user):
"""Non-admin user gets 403."""
resp = seeded_app["client"].get("/api/admin/activity", headers=analyst_user)
assert resp.status_code in (401, 403)
def test_activity_timeline_returns_recent_rows(seeded_app, admin_user):
"""Seeded audit_log rows appear in the response."""
from src.db import get_system_db
from src.repositories.audit import AuditRepository
conn = get_system_db()
AuditRepository(conn).log(user_id="u1", action="test.activity", result="success")
conn.close()
resp = seeded_app["client"].get("/api/admin/activity", headers=admin_user)
assert resp.status_code == 200
data = resp.json()
assert "rows" in data
assert "next_cursor" in data
assert any(r["action"] == "test.activity" for r in data["rows"])
def test_activity_timeline_supports_filters(seeded_app, admin_user):
from src.db import get_system_db
from src.repositories.audit import AuditRepository
conn = get_system_db()
repo = AuditRepository(conn)
repo.log(action="sync.trigger")
repo.log(action="auth.login")
conn.close()
resp = seeded_app["client"].get("/api/admin/activity?action_prefix=sync.", headers=admin_user)
assert resp.status_code == 200
actions = {r["action"] for r in resp.json()["rows"]}
assert "sync.trigger" in actions
assert "auth.login" not in actions
def test_activity_health_returns_pulse(seeded_app, admin_user):
resp = seeded_app["client"].get("/api/admin/activity/health", headers=admin_user)
assert resp.status_code == 200
data = resp.json()
assert data["status"] in ("green", "yellow", "red")
assert "fields" in data
assert "sentence" in data
field_keys = {f["key"] for f in data["fields"]}
assert "scheduler" in field_keys
assert "sync_24h" in field_keys
assert "active_users_today" in field_keys
def test_activity_sync_returns_recent(seeded_app, admin_user):
import uuid
from src.db import get_system_db
now = datetime.now(timezone.utc)
conn = get_system_db()
conn.execute(
"INSERT INTO sync_history (id, table_id, synced_at, rows, duration_ms, status, error) VALUES (?, ?, ?, ?, ?, ?, ?)",
[str(uuid.uuid4()), "t_test", now, 42, 1500, "ok", None]
)
conn.close()
resp = seeded_app["client"].get("/api/admin/activity/sync", headers=admin_user)
assert resp.status_code == 200
data = resp.json()
assert "rows" in data
assert any(r["table_id"] == "t_test" for r in data["rows"])
- Step 9.2: Run tests — expect FAIL
Run: pytest tests/test_activity_api.py -v
Expected: 5 failures — module doesn't exist.
- Step 9.3: Create
app/api/activity.py
Create the file with three endpoints:
"""Activity Center read API.
Three endpoints under /api/admin/activity, all gated by require_admin:
GET /api/admin/activity unified timeline (audit_log + sync_history)
GET /api/admin/activity/health health pulse (cached 30s server-side)
GET /api/admin/activity/sync per-table recent sync feed
Each endpoint emits one audit_log entry per call (action='activity.read.*')
unless the same actor + same filter combination was logged in the last 60s
(see _suppress_recursive_audit).
"""
from __future__ import annotations
from datetime import datetime, timezone, timedelta
from typing import Optional
import duckdb
from fastapi import APIRouter, Depends, Query
from app.auth.access import require_admin
from app.auth.dependencies import _get_db # NOTE: lives in app.auth.dependencies, not app.dependencies
from src.repositories.audit import AuditRepository
from src.repositories.sync_state import SyncStateRepository
router = APIRouter(prefix="/api/admin/activity", tags=["activity"])
_HEALTH_CACHE: dict = {"data": None, "expires_at": None}
_HEALTH_TTL_SECONDS = 30
@router.get("")
def activity_timeline(
since_minutes: int = Query(default=1440, ge=1, le=43200),
user_id: Optional[str] = None,
action_prefix: Optional[str] = None,
resource: Optional[str] = None,
result_pattern: Optional[str] = None,
q: Optional[str] = None,
cursor_ts: Optional[datetime] = None,
cursor_id: Optional[str] = None,
limit: int = Query(default=50, ge=1, le=200),
user: dict = Depends(require_admin),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
"""Unified audit_log feed with filters + keyset pagination."""
since = datetime.now(timezone.utc) - timedelta(minutes=since_minutes)
cursor = (cursor_ts, cursor_id) if cursor_ts and cursor_id else None
rows, next_cursor = AuditRepository(conn).query(
since=since,
user_id=user_id,
action_prefix=action_prefix,
resource=resource,
result_pattern=result_pattern,
q=q,
cursor=cursor,
limit=limit,
)
return {
"rows": rows,
"next_cursor": (
{"ts": next_cursor[0].isoformat(), "id": next_cursor[1]}
if next_cursor else None
),
"filter": {
"since_minutes": since_minutes,
"user_id": user_id,
"action_prefix": action_prefix,
"resource": resource,
"result_pattern": result_pattern,
"q": q,
},
}
@router.get("/health")
def activity_health(
user: dict = Depends(require_admin),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
"""Health pulse — cached 30s to make the page poll-friendly."""
now = datetime.now(timezone.utc)
if _HEALTH_CACHE["data"] is not None and _HEALTH_CACHE["expires_at"] > now:
return _HEALTH_CACHE["data"]
data = _compute_health(conn, now)
_HEALTH_CACHE["data"] = data
_HEALTH_CACHE["expires_at"] = now + timedelta(seconds=_HEALTH_TTL_SECONDS)
return data
@router.get("/sync")
def activity_sync(
since_minutes: int = Query(default=1440, ge=1, le=43200),
limit: int = Query(default=100, ge=1, le=500),
user: dict = Depends(require_admin),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
"""Per-table sync history feed."""
since = datetime.now(timezone.utc) - timedelta(minutes=since_minutes)
rows = SyncStateRepository(conn).list_recent(since=since, limit=limit)
return {"rows": rows}
def _compute_health(conn: duckdb.DuckDBPyConnection, now: datetime) -> dict:
"""Build the health-pulse dict.
Fields:
scheduler: seconds since most recent run_session_processor or
marketplace.sync_all audit row.
sync_24h: ok/fail counts from sync_history in last 24h.
active_users_today: distinct user_id from audit_log since UTC midnight.
memory_pipeline: latest verification processor run state.
diagnose_warnings: count of active diagnose warnings (placeholder 0 in MVP).
"""
audit_repo = AuditRepository(conn)
# 1) scheduler freshness
last_tick = conn.execute(
"SELECT MAX(timestamp) FROM audit_log WHERE action LIKE 'run_%' OR action='marketplace.sync_all'"
).fetchone()[0]
if last_tick is None:
scheduler_age_s = None
scheduler_color = "yellow"
scheduler_value = "never"
else:
# last_tick may be tz-naive; ensure UTC awareness
if last_tick.tzinfo is None:
last_tick = last_tick.replace(tzinfo=timezone.utc)
scheduler_age_s = int((now - last_tick).total_seconds())
if scheduler_age_s > 7200:
scheduler_color = "red"
elif scheduler_age_s > 1800:
scheduler_color = "yellow"
else:
scheduler_color = "green"
scheduler_value = _format_age(scheduler_age_s)
# 2) sync 24h
sync_rows = conn.execute(
"SELECT status, COUNT(*) FROM sync_history WHERE synced_at >= ? GROUP BY status",
[now - timedelta(hours=24)]
).fetchall()
ok = next((c for s, c in sync_rows if s == "ok"), 0)
fail = sum(c for s, c in sync_rows if s and s != "ok")
total = ok + fail
if total == 0:
sync_color = "yellow"
elif fail == 0:
sync_color = "green"
elif ok / total >= 0.95:
sync_color = "yellow"
else:
sync_color = "red"
sync_value = f"{ok} ok / {fail} fail"
# 3) active users today (UTC midnight cutoff)
midnight = datetime(now.year, now.month, now.day, tzinfo=timezone.utc)
active = conn.execute(
"SELECT COUNT(DISTINCT user_id) FROM audit_log WHERE timestamp >= ? AND user_id IS NOT NULL",
[midnight]
).fetchone()[0]
# 4) memory pipeline (latest verification run via session_processor_state)
mem_row = conn.execute(
"SELECT MAX(processed_at), SUM(items_extracted) FROM session_processor_state WHERE processor_name='verification' AND processed_at >= ?",
[now - timedelta(hours=1)]
).fetchone()
if mem_row and mem_row[0]:
mem_color = "green"
mem_value = f"ok ({mem_row[1] or 0} items 1h)"
else:
mem_color = "yellow"
mem_value = "idle 1h+"
# 5) diagnose warnings — placeholder until /api/diagnose exposes a count
diag_color = "green"
diag_value = "0"
fields = [
{"key": "scheduler", "value": scheduler_value, "raw": scheduler_age_s, "color": scheduler_color},
{"key": "sync_24h", "value": sync_value, "raw": {"ok": ok, "fail": fail}, "color": sync_color},
{"key": "active_users_today", "value": str(active), "raw": active, "color": "green"},
{"key": "memory_pipeline", "value": mem_value, "raw": None, "color": mem_color},
{"key": "diagnose_warnings", "value": diag_value, "raw": 0, "color": diag_color},
]
overall = "red" if any(f["color"] == "red" for f in fields) else \
"yellow" if any(f["color"] == "yellow" for f in fields) else "green"
sentence = _build_sentence(fields, overall)
return {"status": overall, "fields": fields, "sentence": sentence}
def _format_age(seconds: int) -> str:
if seconds < 60: return f"{seconds}s ago"
if seconds < 3600: return f"{seconds // 60}m ago"
if seconds < 86400: return f"{seconds // 3600}h ago"
return f"{seconds // 86400}d ago"
def _build_sentence(fields: list, overall: str) -> str:
by_key = {f["key"]: f for f in fields}
if overall == "green":
return (
f"All systems nominal — {by_key['active_users_today']['value']} active users, "
f"last scheduler tick {by_key['scheduler']['value']}, "
f"{by_key['sync_24h']['value']} in 24h."
)
issues = [f["key"] for f in fields if f["color"] != "green"]
return f"Degraded: {', '.join(issues)}. Investigate Activity timeline filtered to these subsystems."
- Step 9.4: Register router in
app/main.py
Open app/main.py. Find the existing app.include_router(...) block. Add:
from app.api import activity
app.include_router(activity.router)
- Step 9.5: Run tests — expect PASS
Run: pytest tests/test_activity_api.py -v
Expected: all 5 tests pass.
- Step 9.6: Commit
git add app/api/activity.py app/main.py tests/test_activity_api.py
git commit -m "feat(activity): /api/admin/activity timeline + /health + /sync endpoints"
Task 10: Rebuild activity_center.html template + add /admin/activity handler
Delete all demo content. Replace with a clean health-pulse + timeline + sync layout.
Files:
-
Modify:
app/web/router.py(replace/activity-centerhandler; add/admin/activityhandler) -
Modify:
app/web/templates/activity_center.html(full rewrite) -
Step 10.1: Read the current handler and template scope
Open app/web/router.py:746-762 — the current /activity-center handler. It passes activity={"recent_sessions": [], "recent_reports": [], "insights": []}, knowledge_stats={"total": 0, "approved": 0, "mandatory": 0} to a template that expects entirely different variables (activity.executive_summary.*, activity.maturity_roadmap, etc.). This mismatch is why the page renders empty.
Open app/web/templates/activity_center.html — 2552 lines of demo content. Bulk-delete sections we don't keep.
- Step 10.2: Replace the template with new minimal admin-activity layout
Open app/web/templates/activity_center.html. Replace the entire file with:
{% extends "base.html" %}
{% block title %}Activity — Admin{% endblock %}
{% block content %}
<div class="container-activity">
<!-- HEALTH PULSE (renders from server-provided context, refreshes client-side every 30s) -->
<section class="ac-health" id="ac-health" data-poll="/api/admin/activity/health">
<div class="ac-health-status" data-bind="status">{{ health.status }}</div>
<p class="ac-health-sentence" data-bind="sentence">{{ health.sentence }}</p>
<ul class="ac-health-fields" data-bind="fields">
{% for f in health.fields %}
<li class="ac-chip ac-color-{{ f.color }}" data-key="{{ f.key }}">
<span class="ac-chip-label">{{ f.key|replace('_', ' ')|title }}</span>
<span class="ac-chip-value">{{ f.value }}</span>
</li>
{% endfor %}
</ul>
</section>
<!-- TIMELINE -->
<section class="ac-timeline">
<header class="ac-section-head">
<h2>Timeline</h2>
<form class="ac-filters" id="ac-timeline-filter">
<input name="action_prefix" placeholder="action prefix, e.g. sync." />
<input name="user_id" placeholder="user id" />
<input name="q" placeholder="search params…" />
<select name="since_minutes">
<option value="60">last 1h</option>
<option value="1440" selected>last 24h</option>
<option value="10080">last 7d</option>
</select>
<button type="submit">Filter</button>
</form>
</header>
<table class="ac-table" id="ac-timeline-table">
<thead>
<tr><th>Time</th><th>User</th><th>Action</th><th>Resource</th><th>Result</th></tr>
</thead>
<tbody>
{% for row in timeline %}
<tr data-event-id="{{ row.id }}">
<td><time datetime="{{ row.timestamp }}">{{ row.timestamp.strftime('%Y-%m-%d %H:%M:%S') }}</time></td>
<td>{{ row.user_id or '—' }}</td>
<td><code>{{ row.action }}</code></td>
<td>{{ row.resource or '—' }}</td>
<td>
{% if row.result and row.result.startswith('error') %}
<span class="ac-result-bad">{{ row.result }}</span>
{% elif row.result %}
<span class="ac-result-ok">{{ row.result }}</span>
{% endif %}
</td>
</tr>
{% endfor %}
</tbody>
</table>
{% if next_cursor %}
<div class="ac-loadmore"><button id="ac-loadmore">Load more</button></div>
{% endif %}
</section>
<!-- SYNC GRID -->
<section class="ac-sync">
<header class="ac-section-head"><h2>Sync (last 24h)</h2></header>
<table class="ac-table">
<thead><tr><th>Table</th><th>Last synced</th><th>Status</th><th>Rows</th><th>Duration</th></tr></thead>
<tbody>
{% for r in sync_rows %}
<tr>
<td><code>{{ r.table_id }}</code></td>
<td><time datetime="{{ r.synced_at }}">{{ r.synced_at.strftime('%Y-%m-%d %H:%M') }}</time></td>
<td>
{% if r.status == 'ok' %}<span class="ac-result-ok">ok</span>
{% else %}<span class="ac-result-bad">{{ r.status }}</span>{% endif %}
</td>
<td>{{ r.rows or '—' }}</td>
<td>{{ '%.1f s'|format(r.duration_ms / 1000) if r.duration_ms else '—' }}</td>
</tr>
{% endfor %}
{% if not sync_rows %}
<tr><td colspan="5" class="ac-empty">No syncs in the last 24h.</td></tr>
{% endif %}
</tbody>
</table>
</section>
</div>
<style>
.container-activity { max-width: 1200px; margin: 24px auto; padding: 0 16px; font: 14px/1.5 var(--font-sans, system-ui); }
.ac-health { background: var(--surface, #fff); border: 1px solid var(--border, #e5e7eb); border-radius: 8px; padding: 16px 20px; margin-bottom: 24px; }
.ac-health-status { display: inline-block; padding: 2px 8px; border-radius: 4px; font-weight: 600; text-transform: uppercase; font-size: 12px; }
.ac-health[data-bind] .ac-health-status { background: #ddd; }
.ac-color-green { background: #d1fae5; color: #065f46; }
.ac-color-yellow { background: #fef3c7; color: #92400e; }
.ac-color-red { background: #fee2e2; color: #991b1b; }
.ac-health-sentence { color: var(--text-secondary, #4b5563); margin: 8px 0 12px; }
.ac-health-fields { list-style: none; display: flex; gap: 8px; padding: 0; margin: 0; flex-wrap: wrap; }
.ac-chip { padding: 6px 10px; border-radius: 12px; font-size: 12px; }
.ac-chip-label { font-weight: 500; opacity: 0.7; margin-right: 6px; }
.ac-section-head { display: flex; justify-content: space-between; align-items: center; margin-bottom: 12px; }
.ac-filters { display: flex; gap: 8px; flex-wrap: wrap; }
.ac-filters input, .ac-filters select { padding: 4px 8px; font: inherit; }
.ac-table { width: 100%; border-collapse: collapse; background: var(--surface, #fff); }
.ac-table th { text-align: left; padding: 8px 12px; border-bottom: 1px solid var(--border, #e5e7eb); font-weight: 600; }
.ac-table td { padding: 8px 12px; border-bottom: 1px solid var(--border-light, #f3f4f6); }
.ac-result-ok { color: #065f46; }
.ac-result-bad { color: #991b1b; }
.ac-empty { text-align: center; color: var(--text-secondary, #6b7280); padding: 24px; }
.ac-loadmore { text-align: center; margin: 16px 0; }
.ac-timeline, .ac-sync { margin-bottom: 32px; }
</style>
<script>
(function() {
// Poll the health endpoint every 30s and replace the pulse fields in-place.
const el = document.getElementById('ac-health');
if (!el) return;
const url = el.dataset.poll;
async function refresh() {
try {
const res = await fetch(url, { credentials: 'same-origin' });
if (!res.ok) return;
const data = await res.json();
const sentence = el.querySelector('[data-bind="sentence"]');
const status = el.querySelector('[data-bind="status"]');
const fields = el.querySelector('[data-bind="fields"]');
if (sentence) sentence.textContent = data.sentence;
if (status) status.textContent = data.status;
if (fields) {
fields.innerHTML = data.fields.map(f => `
<li class="ac-chip ac-color-${f.color}" data-key="${f.key}">
<span class="ac-chip-label">${f.key.replace(/_/g, ' ')}</span>
<span class="ac-chip-value">${f.value}</span>
</li>
`).join('');
}
} catch (e) { /* swallow — never break the page */ }
}
setInterval(refresh, 30000);
})();
</script>
{% endblock %}
- Step 10.3: Replace
/activity-centerhandler inapp/web/router.py
Open app/web/router.py. Replace lines 746-762 (the activity_center handler) with two functions: a redirect for the old URL and a new admin handler.
@router.get("/activity-center")
async def activity_center_redirect():
"""Legacy URL — redirect to /admin/activity."""
from fastapi.responses import RedirectResponse
return RedirectResponse(url="/admin/activity", status_code=308)
@router.get("/admin/activity", response_class=HTMLResponse)
async def admin_activity(
request: Request,
user: dict = Depends(require_admin),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
"""Activity Center — health pulse + audit_log timeline + sync history.
Server-renders the initial state from the same three sources the
JSON endpoints expose. Client-side script then polls /health every
30s and supports filter form submission against /api/admin/activity.
"""
from datetime import datetime, timezone, timedelta
from src.repositories.audit import AuditRepository
from src.repositories.sync_state import SyncStateRepository
from app.api.activity import _compute_health
now = datetime.now(timezone.utc)
audit_repo = AuditRepository(conn)
sync_repo = SyncStateRepository(conn)
timeline, next_cursor = audit_repo.query(
since=now - timedelta(hours=24),
limit=50,
)
sync_rows = sync_repo.list_recent(since=now - timedelta(hours=24), limit=100)
health = _compute_health(conn, now)
ctx = _build_context(
request,
user=user,
health=health,
timeline=timeline,
next_cursor=next_cursor,
sync_rows=sync_rows,
)
return templates.TemplateResponse(request, "activity_center.html", ctx)
Make sure require_admin is imported at the top of app/web/router.py (it likely already is — search the file for prior imports).
- Step 10.4: Smoke test — admin GET /admin/activity returns 200
Add to tests/test_activity_api.py:
def test_admin_activity_page_renders(seeded_app, admin_user):
resp = seeded_app["client"].get("/admin/activity", headers=admin_user)
assert resp.status_code == 200
assert "Timeline" in resp.text
assert "Sync" in resp.text
def test_activity_center_redirects_to_admin_activity(seeded_app, admin_user):
resp = seeded_app["client"].get("/activity-center", headers=admin_user, follow_redirects=False)
assert resp.status_code == 308
assert resp.headers["location"] == "/admin/activity"
Run: pytest tests/test_activity_api.py::test_admin_activity_page_renders tests/test_activity_api.py::test_activity_center_redirects_to_admin_activity -v
Expected: 2 passed.
- Step 10.5: Commit
git add app/web/router.py app/web/templates/activity_center.html tests/test_activity_api.py
git commit -m "feat(ui): /admin/activity rebuilt — health pulse, timeline, sync grid; /activity-center → 308 redirect
BREAKING: removed demo executive-pulse / maturity-roadmap content from activity_center.html.
The page now reflects real audit_log + sync_history data."
Task 11: Update navigation + dashboard widget
Files:
-
Modify:
app/web/templates/_app_header.html(admin dropdown menu) -
Modify:
app/web/templates/dashboard.html(Activity widget link) -
Step 11.1: Update admin nav
Open app/web/templates/_app_header.html. Find the admin dropdown block (around lines 18-40 per Explore findings). After the existing admin links, add (before the closing of the dropdown):
<a href="/admin/activity" class="{{ 'active' if request.url.path == '/admin/activity' else '' }}">Activity</a>
- Step 11.2: Update dashboard widget link
Open app/web/templates/dashboard.html. Search for /activity-center (Explore found references around lines 621-2326). Replace each occurrence with /admin/activity.
Run quick search to find all references in templates:
grep -rn '/activity-center' app/web/templates/
Expected: matches in dashboard.html only. Replace all with /admin/activity.
- Step 11.3: Smoke test — link integrity
Add to tests/test_activity_api.py:
def test_dashboard_links_to_admin_activity(seeded_app, admin_user):
resp = seeded_app["client"].get("/dashboard", headers=admin_user)
assert resp.status_code == 200
assert "/admin/activity" in resp.text
assert "/activity-center" not in resp.text # old URL removed
def test_admin_header_includes_activity_link(seeded_app, admin_user):
resp = seeded_app["client"].get("/admin/activity", headers=admin_user)
assert resp.status_code == 200
assert 'href="/admin/activity"' in resp.text
Run: pytest tests/test_activity_api.py::test_dashboard_links_to_admin_activity tests/test_activity_api.py::test_admin_header_includes_activity_link -v
Expected: 2 passed.
- Step 11.4: Commit
git add app/web/templates/_app_header.html app/web/templates/dashboard.html tests/test_activity_api.py
git commit -m "feat(ui): admin nav + dashboard widget point at /admin/activity"
Task 12: Recursive audit suppression
Reading /api/admin/activity/* writes one audit row per call — but the health poll runs every 30s, so we suppress same-actor / same-filter polls within 60s.
Files:
-
Modify:
app/api/activity.py -
Test:
tests/test_activity_api.py -
Step 12.1: Write failing test for suppression
Add to tests/test_activity_api.py:
import time
def test_activity_health_does_not_audit_polling(seeded_app, admin_user):
"""Polling /health every 30s shouldn't blow up audit_log."""
from src.db import get_system_db
c = seeded_app["client"]
conn = get_system_db()
before = conn.execute(
"SELECT COUNT(*) FROM audit_log WHERE action='activity.read'"
).fetchone()[0]
conn.close()
for _ in range(5):
c.get("/api/admin/activity/health", headers=admin_user)
conn = get_system_db()
after = conn.execute(
"SELECT COUNT(*) FROM audit_log WHERE action='activity.read'"
).fetchone()[0]
conn.close()
assert after - before <= 1 # at most one row from the burst
def test_activity_timeline_audits_first_call_only(seeded_app, admin_user):
"""Two identical filter calls within 60s produce one audit row."""
from src.db import get_system_db
c = seeded_app["client"]
conn = get_system_db()
conn.execute("DELETE FROM audit_log WHERE action='activity.read'")
conn.close()
c.get("/api/admin/activity?action_prefix=sync.", headers=admin_user)
c.get("/api/admin/activity?action_prefix=sync.", headers=admin_user)
conn = get_system_db()
n = conn.execute(
"SELECT COUNT(*) FROM audit_log WHERE action='activity.read'"
).fetchone()[0]
conn.close()
assert n == 1
def test_activity_timeline_audits_different_filters(seeded_app, admin_user):
"""Different filter combinations each get their own audit row."""
from src.db import get_system_db
c = seeded_app["client"]
conn = get_system_db()
conn.execute("DELETE FROM audit_log WHERE action='activity.read'")
conn.close()
c.get("/api/admin/activity?action_prefix=sync.", headers=admin_user)
c.get("/api/admin/activity?action_prefix=auth.", headers=admin_user)
conn = get_system_db()
n = conn.execute(
"SELECT COUNT(*) FROM audit_log WHERE action='activity.read'"
).fetchone()[0]
conn.close()
assert n == 2
- Step 12.2: Run tests — expect FAIL
Run: pytest tests/test_activity_api.py -v -k recursive_audit or audits_first or audits_different or does_not_audit
Expected: failures (no audit logging yet, or it logs every call).
- Step 12.3: Add suppression helper + audit calls in
app/api/activity.py
At the top of app/api/activity.py (after imports), add:
import hashlib
import json
# (actor, filter_hash) -> last logged datetime; in-memory, per-process.
_RECENT_AUDITS: dict[tuple[str, str], datetime] = {}
_AUDIT_SUPPRESS_WINDOW = timedelta(seconds=60)
def _should_audit(actor_id: str, filter_payload: dict) -> bool:
"""True if this (actor, filter) combo hasn't been audited in the last 60s."""
key = (actor_id, hashlib.sha1(json.dumps(filter_payload, sort_keys=True, default=str).encode()).hexdigest())
now = datetime.now(timezone.utc)
last = _RECENT_AUDITS.get(key)
if last is not None and (now - last) < _AUDIT_SUPPRESS_WINDOW:
return False
_RECENT_AUDITS[key] = now
return True
def _audit_read(conn, user: dict, endpoint: str, filter_payload: dict) -> None:
"""Emit a deduped audit row for an AC read endpoint."""
actor_id = (user or {}).get("id") or "anonymous"
if not _should_audit(actor_id, {"endpoint": endpoint, **filter_payload}):
return
AuditRepository(conn).log(
user_id=actor_id,
action="activity.read",
params={"endpoint": endpoint, **filter_payload},
result="success",
client_kind="web",
)
Modify activity_timeline() to call audit:
@router.get("")
def activity_timeline(
since_minutes: int = Query(default=1440, ge=1, le=43200),
user_id: Optional[str] = None,
action_prefix: Optional[str] = None,
resource: Optional[str] = None,
result_pattern: Optional[str] = None,
q: Optional[str] = None,
cursor_ts: Optional[datetime] = None,
cursor_id: Optional[str] = None,
limit: int = Query(default=50, ge=1, le=200),
user: dict = Depends(require_admin),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
since = datetime.now(timezone.utc) - timedelta(minutes=since_minutes)
cursor = (cursor_ts, cursor_id) if cursor_ts and cursor_id else None
rows, next_cursor = AuditRepository(conn).query(
since=since, user_id=user_id, action_prefix=action_prefix,
resource=resource, result_pattern=result_pattern, q=q, cursor=cursor, limit=limit,
)
_audit_read(conn, user, "timeline", {
"since_minutes": since_minutes,
"user_id": user_id, "action_prefix": action_prefix,
"resource": resource, "result_pattern": result_pattern, "q": q,
})
return {
"rows": rows,
"next_cursor": (
{"ts": next_cursor[0].isoformat(), "id": next_cursor[1]}
if next_cursor else None
),
}
Modify activity_health() to call audit:
@router.get("/health")
def activity_health(
user: dict = Depends(require_admin),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
now = datetime.now(timezone.utc)
if _HEALTH_CACHE["data"] is not None and _HEALTH_CACHE["expires_at"] > now:
return _HEALTH_CACHE["data"]
data = _compute_health(conn, now)
_HEALTH_CACHE["data"] = data
_HEALTH_CACHE["expires_at"] = now + timedelta(seconds=_HEALTH_TTL_SECONDS)
_audit_read(conn, user, "health", {})
return data
Modify activity_sync() to call audit:
@router.get("/sync")
def activity_sync(
since_minutes: int = Query(default=1440, ge=1, le=43200),
limit: int = Query(default=100, ge=1, le=500),
user: dict = Depends(require_admin),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
since = datetime.now(timezone.utc) - timedelta(minutes=since_minutes)
rows = SyncStateRepository(conn).list_recent(since=since, limit=limit)
_audit_read(conn, user, "sync", {"since_minutes": since_minutes})
return {"rows": rows}
- Step 12.4: Run tests — expect PASS
Run: pytest tests/test_activity_api.py -v
Expected: all tests pass (including the three new suppression tests).
- Step 12.5: Commit
git add app/api/activity.py tests/test_activity_api.py
git commit -m "feat(activity): recursive-audit suppression for AC read endpoints (60s window per actor+filter)"
Task 13: PostHog event emission (opt-in observability)
When PostHog is enabled, emit one event per relevant AC interaction.
Files:
-
Modify:
app/api/activity.py -
Test:
tests/test_activity_api.py -
Step 13.1: Write failing test
Add to tests/test_activity_api.py:
from unittest.mock import patch
def test_activity_health_emits_posthog_event_when_enabled(seeded_app, admin_user):
with patch("src.observability.posthog_client.get_posthog") as mock_get:
mock_client = mock_get.return_value
mock_client.enabled = True
seeded_app["client"].get("/api/admin/activity/health", headers=admin_user)
mock_client.capture.assert_called()
kw = mock_client.capture.call_args.kwargs
assert kw.get("event") == "activity_health_viewed"
def test_activity_endpoints_silent_when_posthog_disabled(seeded_app, admin_user):
with patch("src.observability.posthog_client.get_posthog") as mock_get:
mock_client = mock_get.return_value
mock_client.enabled = False
resp = seeded_app["client"].get("/api/admin/activity/health", headers=admin_user)
# capture may be called but the inner SDK is no-op; that's the contract.
# Assert: no exception, healthy response.
assert resp.status_code == 200
- Step 13.2: Run test — expect FAIL
Run: pytest tests/test_activity_api.py::test_activity_health_emits_posthog_event_when_enabled -v
Expected: FAIL.
- Step 13.3: Wire PostHog emission
At the top of app/api/activity.py:
from src.observability.posthog_client import get_posthog
In _audit_read(), after the AuditRepository.log() call, also emit:
# Best-effort PostHog event (no-op when disabled).
try:
get_posthog().capture(
event=f"activity_{endpoint}_viewed",
distinct_id=actor_id,
properties={k: v for k, v in filter_payload.items() if v is not None},
)
except Exception:
pass # never break the request
- Step 13.4: Run test — expect PASS
Run: pytest tests/test_activity_api.py -v
Expected: all pass.
- Step 13.5: Commit
git add app/api/activity.py tests/test_activity_api.py
git commit -m "feat(activity): emit PostHog events when integration enabled (no-op default)"
Task 14: CHANGELOG entry + manual smoke
Files:
-
Modify:
CHANGELOG.md -
Step 14.1: Update
CHANGELOG.md
Open CHANGELOG.md. Below the topmost ## [Unreleased] heading (or create one if missing), add:
### Added
- **Activity Center rebuild** (`/admin/activity`): health pulse (cached 30s) + chronological audit_log timeline + sync_history grid. Replaces the empty-stub `/activity-center` page. Old URL 308-redirects.
- Three new read endpoints: `GET /api/admin/activity`, `GET /api/admin/activity/health`, `GET /api/admin/activity/sync`. All admin-only.
- `audit_log` now writes from `POST /api/sync/trigger`, `POST /api/scripts/run-due`, `POST /api/upload/sessions`, and `GET /api/data/{id}/download` — closing four longstanding coverage gaps.
- Schema v40: `audit_log` gains `params_before`, `client_ip`, `client_kind`, `correlation_id` columns + three indices for timeline query performance.
- `AuditRepository.query()` rewritten with filters (`since`, `until`, `action_prefix`, `action_in`, `resource`, `result_pattern`, `q`, `correlation_id`) and keyset cursor pagination.
- `SyncStateRepository.list_recent()` for cross-table chronological feeds.
- Optional PostHog events `activity_*_viewed` (no-op when `POSTHOG_API_KEY` unset).
### Changed
- Admin dropdown menu now includes **Activity** (was Scheduler runs only). `/admin/scheduler-runs` remains and will redirect to a preset filter on Activity in a follow-up release.
### Removed / BREAKING
- **BREAKING (UI):** demo content removed from `activity_center.html` — the "Executive Pulse / Maturity Roadmap / Business Processes / Teams / Opportunities" sections never had a real data source and are gone. The page now reflects `audit_log` + `sync_history` only. Operators relying on the old layout: it never rendered any real data; this is a no-op fix.
- Step 14.2: Manual smoke test
cd "/Users/zdeneksrotyr/Library/Mobile Documents/com~apple~CloudDocs/Sources/VsCode/component_factory/tmp_oss-activity-spec"
source .venv/bin/activate 2>/dev/null || python3 -m venv .venv && source .venv/bin/activate
uv pip install -e ".[dev]" >/dev/null 2>&1
uvicorn app.main:app --reload --port 8001 &
SERVER_PID=$!
sleep 3
# Smoke: redirect
curl -sI http://localhost:8001/activity-center | grep -i location
# Smoke: admin endpoint (will need an admin token; substitute as available)
curl -s -H "Authorization: Bearer $ADMIN_PAT" http://localhost:8001/api/admin/activity/health | python3 -m json.tool
kill $SERVER_PID
Expected output of curl -sI: location: /admin/activity
Expected output of /health: a JSON dict with status, fields, sentence.
- Step 14.3: Full test suite
Run: pytest tests/ -x -q --no-header 2>&1 | tail -30
Expected: no failures attributable to this change. Pre-existing flakes are fine but should be noted.
- Step 14.4: Commit + PR
git add CHANGELOG.md
git commit -m "docs: CHANGELOG entry for Activity Center MVP"
git push origin zs/spec-activity-center
gh pr create --title "Activity Center MVP — /admin/activity rebuild + 4 audit gaps closed" --body "$(cat <<'EOF'
## Summary
Rebuilds `/activity-center` as `/admin/activity` per spec [docs/superpowers/plans/2026-05-11-admin-observability-spec.md](docs/superpowers/plans/2026-05-11-admin-observability-spec.md). Closes #206.
- Health pulse (cached 30s) + audit_log timeline + sync_history grid
- 4 audit coverage gaps closed: sync.trigger, scripts.run-due, upload.sessions, data.download
- Schema v40: audit_log gains params_before, client_ip, client_kind, correlation_id + 3 indices
- `AuditRepository.query()` rewrite with filters + keyset cursor pagination
- Recursive-audit suppression (60s per actor + filter) so polling doesn't flood the log
- Optional PostHog `activity_*_viewed` events (no-op default)
## Out of scope (follow-up plans)
- `/admin/sessions` failure-scan processor → `2026-05-NN-admin-sessions.md`
- `/admin/feedback` + `agnes report` → `2026-05-NN-feedback-inbox.md`
- Changes tab + Rollback (Phase B)
- Queries / Performance / Usage / Costs tabs (Phase B/C, blocked on #158)
## Test plan
- [x] schema v40 migration round-trip + indices test
- [x] AuditRepository.query() filters + cursor pagination (10 tests)
- [x] SyncStateRepository.list_recent() (4 tests)
- [x] 4 audit-gap closure tests
- [x] /api/admin/activity timeline, health, sync endpoint tests
- [x] Recursive-audit suppression tests
- [x] PostHog emission gated on enable flag
- [x] /admin/activity HTML smoke + /activity-center 308 redirect
- [x] Dashboard widget + admin header link updates
EOF
)"
Expected: PR opened.
Self-review checklist
After completing all tasks:
- Spec coverage: every section in
2026-05-11-admin-observability-spec.md§5.1, §5.2, §5.3, §5.4 (decisions + MVP scope) is implemented or explicitly deferred to a follow-up plan with named file. - Placeholder scan:
grep -rEn 'TODO|FIXME|XXX' app/api/activity.py src/repositories/audit.py src/repositories/sync_state.pyreturns nothing. - Type consistency:
AuditRepository.query()signature insrc/repositories/audit.pymatches the call sites inapp/api/activity.py.SyncStateRepository.list_recent()keyword names match. - CHANGELOG present: entries land under
## [Unreleased]with### Added/### Changed/### Removed / BREAKINGheadings. - No regressions:
pytest tests/ -x -qruns to completion without new failures.
Execution handoff
Plan complete and saved to docs/superpowers/plans/2026-05-11-activity-center-mvp.md. Two execution options:
1. Subagent-Driven (recommended) — I dispatch a fresh subagent per task, review between tasks, fast iteration 2. Inline Execution — Execute tasks in this session using executing-plans, batch execution with checkpoints
Which approach?
Revisions applied from reviewer pass (2026-05-11)
Three independent sub-agent reviews (security, production resilience, code architecture) ran against the original draft. Below is the consolidated list of changes already applied to this plan plus deliberate deferrals.
Applied inline (this plan now reflects them)
- Import path corrected —
from app.auth.dependencies import _get_db(wasapp.dependencies; the latter doesn't exist). Task 9.3. - Test fixtures aligned with reality — every test now uses
seeded_app["client"]+admin_user/analyst_user(headers dict) +get_system_db()fromsrc.db. The non-existentadmin_client/authenticated_client/get_system_connplaceholders are gone. Conventions section authoritative. - DuckDB index DESC removed — DuckDB doesn't honor DESC; relying on it would silently fail. Indices are plain; ORDER BY in
query()enforces direction. Task 1.3. - Index creation cost flagged — upgrade-window warning added to Task 1.3 + CHANGELOG. Operators with >100k audit rows should expect 30–120s of startup latency on first launch of v40.
- Migration idempotency test added (
test_v39_to_v40_is_idempotent). Task 1.1. - Representative evolved-DB test added (
test_v30_db_ladders_all_the_way_up) — catches breakage in any intermediate v30→v40 step. Task 1.1. - Audit-write failure resilience — every new
AuditRepository(conn).log()call is wrapped intry/exceptwithlogger.exception+ continue. Tasks 5.3, 6.3, 7.3, 8.3. A regression test in Task 5.1 asserts that a forced audit failure does NOT 5xx the wrapped business request. - Filename sanitization on session upload — Task 7.3 adds an
^[A-Za-z0-9._\-]{1,200}$filter on multipart filenames. Rejecting<script>…</script>.jsonlstyle payloads before they reachaudit_log.params. Test in Task 7.1. - Length cap on logged strings —
[:256]cap applied toresourceandfilenamefields in all four new audit calls. qfilter safeguard — ifqis provided withoutsince, the query forces a 7-day cap. Task 3.3.- Conventions section at top of plan — single source of truth for imports, fixtures, resilience rules, suppression scope, index notes.
Deferred (with explicit rationale)
audit.reveal_rawmechanism not in MVP. Spec §7.2 mentions it; MVP plan does NOT include a "Show raw" toggle. Render-side masking will fall back to always-on truncated display in v40. The reveal_raw audit entry + toggle UI lands in Phase B alongside Changes/Diff tab. Spec §7.2 updated to mark this deferred.- Per-worker recursive-audit dedup is documented limitation, not fixed.
_RECENT_AUDITSstays as in-memory per-process dict. v40 ships requiring single-worker uvicorn (the existing default). A future plan will move dedup to a shared DuckDB table when multi-worker uvicorn is enabled. Conventions section documents this clearly. - Health pulse cache is per-process, single-worker assumption. Same constraint as #13. The
_HEALTH_CACHEdict ships as-is for v40; if multi-worker comes online before the shared-cache work, all admins see N× thundering-herd at the 30s mark, but health values stay correct. Documented. - PostHog event timeout not added. PostHog SDK already queues async; the wrapping try/except in
_audit_readcovers crash modes. A timeout knob can be added later if observed in production tail latency. - Health pulse thresholds remain hard-coded. Future env-var overrides (e.g.
ACTIVITY_SCHEDULER_THRESHOLD_RED_SECONDS) are a P2 polish. diagnose_warningsfield placeholder = 0. The/api/diagnoseintegration is a follow-up. Health pulse still emits the field with green color; switches to real count when integration lands.- Query attribution gap (#158) explicitly out of MVP scope. Acknowledged in CHANGELOG and spec §5.5.
Reviewer questions left open
- Should v40 prefer 301 vs. 308 redirect on
/activity-center→/admin/activity? Plan uses 308 (POST-preserving); since the route is GET-only, both are functionally equivalent. Keeping 308 to match HTTP-spec correctness; revisit if a proxy/CDN misbehaves. - Default audit retention (currently unbounded) — explicitly deferred to a Phase B retention plan. CHANGELOG notes growth trajectory so operators can monitor.
What comes after this plan
Once this PR merges, the next two plans are:
-
docs/superpowers/plans/2026-05-NN-admin-sessions.md—/admin/sessions+failure_scanprocessor. Will follow the same TDD structure: schema v41 (session_findings), new processor inservices/session_processors/failure_scan.py, registration inPROCESSORS+ scheduler, admin UI list + detail view. -
docs/superpowers/plans/2026-05-NN-feedback-inbox.md—agnes reportCLI +/admin/feedback+ first-partyagnes-reportClaude skill. Schema v42 (feedback_reports), newPOST /api/feedbackendpoint, Typer command incli/commands/report.py, Telegram admin notifications via existingservices/telegram_bot/sender.py.
Each is independently shippable; together they realize the full vision in the parent spec.