* feat(unified-stack): Browse + My Stack + Recipes + RBAC matrix (v49–v55)
Squash of 94 commits spanning the v49 → v55 unified-stack rewrite.
Full per-feature breakdown lives in CHANGELOG.md under [Unreleased].
Major buckets:
* v49 schema — first-class user_groups + user_group_members +
resource_grants; admin can CRUD groups and grants; Google
Workspace nightly sync writes into the new tables.
* v49 data_packages — admin-curated bundles of tables, RBAC-gated,
first-class section on /catalog Browse + My Stack.
* v49 memory_domains — row-backed (replaces hardcoded VALID_DOMAINS
enum); admin can CRUD; grants follow the same shape as tables and
packages.
* v50 cover_image_url + admin sidebar collapsibles + per-row Mode
tooltip + admin queue domain badges + admin "+ New Item" seed flow.
* v51 lifecycle status (prod/poc/coming-soon/draft) + category +
palette swatches on admin modals.
* v52 per-table detail page /catalog/t/<id>.
* v53 Recipes — admin-curated SQL templates as a second tab on
/catalog with full Edit/Delete admin affordances.
* v54 soft-delete (deleted_at) + Undo toast for packages, memory
domains, and recipes; hard_delete() retained as escape hatch.
* v55 Recipes RBAC — ResourceType.RECIPE registered, inline Group
Access matrix on Create + Edit Recipe modals (mirrors the Memory
Domain pattern).
* Activity Center per-resource filter (resource_prefix LIKE-anchored
on audit_log.resource); admin nav g+letter keyboard shortcuts;
loadAdminTablesLayout N+1 → single endpoint; /api/memory 30s
page-level cache.
* CI hardening — Keboola legacy tests pytest.importorskip; perf-
smoke threshold widened to stop cold-cache flake.
5002 tests passing, 35 skipped.
* feat(p2 backlog): Cmd-K palette + suggest-a-domain + nightly E2E + v55 schema
10-item P2 sweep on top of the unified-stack squash. New behaviour:
* Cmd-K admin command palette (base.html) — fuzzy-search overlay over
admin + user-facing routes. Arrows/Enter to navigate, Esc to close.
* Stack-tabs digit shortcuts — 1/2/3 switch Browse / My Stack /
Recipes on /catalog + /corporate-memory.
* Friendlier non-admin empty state on /corporate-memory, plus a
"Suggest a domain" CTA → POST /api/memory-domain-suggestions, admin
queue with approve/reject. Backed by a new memory_domain_suggestions
table (schema v55).
* /admin/corporate-memory 7-tab strip grouped under Moderation /
Catalog parent labels.
* Bulk-assign table → package dropdown annotates each option with
"(N of M tables already in)" so the existing distribution is visible
before picking a target.
* GET /api/memory + /tree accept is_required filter; admin status
dropdowns route the "Required" sentinel onto it (status no longer
holds 'mandatory' post-v49, so the old dropdown returned nothing).
* chip-input.js is now opt-in per template via {% block extra_scripts %}
instead of loaded globally on every page from base.html.
* Edit-modal close helpers consolidated onto _closeEditModalById();
docs the per-source-type modal architecture decision.
* New .github/workflows/e2e-nightly.yml runs agent-browser smoke
scripts (scripts/e2e/smoke_*.sh) against a docker-compose stack
nightly at 04:30 UTC; failures open an agent-browser-nightly issue.
5012 tests passing, 35 skipped.
* fix(visual audit): 6 page regressions on memory + data-package surfaces
agent-browser walkthrough of every memory + data-package page in the PR
turned up 6 real bugs. Fixes:
1. Admin memory modals were dead. Duplicate `let _cmdNewDomainId`
declarations from the deprecated step-2 RBAC stubs in
admin_corporate_memory.html collided with the live state vars
declared earlier in the same <script> → SyntaxError on parse →
the entire second script block silently failed → every inline
onclick= handler defined there (`+ New Memory Domain`, Edit, etc.)
was a no-op. Removed the duplicate stubs.
2. /catalog/t/<table_id> + /catalog/r/<slug> rendered unstyled.
Both templates injected their CSS via {% block head %} but
base.html exposes {% block head_extra %} — wrong block name
meant <style> rules never reached the rendered HTML. Renamed
to head_extra. Hero card, section cards, dark SQL block, proper
full-width inputs all now render as designed.
3. L49 leak — "MANDATORY" KPI label + "Make Mandatory" row buttons
on /admin/corporate-memory still used the old word. Renamed to
"Required" / "Mark as Required" so UI matches the data model
(v49 split moved the Required tier onto the orthogonal
is_required boolean; status no longer holds 'mandatory').
4. Activity Center Resource dropdown didn't know the v55
`memory_domain_suggestion:` namespace — added it.
5. Tab strip on /admin/corporate-memory wrapped text 2× per button
on narrow viewports after the L50 MODERATION/CATALOG group
labels pushed total width past most viewports. Switched the
strip to flex-wrap:nowrap + overflow-x:auto with
white-space:nowrap + flex-shrink:0 on every direct child so the
tabs stay one row and slide horizontally when they overflow.
5012 tests passing, 35 skipped.
* rebase-cleanup: align with main's 0.54.25-27 API design + comment fix
Three follow-on fixes after rebasing onto origin/main (0.54.27):
* admin_tables.html: dropped a stray nested ``{% if data_source_type
== 'keboola' %}`` around ``prefillFromKeboolaTable`` (main never had
it; the outer Phase F2 guard already covers it) and reworded a JS
comment that contained literal ``{% %}`` tokens which Jinja was
parsing as a real tag → unbalanced if/endif → 30 template render
failures across the suite.
* /api/stack/subscription/{type}/{id}: DELETE now returns 204 instead
of 200 per the 0.54.26 design rules. CLI client + parity tests
updated to accept 2xx / assert 204.
* Memory-domain suggestion approve/reject paths added to
``_VERB_PATH_ALLOWLIST`` — they are pending → approved/rejected
state-machine transitions (approve also creates the real
memory_domains row as a side effect), so the RPC shape is
intentional rather than a missed PATCH refactor.
5035 tests passing, 35 skipped.
* fix(catalog_table_detail): real polish pass — hero glyph, dedup pills, rows/size meta, scoped sync CTA
The previous fix only got the block-name typo so the existing CSS rendered.
The actual layout was still wireframe-tier on close inspection:
* No cover glyph in the hero (a flat white card with title + meta line);
data-package + memory-domain detail pages both have a colored icon
square. Restored parity — table.icon emoji if set, otherwise initials
on a colored square using table.color.
* "INTERNAL" pill rendered twice for agnes_audit etc. — the mode pill
and the source-type pill happened to be identical strings. Now skip
the source pill when it matches the mode (`internal == internal`).
* Bucket / source_table code chip showed `Agnes Internal.audit_log` for
internal rows — meaningless to a user. Hidden when source_type is
internal.
* `pairs_well_with` admin input was a comma-separated `<input>` always
visible. Wrapped all 4 sections in an Edit-on-demand toggle: read-
only display by default, "+ Add" / "Edit" button on the right edge
of each section header reveals the inline form, Cancel hides it.
* "Trigger sync now" was a cramped link squashed into the empty-state
flex row (visible as `Tr…` overflow before). Promoted to a proper
btn-primary button under the empty-state copy. Hidden entirely for
internal tables (which are server-managed — no upstream to pull).
* Hero meta now surfaces row count + payload size (when sync_state has
them) + last sync timestamp on a single line — was missing from the
original.
* Mode pills colored by tier (local=green, remote=amber, materialized=
blue, internal=gray) so the basic fact about a table reads at a
glance, not from upper-cased ALL-CAPS text alone.
* tests(v56): TDD baseline for extended data-packages content + per-table docs
68 failing tests across 8 files spec the v56 surface before any
implementation lands:
* test_schema_v55_to_v56_migration.py — schema bump, additive ALTERs
on data_packages + table_registry, idempotency, sequential-upgrade
preservation
* test_data_packages_repo_v56.py — repo create/update/get/list for
owner_name, owner_team, tags, long_description, when_to_use,
when_not_to_use, example_questions (JSON list round-trip, empty
defaults, partial-update preservation)
* test_table_registry_v56_docs.py — update_docs for grain, platforms,
partition_col, history, gotchas; preserves v52 docs columns
* test_api_data_packages_v56.py — PUT/POST/GET for all new fields,
field-level validation (tag count, bullet length, description size),
virtual badge derivation (curated/new)
* test_api_registry_docs_v56.py — PATCH /api/admin/registry/{id}/docs
for v56 fields, validation, RBAC unchanged
* test_web_catalog_package_detail_v56.py — /catalog/p/<slug> rewrite
asserts on rendered owner line, tag pills, badges, What it is,
Use it when, Skip it when, Example questions, per-table extended
detail in collapsible row, key-gotcha distinctness, admin-only Edit
* test_web_stack_card_v56_metadata.py — Browse-grid card additions
(owner chip, tag chips, badges) without breaking back-compat for
rows missing the new fields
* test_data_packages_no_vendor_content.py — CI guard: scans app/ +
src/ + cli/ + config/ + scripts/ for Groupon-specific tokens from
the colleague's spec MD; fails if any leak into OSS surfaces
* test_db_schema_version.py — bumped 55 → 56 with rationale
Plus updates schema-version assertion to 56. Implementation lands in
subsequent commits (schema migration → repo → API → templates).
* feat(v56): schema + repo for extended data-packages content
Schema additions (ALTER ADD COLUMN IF NOT EXISTS — additive + idempotent):
* data_packages: owner_name, owner_team, tags, long_description,
when_to_use, when_not_to_use, example_questions (JSON-as-VARCHAR for
the lists)
* table_registry: grain, platforms, partition_col, history, gotchas
(extends the v52 sample_questions / things_to_know / pairs_well_with
docs surface with structured per-table content)
Repo extensions:
* DataPackagesRepository.create + update accept the new fields with
the same Optional-is-no-op contract as v51 (pass an empty list to
clear a JSON column)
* _decode_row decodes the new JSON-list columns to Python lists; NULL
rounds back to [] so callers don't branch
* TableRegistryRepository.update_docs grew the v56 fields alongside
the existing v52 ones — single PATCH can write either tier
atomically
* TableRegistryRepository._decode_row picks up platforms + gotchas in
the same NULL-tolerant decoder
22 repo + migration tests passing. API + UI land in subsequent commits.
* feat(v56): API surface for extended data-packages + per-table docs
CreateDataPackageRequest + UpdateDataPackageRequest grew the v56 fields
(owner_name, owner_team, tags, long_description, when_to_use,
when_not_to_use, example_questions) with per-field validators that
match the Foundry spec checklist:
* tags: ≤8 entries × ≤30 chars
* long_description: ≤4000 chars
* use/skip: ≤8 bullets × ≤200 chars
* example_questions: ≤12 × ≤200 chars
_serialize emits all v56 fields plus a virtual ``badges`` list derived
server-side at render time (no DB column needed): "curated" when the
creator is in the Admin group, "new" within 30 days of created_at.
Backdating created_at or admin-status changes pick up automatically.
PATCH /api/admin/registry/{id}/docs extended with v56 structured
per-table fields (grain, platforms, partition_col, history, gotchas).
gotchas: list of {key: bool, body: str} Pydantic models with the same
≤8 cap; first key=true entry becomes the Key gotcha on the rendered
package detail page. PATCH echoes the fresh state so callers can
re-render without a second GET.
26 API tests passing (16 data-packages + 10 registry-docs).
* feat(v56): /catalog/p/<slug> rewrite + Browse-grid card augmentation
The third (and final) v56 commit lights up the UI surfaces backed by
the schema + API commits earlier in this PR:
* /catalog/p/<slug> template rebuilt around the Foundry spec's
section ladder — hero (icon + name + badges + owner + tags +
description + meta + Add-to-stack), "What it is" markdown body,
paired "Use it when / Skip it when" panels, "Tables in this
package" with collapsible per-table extended detail (grain /
platforms / partition_col / history / gotchas + sample questions),
and an "Example questions you can ask Claude" prompt panel. Each
section guarded by ``{% if pkg.<field> %}`` — empty content fields
hide the section entirely (no "No X yet" placeholder noise on the
public-facing drilldown).
* router catalog_package_detail hydrates per-table v56 fields onto
the tables list + derives the virtual badges (curated / new)
server-side from creator-in-Admin + 30-day created_at.
* StackResolver.ResourceEntry grew owner_name / owner_team / tags /
badges; _fetch_entries pulls the v56 columns + computes badges
once per fetch using a single Admin-group SELECT.
* _data_package_entry_dict adapter passes the new fields through to
the macro; tags are merged source-type pills + admin-authored
category tags per the spec convention.
* _stack_card.html renders the v56 badges (top-left, data-badge=
hooks) + the owner chip (data-card-owner hook) without breaking
back-compat — pre-v56 rows render unchanged.
* Admin PUT handler strips the v56 docs fields from the
read-modify-write merged dict so register() doesn't blow up
with the now-larger row shape (same pattern as the v52 docs
fields stripping).
5115 tests passing (+98 v56 + 18 fixed regressions from the merged-
register PUT path), 35 skipped.
* fix(rbac): Edit-on-package + Group-access 'required' persistence + CI vendor guard
Three related bugs reported on the merged-with-main branch:
1. Clicking Edit on a Data Package card landed on /admin/tables with
a `#<pkg.id>` hash that nothing listened to — admin saw the global
table listing, not the editor for that specific package. Added a
`?edit_package=<pkg_id>` query-param handler in admin_tables.html
(analog to the existing `?edit=<table_id>` and `?assign_to=<pkg_id>`
patterns) that calls openEditDataPackageModal on DOMContentLoaded
after a 250ms layout settle. Updated the package-detail Edit link
to use the new query param.
2. Setting Group Access to 'required' didn't persist — re-opening
the modal showed 'available'. Root cause was the v49
``resource_grants.requirement`` enum existing in the DB but the
POST /api/admin/grants endpoint not surfacing it: ``CreateGrantRequest``
declared only group_id + resource_type + resource_id, so Pydantic
silently dropped the matrix's ``requirement: 'required'`` payload
and the new row landed at the DB column default ('available').
Plumbed ``requirement`` through ``CreateGrantRequest`` →
``ResourceGrantsRepository.create`` so the value persists in one
round-trip. Plus a UNIQUE-constraint race in the matrix
diff-apply: DELETE-old + POST-new ran in parallel via
``Promise.allSettled``, so POST could fire first and trip the
unique check before DELETE freed the slot. Switched to sequential
(await all deletes; then await all writes) across all three
matrices (Edit Data Package, Edit Memory Domain, Edit Recipe).
3. CI vendor-content guard ``test_no_groupon_specific_strings_in_oss``
tripped on two of my own docstrings: a "Foundry Data team" mention
in two src/db.py comments + an ``s1_session_landings`` example in
cli/skills/agnes-table-registration.md. Rephrased the comments to
"extended-descriptions admin spec" and replaced the example with
a generic ``events_daily`` table name.
5164 tests passing, 35 skipped (+4 regression tests pinning the POST
/api/admin/grants requirement contract). Vendor guard back to green.
* fix(catalog): admin Browse path drops v58 card fields
The /catalog and /memory admin god-mode branch built ResourceEntry
instances inline from pkg_repo.list() / domains_repo.list() and skipped
owner_name, owner_team, tags, and derived badges (curated/new). Visible
symptom: a package with an owner + tags rendered with the v56 chrome
for non-admin viewers but as a bare card for admins.
Adds StackResolver.browse_admin(user_id, resource_type) — admin god-mode
Browse that walks the full table but routes through the same
_fetch_entries enrichment pass as browse(), so admin + non-admin Browse
stay visually consistent. Both /catalog and /corporate-memory routes
switch to it.
Regression test in tests/test_stack_resolver_browse_admin.py covers:
owner/tags propagation, new/curated badge derivation, in_stack from
admin subscriptions, all-packages-regardless-of-grants, and the
ValueError for unsupported resource types.
* fix(catalog): three /catalog tab-strip UX bugs
1. Required Remove → red toast
browse_admin passed empty required_ids to _fetch_entries, so the
admin's own required grants surfaced as 'available' and the macro
rendered an actionable Remove button that POST /unsubscribe 400'd
on. Now derives required_ids from the admin's own groups so
Required packages render with the disabled "In stack (required)"
button. Regression test in test_stack_resolver_browse_admin.py.
2. Remove green-toasts but card stays until refresh
The My-Stack empty-state placeholder was only emitted server-side
when stack_entries was empty at render time. Removing the last
card left the tab completely blank — users read that as "Remove
didn't work, let me refresh". Both grid + empty-state are now
always rendered with one of them initially hidden; the JS swaps
visibility on add/remove instead of injecting DOM. Same fix in
/corporate-memory.
3. "What are Recipes?" + ambiguous (admin) suffix
Recipes tab now carries its own curator-block explainer (the
shared one was moved inside Browse view so it doesn't bleed
across tabs). The grey "(admin)" suffix becomes a yellow
.admin-only-hint chip with a title tooltip — visibility hint is
now unambiguous: yellow chip = "only you see this", non-admins
don't see the affordance at all.
* schema: renumber v51..v58 → v52..v59 to make room for main's v51
Main 0.54.29 introduced a NEW v51 (table_registry.bq_fqn — issue #343)
that releases ahead of this branch. The unified-stack chain v51..v58
shifts up by one so main's v51 stays as the released schema and ours
become v52..v59. Function names, internal version bumps, dispatch
ladder thresholds, and the migration-test references all move
together. Subsequent merge with main lands the bq_fqn column at the
freed v51 slot.
* fix(seed): seed admin lands in BOTH Admin AND Everyone groups
The LOCAL_DEV_MODE / SEED_ADMIN_EMAIL bootstrap only added the seed
user to Admin. Everyone-scoped grants — the canonical "every-user-
sees-this" pattern for Required onboarding — didn't surface for the
seed admin's own /catalog because they weren't in Everyone. Symptom:
admin grants a Required-tier package to Everyone, then sees it on
/catalog still rendered with an "Add to stack" button (because the
admin's resolved required_ids was empty for that package).
The dual-membership keeps Admin (authorization) and Everyone
(default-grant target) intentionally separate per the design comment
on UserRepository.create — every membership remains traceable to a
concrete row, just now with a system_seed row in Everyone too. Both
INSERTs go through UserGroupMembersRepository.add_member which is
idempotent on (user_id, group_id), so re-fires on every lifespan
startup don't duplicate rows.
Regression test in test_main_seed_admin_everyone.py.
* style: unify admin-only hints across marketplace + memory detail pages
Replaces three stale ``(admin)`` parentheticals with the same yellow
``admin-only`` chip introduced for /catalog tab actions. Same tooltip
copy ("Visible only to admins — analysts won't see this …") so the
visibility hint is unmistakable wherever it appears:
- Hard delete on marketplace_plugin_detail (admin-only destructive
action — same gating as the original suffix conveyed).
- Hard delete on marketplace_item_detail (same).
- Edit link on memory_domain_detail (title-attr only before; now a
visible chip too).
Non-admin viewers never saw these affordances — the gates are
unchanged. Pure styling pass for consistency.
* fix(catalog): exclude soft-deleted data packages + memory domains from Browse
``StackResolver._fetch_entries`` and ``browse_admin`` were querying
data_packages / memory_domains without a ``deleted_at IS NULL`` guard.
A package soft-deleted via /admin/* (v54 soft-delete contract) stayed
visible on /catalog and /memory until either an Undo or a hard delete
— directly contradicting the soft-delete UX which is supposed to
remove the affordance immediately and only retain the row for the
Undo window.
The repository accessors (DataPackagesRepository.list,
MemoryDomainsRepository.list, list_packages_of_table, etc.) already
filter deleted rows; this commit brings the resolver's direct SQL in
line with that contract.
Regression test in test_stack_resolver_browse_admin.py.
* fix(catalog): Add/Remove updates full card chrome, not just button
The previous _applyStackChange flipped only the footer button label —
the card border (.is-in-stack class), top-right "In stack" badge, and
button color class (--add / --remove) stayed at their server-rendered
state. After Add the user saw the button checkmark but the rest of
the card still looked like "available, not in stack". They read this
as "the change didn't take — let me refresh".
This commit makes the optimistic update mirror what the server-side
macro renders for the new state:
* ``c.classList.toggle('is-in-stack', becameInStack)`` — flips the
border + visual state class.
* Top-right ``.stack-card__req-badge--instack`` badge is injected on
Add, removed on Remove (skipped when ``data-requirement='required'``
— that slot is owned by the Required badge).
* Button text is "Remove" / "+ Add to stack" matching the macro
(was "✓ In stack" which was visually nice but inconsistent).
* Button color class --add / --remove swaps so the destructive Remove
tint kicks in immediately.
The clone-into-My-Stack path applies the same updates so the new card
in My Stack reads identically to a server-rendered in_stack card.
Mirrored in /corporate-memory.
* fix(memory): four Devin-review bugs on /memory drill-down + manifest
PR #333 Devin review surfaced four real bugs that ship a broken
/memory experience even though the unit tests passed.
1. Manifest md5 omits is_required + content (app/api/sync.py:836-840)
_build_memory_domains_section hashed only (id|title|status) per
item. _build_per_domain_markdown routes items between "## Required"
and "## Approved" by is_required and embeds full content — so an
admin edit of either dimension left the manifest md5 unchanged,
`agnes pull` skipped the re-fetch, and the analyst kept a stale
bundle.md. Now both fields participate in the hash.
2. required_count always 0 (src/repositories/memory_domains.py)
list_items_of_domain only SELECTed (id, title, status) so the
`it.get("is_required")` in the manifest builder always evaluated
to None → required_count = 0 regardless of actual state. The
manifest builder advertised a count it could never compute. Now
projects is_required + content too (required by fix 1 anyway).
3. Vote URL 404 (memory_domain_detail.html:289-290)
Constructed `/api/memory/items/{id}/vote` but the route is
`/api/memory/{id}/vote`. Every upvote/downvote button was a
silent no-op.
4. Dismiss/undismiss URL + method both wrong (memory_domain_detail.html:296-305)
Constructed `/api/memory/items/{id}/dismiss` (extra /items/) and
/undismiss (no such route — undismiss is DELETE on /dismiss).
Both buttons silently 404'd. Now POST + DELETE on
`/api/memory/{id}/dismiss` per app/api/memory.py:635/675.
* fix: multi-agent reviewer findings — vendor-token scrubs + manifest md5 predicate + soft-delete filter
Three reviewer findings from the multi-agent review on PR #333,
fixed in-place per CLAUDE.md issue-economy rule.
Reviewer-rules (Important — vendor-agnostic OSS):
- app/main.py:218 comment: replaced 'foundryai-prod' with generic
'a customer prod instance' phrasing. Public OSS repo must not
carry customer-specific tokens (CLAUDE.md § Project conventions).
- tests/test_table_registry_v56_docs.py:70 fixture string:
replaced "user_brand_affiliation = 'groupon'" with 'acme' on
the same rule.
Reviewer-architecture (closes still-unresolved Devin 🚩 ANALYSIS):
- app/api/sync.py _build_memory_domains_section: md5 hash loop now
filters items to the SAME predicate the bundle renderer uses
(is_required OR status='approved'). Pre-fix the hash iterated ALL
items but _build_per_domain_markdown only rendered the union of
required items + approved-non-required items — so an admin edit
to a pending/rejected non-required item flipped the md5 against
an identical-bytes bundle, triggering a wasteful re-fetch on
every analyst's next 'agnes pull'. The earlier commit fixed the
hash-input fields (is_required + content); this closes the
set-of-items asymmetry Devin separately flagged.
Reviewer-RBAC (minor cleanup):
- app/resource_types.py _data_package_blocks and _memory_domain_blocks
now filter 'WHERE deleted_at IS NULL' (v54 soft-delete column) so
the /admin/access UI doesn't surface soft-deleted entities as
grantable. Mirrors the existing filter on _recipe_blocks. No
security leak pre-fix (resolver double-filters and re-checks at
serve time), just UI cleanliness.
- app/services/stack_resolver.py add_to_stack: docstring note
added explaining that authorization is enforced at the API layer
(app/api/stack.py can_access gate), not at the resolver. The
initial review suggested adding a defensive 403 here, but that
broke 5 existing tests that legitimately call add_to_stack
directly without setting up grants first; the docstring captures
the contract instead. stack() already intersects subscriptions
with current available_ids on every read, so a 'zombie' row from
a misuse never leaks into the user-facing manifest.
* release: 0.55.0 — unified Browse + My Stack (Data Packages + Memory), schema v48→v59, 3 BREAKING
* feat(cli): agnes marketplace search/detail/add/remove + retire stale subcommands
Unified CLI surface for the v28+ marketplace: search across Curated and
Flea Market (RBAC-filtered server-side), drill into a single item's
detail, add/remove from your stack. Replaces opt-out era commands that
no longer reflect how users compose their stack.
CLI changes:
- Added: agnes marketplace {search,detail,add,remove}
- Removed: agnes my-stack toggle (opt-out semantics, curated-only)
- Removed: agnes store {list,show,install,uninstall} (consumer-side ops
moved under marketplace; store now covers only creator-side upload,
update, delete, mine)
ID format unifies curated and flea: marketplace_id/plugin_name (slash)
routes to /api/marketplace/curated/..., bare UUID routes to
/api/store/entities/... (flea bundles skills/agents into a synthetic
plugin server-side, so the analyst sees a single add/remove surface).
Templates:
- claude_md_template.txt: rewritten marketplace section as operational
guidance for Claude Code (discovery, stack management, behaviour
notes). Dropped the static {% if marketplaces %} listing — the CLI is
the source of truth for what's in the stack at any moment, so a
snapshot rendered at init time would lie the moment the user runs
agnes marketplace add/remove. Same discipline already applied to
tables and metrics.
- agnes_workspace_template.txt: cheat sheet adds 5 marketplace
one-liners; keeps the file's reference-doc tone (the original
commit's intent: 'what is this thing, how does it work, how do I
uninstall it').
Docs: HOWTO/05-customizing-skills.md rewritten around the new CLI flow;
the opt-out section is replaced by 'Removing items from your stack'.
Tests: new test_cli_marketplace.py covers all four subcommands incl.
RBAC/409 paths (system plugin guard, not-approved flea entity);
test_cli_store.py trimmed to the retained creator-side commands.
* release: 0.54.1 — agnes marketplace CLI redesign + retire stale subcommands
Last commit on the PR per CLAUDE.md hard rule. Patch bump (0.54.0 →
0.54.1) bundling the BREAKING removals of `agnes my-stack toggle` and
`agnes store {list,show,install,uninstall}` plus the new unified
`agnes marketplace {search,detail,add,remove}` surface.
No DB migration; no operator-facing config change. Operators on
floating tags (`:stable`) auto-upgrade transparently. Analyst CLI
upgrade prompt fires on next `agnes pull`; users invoking the
retired commands get "No such command" with the new `agnes
marketplace` substitution called out in the BREAKING bullets.
---------
Co-authored-by: Minas Arustamyan <arustamyan.minas@gmail.com>
Co-authored-by: ZdenekSrotyr <zdenek.srotyr@keboola.com>
* Capture session paths via SessionStart hook + lock parallel pushes
Replace the encoding-based scan of ~/.claude/projects/<encoded-cwd>/ with
a queue file populated by a new `agnes capture-session` SessionStart hook.
The hook reads the documented `transcript_path` field from Claude Code's
hook stdin JSON, sidestepping the cwd-to-folder encoding (which is an
internal implementation detail and varies by Claude Code version).
- New `agnes capture-session` subcommand appends transcript_path to
<workspace>/.claude/agnes-sessions.txt. Silent on all malformed input
so a hook chain failure doesn't clutter Claude Code startup.
- `agnes push` now consumes the queue: atomic snapshot rename guards
against hooks writing during the push window, successful uploads land
in agnes-sessions-uploaded.txt (TSV: timestamp + path), failed paths
are requeued.
- Cross-platform single-instance lock via the filelock package (fcntl
on POSIX, msvcrt on Windows). Concurrent SessionEnd hooks — common
when the user closes several sessions at once — silent-exit on the
losing side instead of all racing the upload.
- Recovery: pre-existing snapshot files from a crashed push are picked
up and processed before the live queue.
- The SessionStart `agnes push` self-heal entry is dropped — it became
redundant once the queue persists across runs (orphans from headless /
crashed sessions ship out on the next interactive SessionEnd push).
Existing workspaces auto-migrate via the marker-based replace logic.
- Legacy encoding scan stays available behind `--legacy-scan` for one-
off backfills of sessions predating the queue.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add /agnes-private + statusLine indicator for private sessions
Users handling sensitive data inside Claude Code can now opt a session
out of the Agnes upload pipeline, either proactively (right after session
start) or reactively (mid-session). The `/agnes-private` slash command
runs `agnes mark-private` deterministically via `!`-prefix direct bash —
no AI in the loop. A workspace-installed statusLine surfaces a
`🔒 agnes-private` indicator in Claude Code's status bar so the user
sees the state at a glance.
Authoritative source of "do not upload" is a separate file
`<workspace>/.claude/agnes-sessions-private.txt` (one session_id per
line). Both `capture-session` (queue writer) and `push` (queue reader)
consult the list. This makes the slash-command / SessionStart-hook race
impossible by construction: whichever runs first, the session is correctly
filtered out.
- `agnes mark-private` reads `CLAUDE_CODE_SESSION_ID` from env (set by
Claude Code in every bash subprocess it spawns — stable documented API)
and appends to the private list.
- `agnes statusline` reads the session JSON Claude Code pipes on stdin,
checks the private list, and emits the indicator or nothing. Optimized
for the high call frequency of statusLine renders.
- `capture-session` extracts session_id from hook stdin and skips queue
write when the ID is already on the private list (race protection).
- `push` filters snapshot entries by the private list and appends to a
per-workspace audit log `agnes-sessions-private-skipped.txt`.
- Queue format migrated from `<path>` to `<session_id>\t<path>`; legacy
one-column lines still parse (empty session_id, still upload, can't be
marked private retroactively — fine, they pre-date the feature).
- `install_claude_hooks` writes a workspace statusLine unless the user
already has a custom one (warn + preserve). Idempotent re-init.
- `install_claude_commands` ships `agnes-private.md` alongside
`update-agnes-plugins.md`. Per-template fallback so a missing template
doesn't get clobbered with the wrong content.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix setup-prompt + CLAUDE.md marketplace copy + drop skills step
Three issues against the post-PR-#240 / post-PR-#237 state:
1. Setup prompt's marketplace block trailer (both has-stack and
empty-stack variants) claimed the SessionStart hook keeps the
marketplace clone in sync via `agnes refresh-marketplace --quiet`
on every session and that admin grants land automatically — both
false since PR #237 (0.47.x) moved the install/update path out of
the hook into the `/update-agnes-plugins` slash command. The hook
is `--check`-only: detects server-side changes, prompts the user
to run the slash command, which does the full reconcile
interactively with output visible in the transcript.
2. The empty-stack variant framed composition as "admin grants only",
missing the actual three-source served stack:
(admin RBAC ∩ /marketplace subscriptions)
∪ system-mandatory plugins (admin-pinned, auto-applied)
∪ Flea market installs (skills/agents bundled, plugins standalone)
Updated copy spells out all three sources so analysts know where
their stack picks live, and what the SessionStart hook actually
does on change detection.
3. CLAUDE.md template's "Agnes Marketplace" section conflated
eligibility (`resolve_allowed_plugins` — what's listed) with served
stack (`resolve_user_marketplace` — what actually reaches Claude
Code). The two are different: a user can be RBAC-eligible for a
plugin without having subscribed to it on /marketplace. Rewrote
the section to distinguish the eligibility set from the served
stack and to describe the `--check`-only hook accurately.
Plus: deleted the setup prompt's interactive Skills step (final step
before Confirm). The named-opinion question — "do you want me to
bulk-copy every skill into ~/.claude/skills/agnes/ or pull on-demand
via `agnes skills show <name>`?" — had no obvious right answer for
new users at the tail end of a wall of technical steps. On-demand
lookup is the one-size-fits-all default; `agnes skills list/show`
remain discoverable and the CLAUDE.md template references specific
skills inline (e.g. agnes-data-querying in the BigQuery section)
where they're relevant. Layout: Confirm shifts from step 9 to step 8.
Tests updated, full setup/marketplace/welcome surface green (115
passed). Remaining full-suite failures are pre-existing (BQ/Keboola
fixtures, Windows charmap collection error in test_v26_keboola_e2e)
— verified against a clean stash, unrelated to this diff.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix session-queue race + snapshot PID-reuse data loss
Two blocker fixes from the PR #242 review:
1. Concurrent SessionStart hooks could corrupt the queue file on
Windows. Python's `open(path, "a")` is not atomic there — the CRT
does not pass FILE_APPEND_DATA to CreateFile, so concurrent
appenders (user opening several Claude Code windows simultaneously)
could interleave bytes mid-line. The malformed lines then silently
fail the parser and the entries are dropped.
Fix: wrap append_to_queue, requeue_failed, and snapshot_queue in a
short-lived FileLock on a dedicated `agnes-queue.lock`. Separate
from `agnes-push.lock` so capture-session hooks don't block on the
push command. New test_append_concurrent_threads_no_corruption
reproduces the race with 4 threads x 50 appends.
2. Snapshot filenames embedded only the PID (`agnes-sessions.snapshot.
<PID>.txt`). After a crashed push left a snapshot on disk and the
OS recycled the PID for a new push, `os.rename` would atomically
overwrite the recovery snapshot — every entry in it lost, silently.
Fix: append a uuid8 hex tail (`agnes-sessions.snapshot.<PID>.
<uuid8>.txt`). find_recovery_snapshots already globs the prefix
so it picks up both old and new format. New
test_snapshot_filename_is_unique_per_call asserts two consecutive
snapshots under the same PID don't collide.
Targeted tests green (47/47 in session_queue/capture_session/cli_push).
Full suite failures unchanged from baseline (pre-existing BQ/Keboola
fixture issues per CLAUDE.md).
* Auto-refresh workspace hooks + bash-wrap all hook entries (Windows)
Fixes from PR #242 second review (ZdenekSrotyr):
1. `uv.lock` regenerated to include `filelock 3.29.0` (declared in
pyproject.toml but missing from the lock file — CI's
lockfile-consistency check would fail; `uv pip install` on a clean
cache would silently miss the dep).
2. `agnes self-upgrade` now auto-refreshes the workspace Claude Code
hooks via the new `cli.lib.hooks.maybe_refresh_claude_hooks`. Closes
the silent-stop migration gap: a v0.48 workspace would auto-upgrade
the CLI from its existing SessionStart self-upgrade entry but never
pick up the new `agnes capture-session` SessionStart hook, leaving
the queue empty and `agnes push` uploading nothing.
The refresh fires on both the "info is None" fast path (CLI already
current — catches the second SessionStart after a prior upgrade)
and the install-success path. Guarded by `workspace_has_agnes_hooks`
so it never writes `.claude/settings.json` into directories that
aren't Agnes workspaces (e.g. `agnes self-upgrade` invoked from
`~/`). Errors are surfaced on stderr but never flip the upgrade exit
code.
3. All Agnes-managed hooks are now wrapped in `bash -c "..."`. The
self-upgrade+pull chained SessionStart entry was the only one still
shipping unwrapped — Claude Code on Windows runs hook commands
directly without a shell, so the `;` chain + `2>/dev/null` +
`|| true` shell syntax silently no-op'd on native Windows installs
without Git Bash on PATH. Workspaces still on the old form
auto-upgrade via the refresh path above.
Tests: +12 in test_lib_hooks.py (guard semantics, v0.48→v0.49
migration end-to-end, third-party-hook preservation, bash-wrap
invariant). +5 in test_self_upgrade.py (refresh fires on info=None,
fires on install success, skipped on failure, skipped on --check-only,
refresh failure never flips exit code).
130 targeted tests green. The 2 pre-existing Windows path-separator
failures in `test_smoke_test_detects_version_mismatch[uv|pip]` are
unrelated (path mismatch `\fake\uv\bin\agnes` vs `/fake/uv/bin/agnes`
in test asserts, pre-PR baseline).
* CHANGELOG: document PR-242 main features
Closes ZdenekSrotyr #4: the [Unreleased] block was missing entries for
the PR's primary surface — only the post-merge fix bullets and the
unrelated setup-prompt copy change were captured. Adds:
- ### Added: 6 bullets covering the session capture queue + new
`agnes capture-session` subcommand, `/agnes-private` slash + `agnes
mark-private`, `agnes statusline` + statusLine wiring, `--legacy-scan`
opt-in fallback, single-instance push lock, and the new `filelock`
runtime dep.
- ### Changed: BREAKING bullet on the SessionStart / SessionEnd hook
wire format change (capture-session as first SessionStart entry,
push self-heal removed, SessionEnd push detached via nohup, all
entries bash-wrapped). Folds the prior standalone bash-wrap bullet
into this consolidated entry — Z's review flagged the layout shift
as BREAKING, and grouping the related sub-changes makes the
migration story readable in one place.
- Operator migration is auto-handled by `maybe_refresh_claude_hooks`
invoked from `agnes self-upgrade` (separate Changed entry below).
No `agnes init` re-run required. Pre-queue session jsonls on
upgrading workspaces still need a one-off `agnes push --legacy-scan`
— flagged in the BREAKING bullet.
No code change; doc only.
* Drop permanent 4xx uploads instead of requeueing forever
Closes ZdenekSrotyr #5. Previously the push retry path requeued any
non-200 response except the literal "file not found on disk", so 401
(token expired), 403 (RBAC denial), 413 (payload too large), 400
(server-side validation) cycled through every push run forever — the
queue grew without bound and each run re-bombarded the server with the
same deterministically-failing upload.
Now 4xx (except 408 Request Timeout + 429 Too Many Requests, which the
HTTP spec marks as transient) is dropped and audit-logged to
`<workspace>/.claude/agnes-sessions-failed.txt`:
<iso_ts>\t<session_id>\t<status>\t<transcript_path>
5xx and network errors continue to requeue — those reflect server /
transport state that can change between runs, so retry is the right
behavior.
The audit log piggybacks on the push single-instance lock
(agnes-push.lock) — push is the only writer to this file, same as the
existing `mark_uploaded` and `mark_private_skipped` paths, so no
separate filelock is needed.
`agnes push --json` surfaces a new `dropped_permanent` counter; non-
quiet stdout mentions the audit-log path so operators tailing the
output have a pointer to the forensic trail.
Tests: +7 in test_cli_push.py (401/400/403/413 → drop; 408/429 →
requeue; 500/502/503 → requeue; network exception → requeue;
--json `dropped_permanent` counter; stdout audit-log pointer). +1 in
test_session_queue.py (mark_failed_permanent TSV format).
127/129 targeted tests green. The 2 pre-existing Windows
path-separator failures in `test_smoke_test_detects_version_mismatch
[uv|pip]` are unrelated (path mismatch `\fake\uv\bin\agnes` vs
`/fake/uv/bin/agnes` in test asserts, pre-PR baseline).
* Catch OSError in push lock acquisition
Closes ZdenekSrotyr #8. `acquire_or_skip` in `cli/lib/push_lock.py`
previously caught only `filelock.Timeout`. Any `OSError` from
`FileLock.acquire` — read-only filesystem, permission denied on
`.claude/`, disk full, hardware I/O failure — propagated as an
unhandled traceback.
Two visible failure modes:
- SessionEnd hook: `|| true` in the wrapper swallowed the error, so
daily pushes silently never ran. Operator had no signal.
- Manual `agnes push`: ugly Python traceback dumped to the terminal
instead of a clean exit.
Now `OSError` is treated the same as `Timeout` — yield `None`, caller
returns cleanly with rc=0. The operator's environment in these
scenarios has bigger problems than missing session uploads, so we
swallow rather than retry-loop or surface a noisy warning.
Test: `test_push_silent_exit_when_filelock_raises_oserror` patches
the `FileLock` used inside `push_lock` to raise OSError on acquire,
verifies push exits 0 with no traceback and the queue is preserved
for the next attempt.
* Address remaining S2 items from PR-242 review
Four items from ZdenekSrotyr's S2 list:
S2.10 — `_install_statusline` truthy check (cli/lib/hooks.py): replace
`if existing:` with explicit `if existing is None or existing == "":`.
Documents and tests the behavior for both edge cases (explicit-null
and empty-string `statusLine`) — both treated as "not configured"
rather than "explicit user opt-out", so we install ours. Two new
tests in test_lib_hooks.py pin the contract.
S2.6 — onboarding docs for /agnes-private. New "Private sessions"
subsection in `config/claude_md_template.txt` (next to Data Sync)
covering the slash command, statusbar indicator, and audit-log
location. One-line tip in `app/web/setup_instructions.py` so the
feature is discoverable at onboarding.
S2.9 — e2e privacy test (tests/test_e2e_privacy.py). Wires
capture_session → mark_private → push against a recording fake
api_post and asserts zero session uploads for the marked one.
Three cases: mark-before-capture (queue write skipped),
mark-after-capture (push-side filter catches it + audit-logs),
control (unmarked sessions upload normally).
David #8 — `--legacy-scan` help text now documents the
private-list gap (legacy entries carry empty session_id, so
the filter is not consulted). The practical impact is bounded —
pre-queue sessions cannot have been marked private since the
private list is a queue-era feature — but the disclaimer in the
help text means an operator running a backfill is not surprised.
68 targeted tests green (3 new e2e + 2 new truthy edge tests +
existing). 2 pre-existing Windows path-separator failures in
test_smoke_test_detects_version_mismatch[uv|pip] unchanged.
Remaining S2 items (statusline mkdir push-back, capture-session
silent-fail follow-up) handled in PR comment + follow-up issue
respectively.
* Address remaining S2 follow-ups (David #8, S2.7, David #11)
Three items left over from Mina's bbf63472 batch — that commit
addressed S2.6/S2.9/S2.10 + documented David #8 in help text but
deferred the actual implementations of S2.7, David #11, and the real
David #8 fix to follow-ups. This commit closes them.
David #8 — `agnes push --legacy-scan` now consults the private list.
Claude Code names jsonls `<session-id>.jsonl`, so the file stem IS
the session id; the legacy-scan path can apply the same private filter
the queue path uses. Both the dry-run and live-upload code paths fixed.
Help text updated (no longer warns the filter is bypassed). Two new
tests in test_cli_push.py cover the upload-skip path + the dry-run
`would_skip_private` segregation.
S2.7 — `statusline`/`is_private` no longer mkdir-pollutes arbitrary
workdirs. Split `_claude_dir` into `_claude_dir_writable` (used only
from `add_private`) and `_claude_dir_readonly` (no mkdir). The
read-only public helpers (`private_list_path`, `read_all_private`,
`is_private`) compose the no-mkdir variant by default; `add_private`
opts in via `writable=True`. Added a process-local mtime-keyed cache
around `read_all_private` so in-process callers (push doing one stat
per upload candidate, future `agnes diagnose`) don't re-parse the
file on every check. Cache eviction on `add_private` so a sub-second
write+read sequence doesn't see stale data even on coarse-mtime
filesystems. Two new tests pin the no-mkdir contract + the
in-same-second add+read consistency.
David #11 — `agnes capture-session` writes a breadcrumb log on every
invocation. New `<workspace>/.claude/agnes-capture-session.log` TSV:
`<iso_ts>\t<outcome>\t<detail>` where outcome covers every silent-
exit path (`ok`, `private_skip`, `empty_stdin`, `bad_json`,
`not_object`, `no_transcript_path`, `stdin_read_error`,
`write_error`). Gives operators a signal to detect "hook fires but
queue stays empty" — without it, an upstream Claude Code stdin-
contract change is invisible because the hook always exits 0. Log
rolls at 256 KiB so it doesn't grow unbounded on long-lived
workspaces. Best-effort: a breadcrumb-write failure is itself
swallowed so the hook contract stays "exit 0 always". Skipped in
non-Agnes workdirs (no `.claude/` exists) so opening Claude Code
in `~/` doesn't pollute it. Five new tests in test_capture_session.py
cover the success / bad_json / no_transcript_path / private_skip /
no-pollute paths.
115 targeted tests green (test_cli_push, test_capture_session,
test_private_list, test_session_queue, test_e2e_privacy,
test_lib_hooks, test_statusline, test_mark_private).
---------
Co-authored-by: Minas Arustamyan <arustamyan.minas@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: ZdenekSrotyr <zdenek.srotyr@keboola.com>
* feat(home+news): state-aware /home + /news + admin-edited news section
Squash of the vr/home-page feature work for clean rebase onto main.
Original 18-commit history preserved in branch backup/vr-home-page-pre-rebase.
What's in this PR:
**State-aware /home page**
- New `/home` route with hero + auto-mode + connectors (Asana / GWS /
Atlassian) + lookarounds. Onboarded vs not-onboarded state-machine
branches a single template (`home_not_onboarded.html`); the install
steps, "Setup a new Claude Code" CTA (90-day PAT mint), and per-
connector setup prompts hide once `users.onboarded=TRUE`. A
completion badge replaces them.
- "Mark me as offboarded" button reverses the flag without an SQL UPDATE.
- `users.onboarded BOOLEAN` column added; default FALSE; flipped by the
CLI's `agnes init` post-success POST and the `/admin/users` API.
- Connector setup prompts pre-check whether the tool is already
installed/connected before re-running setup.
- GWS scope set widened to include Google Chat (`chat.spaces`,
`chat.messages`).
**Single template + design tokens**
- `dashboard.html` now extends `base.html` via the new
`{% block layout %}` opt-out (full-width pages skip the 800px
`.container`). Net: every page shares one shell.
- `style-custom.css` `:root` extended with `--space-{7,9,10,12}`,
`--radius-2xl`, `--shadow-{card,elevated}`, `--text-{muted,disabled}`,
`--focus-ring`, `--transition-*`, `--width-{narrow,app,wide}` so
inline page styles can migrate incrementally.
**Auth redirects honor AGNES_HOME_ROUTE**
- `safe_next_path` resolves the configured home route when no `default=`
is passed; OAuth callbacks, magic-link clicks, password form, and
LOCAL_DEV_MODE shortcuts now land on `/home` (or whatever the operator
picked) instead of always /dashboard.
**News section + /news permalink + /admin/news editor**
- Schema-bumped `news_template` table (single versioned entity, draft +
publish gate). `published BOOLEAN` distinguishes draft from public;
monotonically-increasing `version` per save; rows >30d pruned on
save except the currently-displayed published version.
- `/home` bottom-of-page renders the latest published intro with a
"Read more →" link to `/news` (which renders the full body).
- `/admin/news` editor with sandboxed live preview, versions table,
per-row Unpublish, Format-help cheatsheet.
- `agnes admin news show / draft / edit / publish / unpublish /
versions / export` (CLI). Talks to the live server via the
`/api/admin/news/*` endpoints (PAT-authed) — no direct DB access
so it coexists with a running uvicorn.
- **Optimistic-lock guard**: `agnes admin news publish --version N` and
PUT/PATCH endpoints accept `expected_version` and 409 with structured
`{error: "version_conflict", expected, actual, actual_by}` when a
concurrent admin replaced the draft. Edit refuses to overwrite a
draft authored by someone else without `--force` or
`--expect-version`.
- nh3 (Rust-backed ammonia) HTML sanitizer; iframe pre-pass strips
any iframe whose src is not on the YouTube/Vimeo/Loom allowlist;
javascript:/data: schemes blocked everywhere.
- Author CSS vocabulary: `.news-hero` (blue gradient hero block),
`.callout`/`.callout-{info,warn,success,danger}`,
`.video-embed`, `.news-section`, `.news-grid-{2,3}`, `.news-cta` —
all consolidated in `style-custom.css` under "News content
vocabulary (shared)" so /home perex, /news body, and /admin/news
preview share one source of styling.
- Code-inside-`<pre>` contrast fix (was unreadable amber-on-silver).
- `.news-content` table styling (border, header band, row-hover).
**`scripts/dev/run-local.sh`** — local uvicorn launcher. Pulls Google
OAuth client id/secret from GCP Secret Manager
(`AGNES_OAUTH_GCP_PROJECT`-driven, no vendor defaults), points
`AGNES_CLI_DIST_DIR` at `./dist` so the wheel endpoint resolves, and
`--dev` flips `LOCAL_DEV_MODE=1` + `AGNES_HOME_ROUTE=/home` for one-
command iteration. `LOCAL_DEV_MODE=1` also enables the FastAPI debug
toolbar.
**CLAUDE.md "Run tests before every push" section** codifies
`pytest tests/ -n auto -q` as non-negotiable before each push.
**Tests**: 51 + 14 + 8 = 73 new tests across news-template repo,
sanitizer, API, web, CLI; plus updated home/auth/template tests for
the new shared-shell architecture.
Origin docs (gitignored, customer-fork content):
docs/brainstorms/home-page-requirements.md,
docs/plans/2026-05-07-001-feat-home-page-plan.md.
* feat(cli): agnes onboarded {on,off,status} — self-scoped flag toggle
User-facing equivalent of the in-page "Mark me as (off)boarded" button
on /home. POSTs /api/me/onboarded with {onboarded, source}; --source
overrides the audit-log marker so flips made from the CLI vs the web
button vs agnes init automation stay distinguishable.
`status` reads via /api/me/profile (when present); falls back to a
quick body-marker scan of /home so the read path doesn't write an
audit_log row. PAT-authed via cli.client.api_post — same convention
as agnes admin news / agnes admin add-user etc.
Tests: 5 covering on/off/status round-trip, idempotency, and
audit-log source recording. Full suite holds at 12 pre-existing
failures (same set as before).
* ui(nav+home): primary nav reorg + green What's new band + /marketplace link fix
Primary nav (post-rebase audit + per-user feedback):
- Items: Home → Marketplace → Data Packages → Memory. Admin dropdown
for admins only. The "Dashboard" label was renamed Home — point still
resolves through `home_route` so customer instances on /dashboard
still land there.
- Activity Center moved into the Admin dropdown. Per-team adoption
analytics is admin-consumed in practice; the route still allows
any authed user for direct deep-links so existing /home tile +
bookmarks keep working.
- Memory link added (→ /corporate-memory) — was previously buried in
the /home "Look around" tiles.
- Setup local agent + My Stack dropped from main nav. Setup is the
/home install flow's home now; My Stack lives as a tab inside
/marketplace.
/home tweaks:
- Plugin marketplace tile now points at /marketplace (was /store —
legacy from before the marketplace rebrand landed in #230).
- "What's new" section header gets a green band (success-flavored
D1FAE5 background, A7F3D0 border, darker green title) so the
bottom-of-page news block visibly distinguishes from the blue
install-hero at the top. Header strip only — body stays white.
Test fix: test_home_route_resolution renamed `dashboard_link_uses_home_route`
→ `home_link_uses_home_route` and asserts `href="/home">Home` instead
of `href="/home">Dashboard` after the label change.
* fix(home): decouple Step 3 + Connect-tools collapse from server onboarded flag
The server-side `users.onboarded` flip happens through two paths:
1. Explicit user click on "Mark me as onboarded" or `agnes onboarded on`.
2. Implicit `agnes init` POST → /api/me/onboarded on success.
Path 2 produced a UX surprise: an analyst running `agnes init` mid-flow
reloaded /home and saw Step 3 (auto-mode) + Connect-your-tools auto-
collapse to summary bars. They were actively working through those
sections — the install POST never signalled "I'm done with the rest
of setup", just "Agnes itself is installed".
Decouple the section-collapse decision from the server flag:
- Step 1 + Step 2 install blocks: still hidden on `onboarded=TRUE`
(their completion is a hard server signal — Agnes IS installed).
- Step 3 + Connect-your-tools: render flat by default in BOTH states.
Wrapped in `<details class="setup-collapsible" open>` so the
browser's native disclosure handles per-section toggle without JS,
but the `<summary>` is CSS-hidden until the page-level
`data-setup-minimized="1"` attribute is set on `.home-mock`.
- New "Minimize setup view" toggle inside the blue install-hero,
rendered only when onboarded. Click flips the data-attr on
`.home-mock` AND removes the `open` attribute from each
`<details>`. State persists in `localStorage["agnes_home_setup_minimized"]`
so the choice survives reloads but is per-device.
- "Show full setup view" (the same button when minimized) re-opens
both `<details>` and clears localStorage.
When minimized, each `<details>` still has its own native expand/
collapse — click the gray summary bar to peek at one section without
toggling the page-level minimize off.
Tests:
- test_step3_and_connectors_render_flat_when_onboarded_by_default —
asserts `<details class="setup-collapsible" ... open>` for both
sections post-onboarding and the absence of any server-rendered
`data-setup-minimized` attribute on the `.home-mock` root.
- test_minimize_toggle_visible_only_when_onboarded — toggle button
rendered only when onboarded.
Full pytest holds at 12 pre-existing failures (same set).
* feat(observability): optional PostHog integration (errors, LLM traces, replay, flags)
Off by default. Activates when POSTHOG_API_KEY is set in env. Defaults
to PostHog Cloud EU; override host for US Cloud or self-hosted.
Coverage:
- FastAPI 500 handler captures unhandled exceptions
- src/orchestrator.py rebuild + rebuild_source failures
- services/scheduler/ HTTP-job failures
- cli/main.py uncaught CLI errors (Typer.Exit/SystemExit/KeyboardInterrupt
skipped; flushes before re-raise so short-lived CLI invocations don't
drop events)
- connectors/llm/anthropic_provider.py + openai_compat.py emit
$ai_generation events with provider, model, latency, token counts
(prompt/completion bodies stay off unless POSTHOG_LLM_PAYLOADS=1
because LLM prompts here routinely include customer SQL/data)
- Browser snippet injected into every text/html response by
PosthogInjectionMiddleware — registered inside the GZip layer so it
sees uncompressed HTML before compression. Many templates are
standalone (their own DOCTYPE) and never extend base.html, so a
per-template include would miss them.
- Frontend: $pageview, $pageleave, JS error capture via window.error
and unhandledrejection handlers, masked session replay
(maskAllInputs: true plus CSS-selector mask for known data surfaces),
feature flags (browser posthog.isFeatureEnabled + server-side
feature_enabled with fallback for older SDKs).
Identification mode operator-configurable: none / id / email / full.
Default email ships user.id + email but never name. CLI entry point
moves from cli.main:app to cli.main:main (Typer wrapper).
Files:
- src/observability/posthog_client.py — lazy singleton, no network
when disabled, single-process flush on shutdown
- src/observability/llm_tracing.py — trace_generation context manager
- app/middleware/posthog_inject.py — HTML rewrite middleware
- app/web/templates/_posthog.html — browser snippet template
- docs/observability.md — operator guide
- config/.env.template — documented POSTHOG_* knobs
- tests/test_posthog_disabled.py + tests/test_posthog_client.py +
tests/test_llm_tracing.py — 18 tests covering disabled state,
identify-mode payloads, $ai_generation shape, error variant.
CHANGELOG entry under [Unreleased] Added.
* feat(observability): tag every PostHog event with environment + release
Splits PostHog dashboards cleanly between localhost / dev / staging /
production without manual tagging on every capture call.
- POSTHOG_ENVIRONMENT explicit override; auto-resolves to "local" when
LOCAL_DEV_MODE=1, else RELEASE_CHANNEL, else AGNES_DEPLOYMENT_ENV,
else "unknown".
- AGNES_VERSION → RELEASE_CHANNEL fallback feeds the `release` property
for "is this error new in this release?" cohorting.
- Backend gets both via the PostHog SDK's super_properties constructor
arg (every captured event picks them up automatically).
- Browser snippet calls posthog.register({environment, release}) inside
the loaded callback so $pageview, $exception, autocapture, etc. all
carry the same labels.
- request.state.user now populated by auth dependencies so the snippet
can actually call posthog.identify(user_id, {email}) for logged-in
users (previously the user block always resolved to None because
nothing wrote to request.state.user).
4 new tests cover env resolution: explicit > LOCAL_DEV_MODE > channel
> unknown, plus super-properties forwarding into the SDK constructor.
* feat(observability): inline user attrs on every PostHog event + debug throw route
PostHog's UI shows person properties on the Person profile page, not
inline on each event — so a reviewer triaging an exception couldn't tell
which user hit the bug without clicking through. Fix it on both sides.
- Backend capture_exception merges user_id / user_email / user_name into
the event properties (gated by POSTHOG_IDENTIFY_PII: none/id/email/full).
Backed by a new _user_props_for_event helper on PosthogClient.
- Browser snippet registers user_id + user_email + user_name as super-
properties via posthog.register({...}) so every $exception, $pageview,
and custom event coming from posthog.captureException() carries them
inline. Mirrors the backend so cross-referencing client/server events
doesn't require a person-profile lookup.
- /api/debug/throw — debug-only endpoint gated by DEBUG=1 (404 in prod).
Runs Depends(get_current_user) first so request.state.user is set when
the unhandled-exception handler captures the event. Lets operators
exercise the full observability path end-to-end without hand-rolling
a TestClient script. Configurable via ?kind=ValueError&msg=...
7 new tests cover: backend user-attr merge across identify modes,
anonymous request fall-through, browser snippet super-prop emission for
logged-in / anonymous / id-only / full-name cases.
* fix(observability): address minasarustamyan PR #231 review
Two bugs caught in review.
1. PosthogInjectionMiddleware dropped Response.background on every
return path. BaseHTTPMiddleware materialises the body and asks
subclasses to return a fresh Response — three paths in dispatch()
omitted background=, silently cancelling any BackgroundTask /
BackgroundTasks the route attached (audit logging, async webhooks,
email sends) with no log line. Fix: route every return through a
_passthrough() helper that forwards background.
Also adds a _MAX_BUFFER_BYTES (4 MB) cap so a streamed-HTML response
can't balloon RSS during buffering. Bigger bodies short-circuit
through with a warning rather than being injected.
Regression tests in tests/test_posthog_inject_middleware.py exercise
four return paths (snippet present, render-fail, double-injection
guard, non-HTML passthrough) plus the streaming-guard short-circuit.
2. $ai_input / $ai_output_choices were emitted without truncation, so
POSTHOG_LLM_PAYLOADS=1 silently dropped events past PostHog's ~32 KB
per-event ingest limit — exactly the calls (large prompts with
schemas / sample rows / SQL) an operator would want to inspect.
Fix: clip both at POSTHOG_LLM_PAYLOAD_MAX_CHARS (default 30000) with
an explicit "…[truncated N chars]" marker so readers don't mistake
truncated captures for complete ones. Metadata (provider, model,
tokens, latency, error) flows regardless. Three new tests cover
default-cap clipping, env-override, and pass-through under the cap.
37 PostHog tests pass.
## Summary
Two bugs in `agnes describe` surfaced from a real analyst session following the CLAUDE.md agent-rails discovery workflow. Together they break `agnes describe` end-to-end for any analyst (or analyst-AI) who follows the documented form.
### A) CLI parsing
`agnes describe TABLE -n 5` failed with `Missing argument 'TABLE_ID'`. Root cause: the command was registered as a `Typer.Typer` subcommand group via `app.add_typer(describe_app, name="describe")` + `@describe_app.callback(invoke_without_command=True)`, and that pattern mis-parses positional + short-int option in some orderings. Same pattern in `cli/commands/schema.py` works only because schema has no INTEGER short option. Fix: switch to flat `@app.command("describe")`.
### B) Server NaN
`/api/v2/sample/<id>` (called by `agnes describe`) returned HTTP 500 with `ValueError: Out of range float values are not JSON compliant: nan` whenever a row contained NaN. Fix: sanitize NaN/±inf to None before JSON serialization.
## Test plan
- [x] `pytest tests/test_cli_describe*.py` — added regression tests pinning `-n` parsing on either side of the positional.
- [x] `pytest tests/test_api_v2_sample*.py` — added regression test for NaN row → JSON `null` (not 500).
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/keboola/agnes-the-ai-analyst/pull/224" target="_blank">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1" alt="Open in Devin Review">
</picture>
</a>
<!-- devin-review-badge-end -->
Adds end-to-end flow for installing and keeping the per-user filtered
Claude Code marketplace in sync with the user's Agnes stack
(admin RBAC grants \ MyAIStack opt-outs U /store installs).
Setup (one-liner in install prompt step 5):
`agnes refresh-marketplace --bootstrap` clones the per-user marketplace
bare repo to ~/.agnes/marketplace, strips PAT from the cloned origin
URL, registers the local path with Claude Code, and installs every
plugin in the served manifest at --scope project. Replaces a 15-line
inline shell sequence that tripped Claude Code's agent-driven `rm -rf`
permission gate.
Auto-refresh (SessionStart hook installed by `agnes init`):
`agnes refresh-marketplace --quiet` runs every Claude Code session,
fetches+resets the clone (server rebuilds as orphan commits, so
pull --ff-only is impossible), and version-aware reconciles:
- missing in workspace -> claude plugin install <name>@agnes --scope project
- version differs -> claude plugin update <name>@agnes
- matches -> skip
Don't auto-uninstall plugins that disappeared from the manifest --
a transient empty manifest from the server would wipe the stack.
Hook output: when --quiet AND something actually changed, emits Claude
Code hook JSON on stdout -- `systemMessage` (transient toast) and
`hookSpecificOutput.additionalContext` (model-side system reminder),
both carrying the change summary plus a "/exit + restart Claude Code"
instruction (Claude only scans plugins at session start).
Windows hook compatibility: the refresh-marketplace hook command is
wrapped in `bash -c "..."` because Claude Code on Windows runs hook
commands directly without invoking a shell, so `2>/dev/null || true`
would otherwise be passed as literal argv tokens.
Cross-cutting:
- cli/lib/marketplace.py: shared CLONE_DIR + MARKETPLACE_NAME constants.
- cli/lib/hooks.py: SessionStart now has two independent entries
(pull + refresh-marketplace) so a failure in one doesn't suppress
the other; legacy `da sync` and prior single-pull layouts upgrade
cleanly on re-init.
- PAT injection on every git fetch via per-invocation credential
helper (token in \$AGNES_TOKEN env, never in argv or .git/config).
- Pre-snapshot of installed plugins captured BEFORE
`claude plugin marketplace update` so silent auto-applied version
bumps still fire notifications.
- scripts/dev/agnes-client-reset.sh: cleans ~/.claude/plugins/marketplaces/agnes,
~/.claude/plugins/cache/agnes, drops uv build cache, documents
workspace-scoped residue that can't be enumerated from the script.
- app/web/setup_instructions.py: legacy AGNES_DEBUG_AUTH path also
uses clone (direct HTTPS marketplace add is broken end-to-end on
every Claude Code distribution -- stores response as single file,
plugin source paths then 404).
28 new tests (test_cli_refresh_marketplace.py) + extended hook + setup
template tests cover bootstrap, fetch+reset ordering, version-aware
reconcile, project-path filtering, hook JSON shape, and the bash-c
Windows wrapper invariant.
Reuses cli.update_check.check() for the version probe — extended with
bypass_disabled=True so explicit user-typed self-upgrade is not silenced
by AGNES_NO_UPDATE_CHECK (which is for the implicit warning loop).
Install path: uv tool install --force when uv is on PATH; otherwise
curl + pip via sys.executable (NOT system python3, NOT --user — both
would land outside the agnes venv and silently no-op the upgrade).
Smoke test execs the binary at the install-resolved path (uv tool dir
joined with agnes-the-ai-analyst/bin/agnes, or sys.executable's sibling
agnes for pip) — never via shutil.which, which can resolve a stale shadow
on PATH and produce a false-positive smoke pass on the OLD version. Smoke
also asserts --version output contains info.latest via PEP 440 Version()
equality (so 0.40.0 does not falsely match 0.40.10).
On smoke fail: rollback to last_known_good.json (written only after a
previous run's smoke passed). Rollback rc is captured and surfaced on
stderr if it also fails. First-ever upgrade or unrecoverable rollback
prints the canonical bootstrap recovery: curl -fsSL <server>/cli/install.sh | bash.
AGNES_SELF_UPGRADE_IN_PROGRESS=1 is set for the duration of the run
and propagated to the smoke-test subprocess. Layer B's _check_version_headers
honors the sentinel and skips the < min hard-stop, so an in-flight
upgrade can never sys.exit(2) itself.
--force invalidates the update_check cache BEFORE probing. --force +
offline = exit 1 with explicit stderr (without --force, offline is silent).
--quiet suppresses progress output but never gags failure stderr.
Three first-try-failure-surface fixes from Pavel's #185 trace + the
template guidance question, all under PR #188's umbrella so they land
together with the file_server / parallel pull / Tier 1 work.
1. CLI clean-error wrapper — new AgnesTransportError raised by the
api_*/stream_download helpers when httpx times out / drops /
refuses, plus a top-level Typer wrapper (cli/main.py) that prints
one-line "Error: …" + actionable hint and exits non-zero. Full
traceback goes to ~/.config/agnes/last-error.log for support
forwarding. Unhandled Exceptions are caught at the same boundary
so no Python traceback ever leaks to the analyst's terminal.
Pavel's #185 Phase 3B: a 30-frame httpx traceback from a slow BQ
--remote query made it look like a CLI bug. Now: clean message +
hint pointing at `agnes snapshot create` / partition-column
guidance.
Entry point in pyproject.toml flipped from `cli.main:app` →
`cli.main:_run_with_clean_errors` so the wrapper actually runs
under the installed `agnes` binary.
2. agnes init / agnes pull --skip-materialize + progress bar.
--skip-materialize omits query_mode='materialized' rows from the
download set so a first init doesn't spend 44 minutes silently
pulling a single 6 GB parquet (Pavel's #185 Phase 1). Rich-driven
per-file progress bar with label/bytes/rate/ETA renders to stderr
when not --quiet and not --json. Aggregates across the parallel
ThreadPoolExecutor workers added earlier in this PR.
3. config/claude_md_template.txt: explicit one-line snippet pointing
at `agnes catalog --json | jq '.tables[] | select(.id=="<id>")'`
for per-table descriptions + restated invariant: "the description
field on each catalog row is the authoritative business-rules
text — re-read live, never copy into this file." Resolves the
regression-or-feature debate between Pavel (wants annotations)
and the user feedback that landed in the prior commit (don't
embed table-specific content; tables change). Catalog command
stays the source of truth.
Ports Minas's PR #172 (against pre-rename `da` CLI on main) and applies
the principle to the post-rename `agnes` CLI. Two distinct failure modes
on Windows consoles whose default codepage is cp1250 (cs-CZ) / cp1252
(en-US):
1. `agnes pull` and other Rich-progress codepaths
UnicodeEncodeError on Braille spinner glyphs. Fix: `cli/main.py`
reconfigures stdout/stderr to UTF-8 with errors='replace' at import
time on `sys.platform == 'win32'` so Rich's legacy-Windows render
path emits decodable bytes. Wrapped in try/except so pytest's
captured streams (which aren't TextIOWrapper) don't break.
2. `agnes skills list` and `agnes skills show`
UnicodeDecodeError when reading skill markdown containing em-dashes /
accented chars. Default `Path.read_text()` uses
locale.getpreferredencoding(False), which is the broken codepage on
Windows. Fix: every call site passes encoding='utf-8' explicitly.
Broader scope than #172 because:
- The bootstrap rewrite renamed/removed several files Minas's PR
patched (`cli/commands/analyst.py` -> rolled into init.py;
`cli/commands/sync.py` -> split into pull/push). Those targets no
longer exist; the equivalent code lives in init.py.
- Other call sites Minas didn't touch (still bare in his branch) are
patched here too — config.py / update_check.py / snapshot_meta.py /
setup.py / skills.py — so the codebase has zero locale-default text
I/O in cli/.
Side cleanup: stale `Run `da`` reference in snapshot_meta.py:88 fixed
to `agnes` while touching the file.
Replaces the BigQuery wrap-view pattern with a discovery + scoped-fetch toolkit driven by the analyst's Claude session. Adds /api/v2/{catalog,schema,sample,scan,scan/estimate}, da catalog/schema/describe/fetch/snapshot/disk-info CLI commands, sqlglot-backed WHERE validator, process-local quota tracker, agent rails skill (cli/skills/agnes-data-querying.md). BREAKING: BQ wrap views off by default — set data_source.bigquery.legacy_wrap_views=true for one cycle. Backward-compat field_validator on primary_key. Catalog cache now matches documented 300s TTL with RBAC fresh per request. Cuts release v0.14.0.
* fix(cli): versioned wheel URL in setup instructions; drop broken /cli/agnes.whl alias (#36)
* fix(cli): inline PEP 427 wheel filename in setup instructions
`uv tool install <server>/cli/agnes.whl` fails with
error: The wheel filename "agnes.whl" is invalid: Must have a version
because uv validates the filename in the URL path *before* fetching — so
the server-side Content-Disposition header (which has the real versioned
filename) is never consulted, and an HTTP redirect does not help either:
uv resolves the filename from the initial URL.
Fix the root cause by inlining the real PEP 427 filename into the setup
snippet the dashboard copies to the clipboard. The wheel filename is
resolved server-side via `_find_wheel()` and substituted into the lines
returned from `setup_instructions.resolve_lines()`, so both the read-only
HTML preview and the JS clipboard renderer get byte-identical output.
Also added `/cli/wheel/{filename}` to serve wheels at their PEP 427 path,
and kept `/cli/agnes.whl` as a 302 redirect for manual/legacy callers —
though that redirect alone is NOT sufficient for `uv tool install` (uv
validates before following redirects) and is there only as defense-in-depth.
Verified locally:
- `uv tool install <server>/cli/wheel/agnes_the_ai_analyst-2.0.0-py3-none-any.whl` succeeds
- `/install` HTML now renders the versioned URL; `/cli/agnes.whl` no longer appears in the rendered snippet
* fix(cli): remove /cli/agnes.whl alias entirely — it only confused users
The bareword alias was never actually usable:
- `uv tool install <server>/cli/agnes.whl` fails at filename validation
before any HTTP fetch, so neither the Content-Disposition header nor a
302 redirect rescued it.
- The 302-to-versioned-path fallback left a visibly "working" URL in
browser / curl -L contexts, which is exactly how the original bug got
reported in the first place ("the URL loads, why doesn't install work?").
Remove the endpoint and scrub all remaining references. The only CLI wheel
URL is now `/cli/wheel/{filename}` with the real PEP 427 filename, which
the setup-instructions template already generates server-side.
Existing tests that referenced /cli/agnes.whl become negative tests
("must not appear") so we don't regress.
* feat(cli): --version flag; sync --dry-run + progress indicator (#38)
* feat(cli): add --version / -V flag
Prints `da <version>` from package metadata (importlib.metadata). Falls
back to "unknown" when the package is not installed (e.g. running from a
source checkout without `uv pip install -e .`), instead of crashing.
Eager typer callback, so `da --version` exits before subcommand
resolution and does not require any auth/config.
* feat(cli): da sync --dry-run + X/N progress indicator
--dry-run reports what would be downloaded/uploaded without hitting the
API or writing local state. Supports the full flag set (--table, --json,
--upload-only); JSON shape is {"dry_run": true, "would_download": [...],
"summary": {...}}.
Progress bar now shows "[X/N] Downloading <table>..." with a Rich
BarColumn + TaskProgressColumn + TimeElapsedColumn instead of a bare
spinner — makes long syncs visible.
* feat(cli): durable sync + server gzip + auto-update check (#41)
* fix(sync): atomic writes + manifest hash verification + retry on transient errors
Three durability hooks around stream_download and the sync command:
1. Atomic writes. stream_download now streams into `<target>.tmp` and
calls os.replace() on success, so the real target file never exists
in a half-written state. On failure the tmp is unlinked — no cleanup
leftovers, no guard needed at read time.
2. Retry with backoff. Transient errors (ConnectError, ReadError,
WriteError, RemoteProtocolError, TimeoutException, 5xx) are retried
up to 3× with 0.3s / 1s / 3s backoff. 4xx (auth, 404) surfaces
immediately — retrying those is pointless.
3. Manifest-hash verification. After download, sync.py computes MD5 of
the target (same 8KiB chunking as app/api/sync.py:_file_hash) and
compares against `server_tables[tid]["hash"]`. Mismatch ⇒ unlink,
record error, skip state commit. The PAR1 structural check survives
as a fallback for legacy manifests without a hash.
Also makes _rebuild_duckdb_views tolerant: single broken parquet is
skipped with a stderr warning instead of killing the whole rebuild.
Supersedes #40 — this commit is a strict super-set (hash check + PAR1
fallback + atomic write + retry). #40 can be closed without merging.
* perf(server): enable GZipMiddleware for JSON / HTML responses
GZipMiddleware at minimum_size=1024 shaves bandwidth on manifest-style
JSON endpoints (/api/sync/manifest, /api/version, …) and the /install
HTML preview. Parquet file downloads are already columnar-compressed so
the middleware sees limited benefit there — but it doesn't hurt, httpx
on the client side decompresses transparently.
Placed after session middleware so gzip wraps the session-Set-Cookie
response too, and before CORSMiddleware so compression is applied to
both cross-origin and same-origin responses.
* feat(cli): auto-check for newer CLI version on startup
Server side
- GET /cli/latest returns {version, wheel_filename, download_url_path}
for whatever wheel is currently in AGNES_CLI_DIST_DIR. Public,
cacheable, no secrets — consumed by the CLI auto-update probe.
Client side
- New cli/update_check.py: reads /cli/latest with a 3s timeout, caches
the result in $DA_CONFIG_DIR/update_check.json for 24h. Cache is
invalidated when the installed version changes (e.g. after a fresh
`uv tool install`) so stale "you're behind" warnings don't linger.
- Root typer callback fires the probe before subcommand dispatch; any
failure is swallowed so a bad network never blocks a working command.
- Outdated → one-line stderr warning:
[update] da 2.0.0 is out of date — latest on this server is 2.1.0.
Upgrade: uv tool install --force <server>/cli/wheel/<…>.whl
- Disable with DA_NO_UPDATE_CHECK=1.
* fix(pr-review): None-guard the upgrade line + skip gzip on parquet paths
Two follow-ups from Devin review on #41.
1. format_outdated_notice(UpdateInfo(download_url=None)) emitted literal
"uv tool install --force None" — copy-pasting that fails. Drop the
upgrade snippet when the URL is absent and keep only the version line.
2. GZipMiddleware compressed everything over 1024 bytes, including the
parquet FileResponses served by /api/data/{tid}/download,
/cli/wheel/{name}, and /cli/download. Parquet is already columnar-
compressed — gzip there is pure CPU + latency with no size win, and
/api/data bodies can reach hundreds of MB. Wrap GZipMiddleware in a
small _SelectiveGZipMiddleware that skips those path prefixes and
delegates the rest to the stock middleware. JSON / HTML endpoints
(manifest, /install, /api/version, …) still get compressed.
* release: bump to 2.1.0 — unify AGNES_VERSION with pyproject.toml version (#42)
Before: two independent version systems. pyproject.toml carried semver
(2.0.0 → wheel filename → `da --version`) while release.yml injected
CalVer into AGNES_VERSION (e.g. 2026.04.155 → /api/version). Users saw
different strings in the CLI vs. the /install page, and the CLI auto-
update check couldn't tell "new deploy, same package version" apart
from "new package version".
Make pyproject.toml [project].version the single product-version source
of truth. release.yml extracts it and feeds AGNES_VERSION, so every
surface (/api/version, /api/health, /cli/latest, `da --version`) agrees
on one number. The CalVer tag keeps doing what CalVer is for: release
identity on the git tag and Docker image tag (versioned_tag).
Also wires AGNES_TAG through the build: release.yml → Dockerfile ARG →
env, so /api/version.image_tag finally reports the actual image tag
instead of the "unknown" fallback.
Bump to 2.1.0 to reflect the PRs shipped on ps/wheel-name-fix: durable
sync (atomic writes + manifest MD5 + retry), server GZip, CLI auto-
update probe, setup snippet PEP 427 URL.
* fix(pr-review): directional version compare in is_outdated()
UpdateInfo.is_outdated() used `self.latest != self.installed`, which
fires in both directions. If the server is rolled back or the user
connects to an older deployment, the CLI would warn "out of date"
and — worse — the formatted notice would prompt
uv tool install --force <older-version>.whl
i.e. an unintended downgrade.
Compare with packaging.version.Version (PEP 440 aware, handles pre-
release tags). Fall back to dotted-int tuple compare if packaging is
somehow missing, and return False on unparseable strings — better to
miss an upgrade hint than to silently suggest a downgrade.
Adds 4 test cases: installed older (True), installed newer (False),
10.0.0 vs 2.1.0 lexical-compare trap (correct), unparseable strings
(False).
Addresses Devin review on #43.
* fix(pr-review): read FastAPI app version from package metadata
app/main.py:80 hardcoded `version="2.0.0"` in the FastAPI constructor.
After #42 bumped pyproject.toml to 2.1.0, /api/version, /cli/latest,
and `da --version` all reported 2.1.0 while /openapi.json and the
/docs UI still advertised 2.0.0.
Read `agnes-the-ai-analyst` version via importlib.metadata (same
pattern cli/main.py:_cli_version already uses), with a `"dev"`
fallback when the package is not installed (source checkout). This
way pyproject.toml stays the single source of truth across every
version surface — /openapi.json now tracks the bump automatically.
Adds a dedicated test file to pin this behavior so a future
regression to a hardcoded literal fails at CI.
Addresses second Devin finding on #43.
* fix(pr-review): _fmt_bytes PiB label + negative cache in update_check
Two more follow-ups from Devin review on #43.
1. _fmt_bytes off-by-unit. The old loop exited at TiB but the fallback
labelled PiB, so 1 PiB rendered as "1024.0 PiB". Restructure: put
every unit inside the loop (KiB through EiB) so the division count
always matches the label. Covers up to 1 ZiB cleanly; anything
beyond renders as "<big>.0 EiB" rather than crashing.
2. Negative cache for failed /cli/latest probes. On a corporate
firewall / VPN that silently drops packets, the 3s HTTP timeout
fired on *every* `da` invocation. Writing a `latest=None` cache
entry with a 5-minute TTL caps that at one probe per 5min. Successful
probes still use the 24h TTL. Reading logic branches on whether the
cached `latest` is None.
Adds TestFmtBytes (2 cases: small/medium sizes and the PiB/EiB fallback
regression), plus two TestSync update-check cases covering negative-
cache reuse and TTL expiry.
Implements Task 4 — five Typer commands under `da metrics`:
- list/show use api_get() to query the server API
- import/export/validate access DuckDB directly via MetricRepository
and TableRegistryRepository (no server required)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ZdenekSrotyr
