26 commits

Author	SHA1	Message	Date
ZdenekSrotyr	64cf78860d	feat(stack): unified Browse + My Stack for Data Packages and Memory (v49 schema) (#333) ... * feat(unified-stack): Browse + My Stack + Recipes + RBAC matrix (v49–v55) Squash of 94 commits spanning the v49 → v55 unified-stack rewrite. Full per-feature breakdown lives in CHANGELOG.md under [Unreleased]. Major buckets: * v49 schema — first-class user_groups + user_group_members + resource_grants; admin can CRUD groups and grants; Google Workspace nightly sync writes into the new tables. * v49 data_packages — admin-curated bundles of tables, RBAC-gated, first-class section on /catalog Browse + My Stack. * v49 memory_domains — row-backed (replaces hardcoded VALID_DOMAINS enum); admin can CRUD; grants follow the same shape as tables and packages. * v50 cover_image_url + admin sidebar collapsibles + per-row Mode tooltip + admin queue domain badges + admin "+ New Item" seed flow. * v51 lifecycle status (prod/poc/coming-soon/draft) + category + palette swatches on admin modals. * v52 per-table detail page /catalog/t/<id>. * v53 Recipes — admin-curated SQL templates as a second tab on /catalog with full Edit/Delete admin affordances. * v54 soft-delete (deleted_at) + Undo toast for packages, memory domains, and recipes; hard_delete() retained as escape hatch. * v55 Recipes RBAC — ResourceType.RECIPE registered, inline Group Access matrix on Create + Edit Recipe modals (mirrors the Memory Domain pattern). * Activity Center per-resource filter (resource_prefix LIKE-anchored on audit_log.resource); admin nav g+letter keyboard shortcuts; loadAdminTablesLayout N+1 → single endpoint; /api/memory 30s page-level cache. * CI hardening — Keboola legacy tests pytest.importorskip; perf- smoke threshold widened to stop cold-cache flake. 5002 tests passing, 35 skipped. * feat(p2 backlog): Cmd-K palette + suggest-a-domain + nightly E2E + v55 schema 10-item P2 sweep on top of the unified-stack squash. New behaviour: * Cmd-K admin command palette (base.html) — fuzzy-search overlay over admin + user-facing routes. Arrows/Enter to navigate, Esc to close. * Stack-tabs digit shortcuts — 1/2/3 switch Browse / My Stack / Recipes on /catalog + /corporate-memory. * Friendlier non-admin empty state on /corporate-memory, plus a "Suggest a domain" CTA → POST /api/memory-domain-suggestions, admin queue with approve/reject. Backed by a new memory_domain_suggestions table (schema v55). * /admin/corporate-memory 7-tab strip grouped under Moderation / Catalog parent labels. * Bulk-assign table → package dropdown annotates each option with "(N of M tables already in)" so the existing distribution is visible before picking a target. * GET /api/memory + /tree accept is_required filter; admin status dropdowns route the "Required" sentinel onto it (status no longer holds 'mandatory' post-v49, so the old dropdown returned nothing). * chip-input.js is now opt-in per template via {% block extra_scripts %} instead of loaded globally on every page from base.html. * Edit-modal close helpers consolidated onto _closeEditModalById(); docs the per-source-type modal architecture decision. * New .github/workflows/e2e-nightly.yml runs agent-browser smoke scripts (scripts/e2e/smoke_*.sh) against a docker-compose stack nightly at 04:30 UTC; failures open an agent-browser-nightly issue. 5012 tests passing, 35 skipped. * fix(visual audit): 6 page regressions on memory + data-package surfaces agent-browser walkthrough of every memory + data-package page in the PR turned up 6 real bugs. Fixes: 1. Admin memory modals were dead. Duplicate `let _cmdNewDomainId` declarations from the deprecated step-2 RBAC stubs in admin_corporate_memory.html collided with the live state vars declared earlier in the same <script> → SyntaxError on parse → the entire second script block silently failed → every inline onclick= handler defined there (`+ New Memory Domain`, Edit, etc.) was a no-op. Removed the duplicate stubs. 2. /catalog/t/<table_id> + /catalog/r/<slug> rendered unstyled. Both templates injected their CSS via {% block head %} but base.html exposes {% block head_extra %} — wrong block name meant <style> rules never reached the rendered HTML. Renamed to head_extra. Hero card, section cards, dark SQL block, proper full-width inputs all now render as designed. 3. L49 leak — "MANDATORY" KPI label + "Make Mandatory" row buttons on /admin/corporate-memory still used the old word. Renamed to "Required" / "Mark as Required" so UI matches the data model (v49 split moved the Required tier onto the orthogonal is_required boolean; status no longer holds 'mandatory'). 4. Activity Center Resource dropdown didn't know the v55 `memory_domain_suggestion:` namespace — added it. 5. Tab strip on /admin/corporate-memory wrapped text 2× per button on narrow viewports after the L50 MODERATION/CATALOG group labels pushed total width past most viewports. Switched the strip to flex-wrap:nowrap + overflow-x:auto with white-space:nowrap + flex-shrink:0 on every direct child so the tabs stay one row and slide horizontally when they overflow. 5012 tests passing, 35 skipped. * rebase-cleanup: align with main's 0.54.25-27 API design + comment fix Three follow-on fixes after rebasing onto origin/main (0.54.27): * admin_tables.html: dropped a stray nested ``{% if data_source_type == 'keboola' %}`` around ``prefillFromKeboolaTable`` (main never had it; the outer Phase F2 guard already covers it) and reworded a JS comment that contained literal ``{% %}`` tokens which Jinja was parsing as a real tag → unbalanced if/endif → 30 template render failures across the suite. * /api/stack/subscription/{type}/{id}: DELETE now returns 204 instead of 200 per the 0.54.26 design rules. CLI client + parity tests updated to accept 2xx / assert 204. * Memory-domain suggestion approve/reject paths added to ``_VERB_PATH_ALLOWLIST`` — they are pending → approved/rejected state-machine transitions (approve also creates the real memory_domains row as a side effect), so the RPC shape is intentional rather than a missed PATCH refactor. 5035 tests passing, 35 skipped. * fix(catalog_table_detail): real polish pass — hero glyph, dedup pills, rows/size meta, scoped sync CTA The previous fix only got the block-name typo so the existing CSS rendered. The actual layout was still wireframe-tier on close inspection: * No cover glyph in the hero (a flat white card with title + meta line); data-package + memory-domain detail pages both have a colored icon square. Restored parity — table.icon emoji if set, otherwise initials on a colored square using table.color. * "INTERNAL" pill rendered twice for agnes_audit etc. — the mode pill and the source-type pill happened to be identical strings. Now skip the source pill when it matches the mode (`internal == internal`). * Bucket / source_table code chip showed `Agnes Internal.audit_log` for internal rows — meaningless to a user. Hidden when source_type is internal. * `pairs_well_with` admin input was a comma-separated `<input>` always visible. Wrapped all 4 sections in an Edit-on-demand toggle: read- only display by default, "+ Add" / "Edit" button on the right edge of each section header reveals the inline form, Cancel hides it. * "Trigger sync now" was a cramped link squashed into the empty-state flex row (visible as `Tr…` overflow before). Promoted to a proper btn-primary button under the empty-state copy. Hidden entirely for internal tables (which are server-managed — no upstream to pull). * Hero meta now surfaces row count + payload size (when sync_state has them) + last sync timestamp on a single line — was missing from the original. * Mode pills colored by tier (local=green, remote=amber, materialized= blue, internal=gray) so the basic fact about a table reads at a glance, not from upper-cased ALL-CAPS text alone. * tests(v56): TDD baseline for extended data-packages content + per-table docs 68 failing tests across 8 files spec the v56 surface before any implementation lands: * test_schema_v55_to_v56_migration.py — schema bump, additive ALTERs on data_packages + table_registry, idempotency, sequential-upgrade preservation * test_data_packages_repo_v56.py — repo create/update/get/list for owner_name, owner_team, tags, long_description, when_to_use, when_not_to_use, example_questions (JSON list round-trip, empty defaults, partial-update preservation) * test_table_registry_v56_docs.py — update_docs for grain, platforms, partition_col, history, gotchas; preserves v52 docs columns * test_api_data_packages_v56.py — PUT/POST/GET for all new fields, field-level validation (tag count, bullet length, description size), virtual badge derivation (curated/new) * test_api_registry_docs_v56.py — PATCH /api/admin/registry/{id}/docs for v56 fields, validation, RBAC unchanged * test_web_catalog_package_detail_v56.py — /catalog/p/<slug> rewrite asserts on rendered owner line, tag pills, badges, What it is, Use it when, Skip it when, Example questions, per-table extended detail in collapsible row, key-gotcha distinctness, admin-only Edit * test_web_stack_card_v56_metadata.py — Browse-grid card additions (owner chip, tag chips, badges) without breaking back-compat for rows missing the new fields * test_data_packages_no_vendor_content.py — CI guard: scans app/ + src/ + cli/ + config/ + scripts/ for Groupon-specific tokens from the colleague's spec MD; fails if any leak into OSS surfaces * test_db_schema_version.py — bumped 55 → 56 with rationale Plus updates schema-version assertion to 56. Implementation lands in subsequent commits (schema migration → repo → API → templates). * feat(v56): schema + repo for extended data-packages content Schema additions (ALTER ADD COLUMN IF NOT EXISTS — additive + idempotent): * data_packages: owner_name, owner_team, tags, long_description, when_to_use, when_not_to_use, example_questions (JSON-as-VARCHAR for the lists) * table_registry: grain, platforms, partition_col, history, gotchas (extends the v52 sample_questions / things_to_know / pairs_well_with docs surface with structured per-table content) Repo extensions: * DataPackagesRepository.create + update accept the new fields with the same Optional-is-no-op contract as v51 (pass an empty list to clear a JSON column) * _decode_row decodes the new JSON-list columns to Python lists; NULL rounds back to [] so callers don't branch * TableRegistryRepository.update_docs grew the v56 fields alongside the existing v52 ones — single PATCH can write either tier atomically * TableRegistryRepository._decode_row picks up platforms + gotchas in the same NULL-tolerant decoder 22 repo + migration tests passing. API + UI land in subsequent commits. * feat(v56): API surface for extended data-packages + per-table docs CreateDataPackageRequest + UpdateDataPackageRequest grew the v56 fields (owner_name, owner_team, tags, long_description, when_to_use, when_not_to_use, example_questions) with per-field validators that match the Foundry spec checklist: * tags: ≤8 entries × ≤30 chars * long_description: ≤4000 chars * use/skip: ≤8 bullets × ≤200 chars * example_questions: ≤12 × ≤200 chars _serialize emits all v56 fields plus a virtual ``badges`` list derived server-side at render time (no DB column needed): "curated" when the creator is in the Admin group, "new" within 30 days of created_at. Backdating created_at or admin-status changes pick up automatically. PATCH /api/admin/registry/{id}/docs extended with v56 structured per-table fields (grain, platforms, partition_col, history, gotchas). gotchas: list of {key: bool, body: str} Pydantic models with the same ≤8 cap; first key=true entry becomes the Key gotcha on the rendered package detail page. PATCH echoes the fresh state so callers can re-render without a second GET. 26 API tests passing (16 data-packages + 10 registry-docs). * feat(v56): /catalog/p/<slug> rewrite + Browse-grid card augmentation The third (and final) v56 commit lights up the UI surfaces backed by the schema + API commits earlier in this PR: * /catalog/p/<slug> template rebuilt around the Foundry spec's section ladder — hero (icon + name + badges + owner + tags + description + meta + Add-to-stack), "What it is" markdown body, paired "Use it when / Skip it when" panels, "Tables in this package" with collapsible per-table extended detail (grain / platforms / partition_col / history / gotchas + sample questions), and an "Example questions you can ask Claude" prompt panel. Each section guarded by ``{% if pkg.<field> %}`` — empty content fields hide the section entirely (no "No X yet" placeholder noise on the public-facing drilldown). * router catalog_package_detail hydrates per-table v56 fields onto the tables list + derives the virtual badges (curated / new) server-side from creator-in-Admin + 30-day created_at. * StackResolver.ResourceEntry grew owner_name / owner_team / tags / badges; _fetch_entries pulls the v56 columns + computes badges once per fetch using a single Admin-group SELECT. * _data_package_entry_dict adapter passes the new fields through to the macro; tags are merged source-type pills + admin-authored category tags per the spec convention. * _stack_card.html renders the v56 badges (top-left, data-badge= hooks) + the owner chip (data-card-owner hook) without breaking back-compat — pre-v56 rows render unchanged. * Admin PUT handler strips the v56 docs fields from the read-modify-write merged dict so register() doesn't blow up with the now-larger row shape (same pattern as the v52 docs fields stripping). 5115 tests passing (+98 v56 + 18 fixed regressions from the merged- register PUT path), 35 skipped. * fix(rbac): Edit-on-package + Group-access 'required' persistence + CI vendor guard Three related bugs reported on the merged-with-main branch: 1. Clicking Edit on a Data Package card landed on /admin/tables with a `#<pkg.id>` hash that nothing listened to — admin saw the global table listing, not the editor for that specific package. Added a `?edit_package=<pkg_id>` query-param handler in admin_tables.html (analog to the existing `?edit=<table_id>` and `?assign_to=<pkg_id>` patterns) that calls openEditDataPackageModal on DOMContentLoaded after a 250ms layout settle. Updated the package-detail Edit link to use the new query param. 2. Setting Group Access to 'required' didn't persist — re-opening the modal showed 'available'. Root cause was the v49 ``resource_grants.requirement`` enum existing in the DB but the POST /api/admin/grants endpoint not surfacing it: ``CreateGrantRequest`` declared only group_id + resource_type + resource_id, so Pydantic silently dropped the matrix's ``requirement: 'required'`` payload and the new row landed at the DB column default ('available'). Plumbed ``requirement`` through ``CreateGrantRequest`` → ``ResourceGrantsRepository.create`` so the value persists in one round-trip. Plus a UNIQUE-constraint race in the matrix diff-apply: DELETE-old + POST-new ran in parallel via ``Promise.allSettled``, so POST could fire first and trip the unique check before DELETE freed the slot. Switched to sequential (await all deletes; then await all writes) across all three matrices (Edit Data Package, Edit Memory Domain, Edit Recipe). 3. CI vendor-content guard ``test_no_groupon_specific_strings_in_oss`` tripped on two of my own docstrings: a "Foundry Data team" mention in two src/db.py comments + an ``s1_session_landings`` example in cli/skills/agnes-table-registration.md. Rephrased the comments to "extended-descriptions admin spec" and replaced the example with a generic ``events_daily`` table name. 5164 tests passing, 35 skipped (+4 regression tests pinning the POST /api/admin/grants requirement contract). Vendor guard back to green. * fix(catalog): admin Browse path drops v58 card fields The /catalog and /memory admin god-mode branch built ResourceEntry instances inline from pkg_repo.list() / domains_repo.list() and skipped owner_name, owner_team, tags, and derived badges (curated/new). Visible symptom: a package with an owner + tags rendered with the v56 chrome for non-admin viewers but as a bare card for admins. Adds StackResolver.browse_admin(user_id, resource_type) — admin god-mode Browse that walks the full table but routes through the same _fetch_entries enrichment pass as browse(), so admin + non-admin Browse stay visually consistent. Both /catalog and /corporate-memory routes switch to it. Regression test in tests/test_stack_resolver_browse_admin.py covers: owner/tags propagation, new/curated badge derivation, in_stack from admin subscriptions, all-packages-regardless-of-grants, and the ValueError for unsupported resource types. * fix(catalog): three /catalog tab-strip UX bugs 1. Required Remove → red toast browse_admin passed empty required_ids to _fetch_entries, so the admin's own required grants surfaced as 'available' and the macro rendered an actionable Remove button that POST /unsubscribe 400'd on. Now derives required_ids from the admin's own groups so Required packages render with the disabled "In stack (required)" button. Regression test in test_stack_resolver_browse_admin.py. 2. Remove green-toasts but card stays until refresh The My-Stack empty-state placeholder was only emitted server-side when stack_entries was empty at render time. Removing the last card left the tab completely blank — users read that as "Remove didn't work, let me refresh". Both grid + empty-state are now always rendered with one of them initially hidden; the JS swaps visibility on add/remove instead of injecting DOM. Same fix in /corporate-memory. 3. "What are Recipes?" + ambiguous (admin) suffix Recipes tab now carries its own curator-block explainer (the shared one was moved inside Browse view so it doesn't bleed across tabs). The grey "(admin)" suffix becomes a yellow .admin-only-hint chip with a title tooltip — visibility hint is now unambiguous: yellow chip = "only you see this", non-admins don't see the affordance at all. * schema: renumber v51..v58 → v52..v59 to make room for main's v51 Main 0.54.29 introduced a NEW v51 (table_registry.bq_fqn — issue #343) that releases ahead of this branch. The unified-stack chain v51..v58 shifts up by one so main's v51 stays as the released schema and ours become v52..v59. Function names, internal version bumps, dispatch ladder thresholds, and the migration-test references all move together. Subsequent merge with main lands the bq_fqn column at the freed v51 slot. * fix(seed): seed admin lands in BOTH Admin AND Everyone groups The LOCAL_DEV_MODE / SEED_ADMIN_EMAIL bootstrap only added the seed user to Admin. Everyone-scoped grants — the canonical "every-user- sees-this" pattern for Required onboarding — didn't surface for the seed admin's own /catalog because they weren't in Everyone. Symptom: admin grants a Required-tier package to Everyone, then sees it on /catalog still rendered with an "Add to stack" button (because the admin's resolved required_ids was empty for that package). The dual-membership keeps Admin (authorization) and Everyone (default-grant target) intentionally separate per the design comment on UserRepository.create — every membership remains traceable to a concrete row, just now with a system_seed row in Everyone too. Both INSERTs go through UserGroupMembersRepository.add_member which is idempotent on (user_id, group_id), so re-fires on every lifespan startup don't duplicate rows. Regression test in test_main_seed_admin_everyone.py. * style: unify admin-only hints across marketplace + memory detail pages Replaces three stale ``(admin)`` parentheticals with the same yellow ``admin-only`` chip introduced for /catalog tab actions. Same tooltip copy ("Visible only to admins — analysts won't see this …") so the visibility hint is unmistakable wherever it appears: - Hard delete on marketplace_plugin_detail (admin-only destructive action — same gating as the original suffix conveyed). - Hard delete on marketplace_item_detail (same). - Edit link on memory_domain_detail (title-attr only before; now a visible chip too). Non-admin viewers never saw these affordances — the gates are unchanged. Pure styling pass for consistency. * fix(catalog): exclude soft-deleted data packages + memory domains from Browse ``StackResolver._fetch_entries`` and ``browse_admin`` were querying data_packages / memory_domains without a ``deleted_at IS NULL`` guard. A package soft-deleted via /admin/* (v54 soft-delete contract) stayed visible on /catalog and /memory until either an Undo or a hard delete — directly contradicting the soft-delete UX which is supposed to remove the affordance immediately and only retain the row for the Undo window. The repository accessors (DataPackagesRepository.list, MemoryDomainsRepository.list, list_packages_of_table, etc.) already filter deleted rows; this commit brings the resolver's direct SQL in line with that contract. Regression test in test_stack_resolver_browse_admin.py. * fix(catalog): Add/Remove updates full card chrome, not just button The previous _applyStackChange flipped only the footer button label — the card border (.is-in-stack class), top-right "In stack" badge, and button color class (--add / --remove) stayed at their server-rendered state. After Add the user saw the button checkmark but the rest of the card still looked like "available, not in stack". They read this as "the change didn't take — let me refresh". This commit makes the optimistic update mirror what the server-side macro renders for the new state: * ``c.classList.toggle('is-in-stack', becameInStack)`` — flips the border + visual state class. * Top-right ``.stack-card__req-badge--instack`` badge is injected on Add, removed on Remove (skipped when ``data-requirement='required'`` — that slot is owned by the Required badge). * Button text is "Remove" / "+ Add to stack" matching the macro (was "✓ In stack" which was visually nice but inconsistent). * Button color class --add / --remove swaps so the destructive Remove tint kicks in immediately. The clone-into-My-Stack path applies the same updates so the new card in My Stack reads identically to a server-rendered in_stack card. Mirrored in /corporate-memory. * fix(memory): four Devin-review bugs on /memory drill-down + manifest PR #333 Devin review surfaced four real bugs that ship a broken /memory experience even though the unit tests passed. 1. Manifest md5 omits is_required + content (app/api/sync.py:836-840) _build_memory_domains_section hashed only (id|title|status) per item. _build_per_domain_markdown routes items between "## Required" and "## Approved" by is_required and embeds full content — so an admin edit of either dimension left the manifest md5 unchanged, `agnes pull` skipped the re-fetch, and the analyst kept a stale bundle.md. Now both fields participate in the hash. 2. required_count always 0 (src/repositories/memory_domains.py) list_items_of_domain only SELECTed (id, title, status) so the `it.get("is_required")` in the manifest builder always evaluated to None → required_count = 0 regardless of actual state. The manifest builder advertised a count it could never compute. Now projects is_required + content too (required by fix 1 anyway). 3. Vote URL 404 (memory_domain_detail.html:289-290) Constructed `/api/memory/items/{id}/vote` but the route is `/api/memory/{id}/vote`. Every upvote/downvote button was a silent no-op. 4. Dismiss/undismiss URL + method both wrong (memory_domain_detail.html:296-305) Constructed `/api/memory/items/{id}/dismiss` (extra /items/) and /undismiss (no such route — undismiss is DELETE on /dismiss). Both buttons silently 404'd. Now POST + DELETE on `/api/memory/{id}/dismiss` per app/api/memory.py:635/675. * fix: multi-agent reviewer findings — vendor-token scrubs + manifest md5 predicate + soft-delete filter Three reviewer findings from the multi-agent review on PR #333, fixed in-place per CLAUDE.md issue-economy rule. Reviewer-rules (Important — vendor-agnostic OSS): - app/main.py:218 comment: replaced 'foundryai-prod' with generic 'a customer prod instance' phrasing. Public OSS repo must not carry customer-specific tokens (CLAUDE.md § Project conventions). - tests/test_table_registry_v56_docs.py:70 fixture string: replaced "user_brand_affiliation = 'groupon'" with 'acme' on the same rule. Reviewer-architecture (closes still-unresolved Devin 🚩 ANALYSIS): - app/api/sync.py _build_memory_domains_section: md5 hash loop now filters items to the SAME predicate the bundle renderer uses (is_required OR status='approved'). Pre-fix the hash iterated ALL items but _build_per_domain_markdown only rendered the union of required items + approved-non-required items — so an admin edit to a pending/rejected non-required item flipped the md5 against an identical-bytes bundle, triggering a wasteful re-fetch on every analyst's next 'agnes pull'. The earlier commit fixed the hash-input fields (is_required + content); this closes the set-of-items asymmetry Devin separately flagged. Reviewer-RBAC (minor cleanup): - app/resource_types.py _data_package_blocks and _memory_domain_blocks now filter 'WHERE deleted_at IS NULL' (v54 soft-delete column) so the /admin/access UI doesn't surface soft-deleted entities as grantable. Mirrors the existing filter on _recipe_blocks. No security leak pre-fix (resolver double-filters and re-checks at serve time), just UI cleanliness. - app/services/stack_resolver.py add_to_stack: docstring note added explaining that authorization is enforced at the API layer (app/api/stack.py can_access gate), not at the resolver. The initial review suggested adding a defensive 403 here, but that broke 5 existing tests that legitimately call add_to_stack directly without setting up grants first; the docstring captures the contract instead. stack() already intersects subscriptions with current available_ids on every read, so a 'zombie' row from a misuse never leaks into the user-facing manifest. * release: 0.55.0 — unified Browse + My Stack (Data Packages + Memory), schema v48→v59, 3 BREAKING	2026-05-19 15:00:15 +02:00
ZdenekSrotyr	c3e82972c8	feat(bq): decouple table_registry bucket from BQ dataset name (#343) (#346) ... * feat(bq): decouple table_registry bucket from BQ dataset name (#343) Adds optional `bq_fqn` column (schema v51) carrying the fully-qualified BigQuery path (project.dataset.table) so the rebuild path no longer has to reconstruct it from the dual-purpose `bucket` field (which is also a UX/RBAC label). - Schema v51 migration + _SYSTEM_SCHEMA carry the nullable column; rows without it keep using the legacy bucket+source_table+ remote_attach.project path (backwards compat). - BQ extractor honors bq_fqn per row when present: dataset/table override on same-project rows; cross-project VIEW path works via bigquery_query(billing, ...); cross-project BASE TABLE skipped with a clear warning (multi-ATTACH per project deferred to follow-up). - Orchestrator pre-pass detects drift between extract.duckdb _remote_attach.url and overlay data_source.bigquery.project, calls rebuild_from_registry to regenerate when they differ. Closes the operational hazard where /admin/server-config edits silently left the on-disk extract pointing at the old project until the next manual sync. - Startup config check warns when project ≠ billing_project without location set (the on-disk symptom is "provider returned no data" silently in metadata cache), and when a warehouse-like data project has no billing_project override (silent 403 serviceusage path). - _resolve_bq_location warning now points at the location config key explicitly so operators see the actionable fix in the log. - POST /api/admin/register-table and PUT /api/admin/registry/{id} accept bq_fqn; malformed values rejected at the API boundary (422). - 25 tests covering parse_bq_fqn matrix, extractor override paths (same-project + cross-project VIEW + cross-project BASE TABLE skip), orchestrator drift sync, startup-validator heuristic, admin models. UI surface for bq_fqn input in /admin/tables intentionally omitted from this PR (3.5k-line template change) — admins can register through the REST API or `agnes admin` CLI in the meantime. Multi-project ATTACH support is the same scope deferral as the cross-project BASE TABLE skip; both ride a follow-up PR. * review fixes: abstract CHANGELOG, merge duplicate Changed, bump docs schema version - CHANGELOG.md: remove customer-specific hostname + incident date range from the orchestrator drift-sync entry (vendor-agnostic OSS rule), fold the entry into the existing [Unreleased] ### Changed section instead of opening a duplicate heading. - docs/architecture.md: bump 'Current schema version' from 19 to 51 to match SCHEMA_VERSION (per agnes-orchestrator skill rule #4). * review fixes: vendor-agnostic test fixture + Schema v51 internal bullet - tests/test_bq_fqn.py: replace customer GCP project ID with generic 'my-warehouse-project' placeholder (vendor-agnostic OSS rule). Test asserts on the warehouse-like heuristic, not the literal project name, so the rename is behavior-neutral. - CHANGELOG.md: add explicit '\*\*Schema v51\*\*' bullet under `### Internal` naming the new version + summarizing the additive nullable column (matches the convention from v47/v48 bullets). * fix(bq): cross-project _detect_table_type bills against extractor project Addresses Devin review on #346 — pre-fix _detect_table_type passed the data project as BOTH the FROM-clause target AND the bigquery_query() first arg (billing project). For cross-project bq_fqn rows where fqn_project != project_id, the data SA holds bigquery.dataViewer on fqn_project but the serviceusage.services.use permission only on project_id, so the call 403'd. init_extract's broad except Exception swallowed the error and silently skipped the row, meaning the cross-project VIEW path at extractor.py:~696 — the PR's primary cross-project use case — never executed. - Add optional billing_project kwarg to _detect_table_type; defaults to project for backwards compat (same-project callers unaffected). - Update the init_extract call site to pass billing_project=project_id explicitly. Same-project rows (fqn_project == project_id) are a no-op; cross-project rows now route billing to the project where the SA actually has services.use. - 2 new tests in TestDetectTableTypeBilling cover (a) explicit billing_project routing to bigquery_query 1st arg + data project staying in FROM, and (b) the backwards-compat default. Plus test_cross_project_detect_call_bills_against_extractor_project pins the call-site wiring — captures the (project, billing_project) pair the extractor passes for a cross-project bq_fqn row. * release: 0.54.29 — bq_fqn decoupling + marketplace refactor + setup-script UX Accumulated [Unreleased] content from #342 (flea marketplace refactor), #344 (setup script step-2 cwd check), and #346 (this PR — bq_fqn column + orchestrator drift sync + startup config check). Schema v51.	2026-05-19 11:17:32 +00:00
minasarustamyan	c6c72b9c00	feat(flea): marketplace refactor — data model, attribution, UI unification (#342) ... * feat(flea): phase-1 — title, tagline, synthetic_name columns + upload UX Schema v49 adds three user-facing metadata columns to store_entities: - title (NOT NULL) — humanized display name shown on marketplace surfaces in later phases. Acronym-aware humanizer in src/store_naming.py (27 entries: MCP, API, OAuth, S3, …) shared with the frontend via Jinja-injected dict so JS pre-fill and Python backfill produce identical output. - tagline (NULL, ≤200 chars) — optional short description for card listings. Long-form `description` stays. - synthetic_name (NOT NULL) — deterministic `<name>-by-<owner_username>` stored as a column for indexing and as the single source of truth for attribution lookups in later phases. Today's bundle bake still uses suffixed_name() at the same call sites. Migration (_v48_to_v49_migrate, Python function — humanize has no SQL equivalent) backfills existing rows: title from humanize_name(strip_archive_suffix(name)), synthetic from the concat formula; tagline stays NULL. Idempotent (ADD COLUMN IF NOT EXISTS + SET NOT NULL no-op on re-run). Upload form (store_upload.html step 2) reorders fields: Title (pre-filled from server-side humanize, JS keeps it in sync until the user edits manually) → Name + dark synthetic preview on one row (matches marketplace_item_detail.html dark code styling, no copy button — preview only) → Short description with character counter → Description (unchanged). Edit form (store_edit.html) mirrors the layout with pre-filled values from the entity row. API: - POST /api/store/entities/preview returns `title` (humanized fallback) for upload form pre-fill. - POST + PUT /api/store/entities accept `title` and `tagline` form fields with 100/200-char validation; PUT recomputes synthetic_name when `name` changes (caller responsibility per repo contract). - StoreEntityResponse exposes all three new fields. Repository: - create() takes title + tagline + synthetic_name as optional kwargs with derived defaults (humanize_name(name) / concat) so existing test fixtures don't need to thread them. - update() supports partial updates on all three; tagline empty string clears via NULL sentinel. - archive() recomputes synthetic_name on rename to the archived slug so the column stays consistent with name. Tests: - New test_schema_v48_to_v49_migration.py: fresh install, populated-row backfill (incl. archived row strip), idempotence, NOT NULL constraint verification. - test_store_naming.py: 14 humanize parametrize cases + acronym dict invariants. - test_store_api.py::TestStoreV49Metadata: preview humanize, POST with explicit + fallback title, 100/200-char rejects, PUT partial update + synthetic recompute on rename. - Schema version assertion bumps (48 → 49) in test_db_schema_version, test_home_stats, test_schema_v42_migration, test_schema_v46_migration. Phase 1 only — surface rendering on cards / detail pages and Claude Code bundle propagation come in later phases. * feat(flea): phase-2 — wire title/tagline/owner through marketplace cards + detail pages Phase 1 (7f4cfcbb) populated the three new columns on store_entities; phase 2 surfaces them across the web presentation layer so the kebab- case slug + bare username no longer leak into user-facing copy. API: - `_flea_to_item` now takes `conn` (both callsites updated) and sets `display_name=entity.title`, `tagline=entity.tagline`, `owner= _resolve_owner_display(conn, owner_user_id, owner_username)` — matches the chain the curated path already uses (users.name → users.email → fallback). The card JS chain `it.display_name || it.name` then renders the friendly form; `name` stays at the suffixed slug as the technical identifier JS uses for fallbacks. - `flea_detail` adds `display_name` + `tagline` to PluginDetailResponse so the standalone skill/agent + plugin detail heroes pick them up through the existing `d.display_name` / `d.tagline` chains. - `_flea_inner_parent_fields` swaps `parent_display_name` from `strip_archive_suffix(name)` to `entity.title or strip_archive_suffix( name)`. Drives parent-plugin label in four surfaces at once: breadcrumb 3rd segment, hero "part of <plugin>" meta-row, helper "This skill is part of <plugin>" panel, and the Details sidebar's "Parent plugin" row. Templates — `marketplace_item_detail.html`: - Pre-render: browser title, hero h1, and hero-window-label read `(entity.title if entity else None) or inner_name or item_name or plugin_name` so the SSR shell shows the friendly title before the JS fetch lands (no flash of kebab-case). - Breadcrumb last segment for flea standalone drops the `d.manifest_name || heroTitle` fallback in favour of just `heroTitle` — manifest_name is the suffixed slug and users explicitly didn't want it in the path. - Hero meta-row for flea standalone is now hidden. The prior "by <author> · N installed · <size>" line duplicated install count (hero telemetry chip below), owner + bundle size (Details sidebar). Templates — `marketplace_plugin_detail.html`: - Same SSR pre-render swap (title, h1, window-label, crumb-name). - Hero tagline element starts hidden; JS shows it only when `d.tagline` is truthy. Pre-fix it fell back to `d.description` (long-form text), which read awkwardly under the h1 and pulled the hero too tall. Description still renders in the "What it does" panel below the hero. - Initial "Loading…" placeholder removed so entities without a tagline don't flash that text mid-fetch. Tests: - New `TestFleaPhase2Presentation` class in test_marketplace_api.py (6 cases): card title + tagline + full-name owner, owner fallback chain when users.name is NULL, flea_detail exposes title + tagline, tagline null when omitted, inner skill parent_display_name uses entity.title (explicit + humanize-fallback variants). - Updated `TestListItems.test_flea_lists_uploads` to assert both `display_name == "Alpha"` (humanized) and `name == "alpha-by-alice"` (suffixed slug compat). - Updated `TestWebPages.test_marketplace_flea_detail_page_renders` to look for the humanized title ("Page Skill") in the SSR shell instead of the kebab-case `page-skill`. * feat(flea): phase-3 — read synthetic_name from DB, suffixed_name() only on write Phase 1 added the column + backfill, repo write paths keep it in sync. Phase 3 routes every READ callsite through `store_entities.synthetic_name` directly instead of recomputing `<name>-by-<owner_username>` on the fly, and switches the collision query off the inline string concat. The `suffixed_name()` primitive now lives exclusively in write flows. Read callsites updated (all read `entity["synthetic_name"]` directly, no fallback — the column is NOT NULL and a missing value would be a real bug worth surfacing as KeyError): - app/api/marketplace.py:_flea_to_item — card MarketplaceItem.name. - app/api/marketplace.py:flea_detail — PluginDetailResponse.manifest_name. - app/api/store.py:_entity_to_response — StoreEntityResponse.invocation_name. - app/api/store.py PUT bundle re-bake — `suffixed` passed to `_bake_plugin_tree`; entity is loaded pre-rename, so its synthetic_name is the OLD value `_bake_plugin_tree` expects. - app/api/store.py PUT rename — `old_suffix` for `_rename_baked_tree`. - app/api/my_stack.py — StoreInstallEntry.invocation_name. - src/marketplace_filter.py — manifest_name in served plugin entry. `suffixed_name` imports removed from marketplace.py, my_stack.py, and marketplace_filter.py (no remaining callsites). store.py keeps the import for its write paths: - POST create (`suffixed = suffixed_name(final_name, username)` → passed to `_bake_plugin_tree` and `repo.create(synthetic_name=...)`). - PUT rename collision check (`new_suffixed`). - PUT rename `new_suffix` for `_rename_baked_tree` (proposed value). - PUT rename `new_synthetic` for `repo.update(synthetic_name=...)`. - Archive `old_suffix` + `new_suffix` for `_rename_baked_tree` (retro-compute pre-archive value after `repo.archive` already overwrote the DB row with the post-archive synthetic). Collision SQL — `_suffixed_already_taken`: WHERE name || '-by-' || owner_username = ? (before) WHERE synthetic_name = ? (after) Same matches today (phase 1 backfill + NOT NULL invariant + write paths in sync); indexable + single source of truth going forward. Repository: - UserStoreInstallsRepository.list_for_user explicit SELECT extended with `se.title`, `se.tagline`, `se.synthetic_name` so my_stack and marketplace_filter callers can read them off the joined row. Tests: - test_store_api.py::test_invocation_name_reads_from_synthetic_column — upload entity, manually override the column with a non-canonical value, verify GET response returns the override (proves read path consumes the column, not recomputes). - test_marketplace_api.py::test_flea_card_and_detail_read_synthetic_name_from_db — same proof for `MarketplaceItem.name` (card) and `PluginDetailResponse.manifest_name` (detail). * feat(flea): phase-4 — rename agnes-store-bundle → flea (synthetic plugin) The synthetic plugin that wraps loose flea-market skills + agents into one Claude Code plugin is renamed from `agnes-store-bundle` to `flea`. Plugin-type flea uploads (their own standalone plugin entry) are unaffected. Constants: - src/marketplace_filter.py: - BUNDLE_PLUGIN_NAME: "agnes-store-bundle" → "flea" (Claude Code plugin manifest name + .claude-plugin/plugin.json name) - BUNDLE_PREFIXED_NAME: "store-bundle" → "flea" (on-disk ZIP / git tree path, now plugins/flea/...) Attribution layer (services/session_processors/usage_lib.py): - FLEA_BUNDLE_PREFIX: "agnes-store-bundle" → "flea". The JSONL invocation identifier going forward is `flea:<skill-name>`. - New `_LEGACY_FLEA_BUNDLE_PREFIXES = ("agnes-store-bundle",)`. `MarketplaceItemLookup.resolve()` + `_attribute_event()` accept BOTH the new and the legacy prefix so historic usage_events (~90-day retention) continue attributing to source='flea'. The tuple becomes a no-op once the rename has been live past the retention window — a follow-up commit can drop it then. - USAGE_PROCESSOR_VERSION bumped 6 → 7 so the session-pipeline reprocess loop re-runs attribution with the new + legacy prefix branches. User-facing copy: - /api/store/bundle.zip Content-Disposition filename: agnes-store-bundle.zip → flea.zip - `agnes admin store pull` default --out: agnes-store-bundle.zip → flea.zip - Docstrings + JS comment + welcome template comment updated. Tests: - skill_flea.jsonl fixture identifier updated to flea:flea-skill. - New skill_flea_legacy.jsonl with the legacy prefix for backward-compat coverage. - New test `test_legacy_agnes_store_bundle_prefix_resolves` replays the legacy fixture and asserts source='flea' attribution still lands. - All other test assertions / mocks substituted mechanically: test_session_processor_usage.py, test_usage_rollups.py, test_marketplace_filter_store.py, test_store_api.py, test_cli_refresh_marketplace.py. - `_seed_flea_entity` (test_usage_rollups.py) + `_seed_attribution` (test_session_processor_usage.py) helpers now supply the NOT NULL `title` + `synthetic_name` columns from phase 1, since they INSERT directly bypassing the repo's create() fallback. Client rollover note (CHANGELOG): `agnes refresh-marketplace` will install the new `flea@agnes` plugin and the local marketplace clone's `plugins/store-bundle/` source folder is removed via `git reset --hard`. Whether Claude Code itself auto-prunes the orphan `agnes-store-bundle @agnes` registry entry is undocumented — to verify empirically on the dev VM. If the orphan entry lingers, a follow-up will add targeted cleanup; until then users can manually run `claude plugin uninstall agnes-store-bundle@agnes`. Verified locally: 98 passed (session_processor_usage + usage_rollups + marketplace_filter_store + cli_refresh_marketplace) + 228 passed/2 skipped (store_api + marketplace_api + admin_store_submissions + store_entity_versions + store_repositories). * fix(flea): phase-5 — attribution keyspace mismatch (closes #335) Pre-fix every flea skill/agent invocation silently fell through to `usage_events.source = 'builtin'`. Root cause: lookup tables in `services/session_processors/usage_lib.py` keyed `_flea_entities` (and the derived `_flea_plugins` set) by `store_entities.name` — the un-suffixed display name. Claude Code writes invocations as `flea:<synthetic_name>` (e.g. `flea:xlsx-by-c-marustamyan`), so `dict.get(local)` always missed and the resolver fell through to builtin. Result: marketplace cards, detail telemetry chips, admin group-by-source all showed 0 flea invocations even when the raw JSONL stream was correct. Phase 1 added the `synthetic_name` column + backfill; phase 4 renamed the bundle prefix to `flea`; phase 5 finally flips the lookup keyspace to match what JSONL writes. usage_lib.py: - `MarketplaceItemLookup.__init__` preload: `SELECT synthetic_name, type FROM store_entities` (was `SELECT name, type`). `_flea_plugins` set derived from those keys, so it now carries synthetic_names too — matches what Claude Code writes when invoking a skill nested inside a flea plugin (`<synthetic>:<inner>`). - `rebuild_rollups` preload: same SELECT change; also derives `flea_plugins` and threads it through `_aggregate_events` / `_rebuild_window`. - `_attribute_event`: signature extended with `flea_plugins`; new branch `if prefix in flea_plugins: return ("flea", default_type, prefix, local)` for flea-plugin-nested skills/agents. This branch was added to `MarketplaceItemLookup.resolve()` in v6 (commit e076ebbe) but the rollup builder's helper was never updated to match, so nested skills inside flea plugins silently dropped out of the daily/window fact tables. - `USAGE_PROCESSOR_VERSION`: 7 → 8. Forces the session-pipeline reprocess loop to re-attribute existing usage_events rows with the corrected lookup so rollup tables fill correctly on the next tick. marketplace.py — 4 API stats lookup callsites switched from `entity["name"]` to `entity["synthetic_name"]`: - `_flea_to_item` (card stats lookup) - `flea_detail` (`_build_telemetry` + `_load_inner_items_stats_by_parent`) - `flea_skill_detail` (inner detail `parent_plugin` key) - `flea_agent_detail` (inner detail `parent_plugin` key) Tests: - `skill_flea.jsonl` invocation: `flea:flea-skill` → `flea:flea-skill-by-alice` (mirrors what Claude Code writes after phase 1/4 — the suffixed synthetic_name). - `test_flea_skill_attributed_with_empty_parent` assertion: rollup `name` column now carries the synthetic_name. No legacy `agnes-store-bundle` prefix backward compat — clean cut per user direction (dev phase, no production data worth preserving). Verified locally: 53 passed targeted (session_processor_usage + usage_rollups + marketplace_filter_store) + 215 passed/2 skipped broader (store_api + marketplace_api + admin_store_submissions + store_entity_versions). * fix(flea): phase-6 — plugin-level rollup aggregation parity for flea Flea plugin entity cards + detail pages showed 0 invocations even though nested skills had correct rollup rows. Root cause: the plugin-level aggregation pass in `_aggregate_events` was hardcoded to `source='curated'` only: if source != "curated" or not parent: continue if group_by_day: pkey = (day, "curated", "plugin", "", parent) else: pkey = ("curated", "plugin", "", parent) So flea plugin entities never got a synthetic `(source='flea', type='plugin', parent_plugin='', name=<synth>)` row aggregating nested invocations. `_load_invocation_stats('flea')` filters `parent_plugin = ''` and returned no row for flea plugin entity cards, so `stats.get(entity["synthetic_name"])` missed and the API exposed 0/0. Triggered by empirical observation on the dev VM — `codex-second-opinion-by-c-marustamyan` plugin showed 0 calls in the listing card while its three inner skills (codex-setup ×3, codex-review ×1, codex-second-opinion ×1) had the expected child rollup rows. Fix: - Extend the guard to `source in ("curated", "flea")`. - Replace the hardcoded `"curated"` in the `pkey` tuple with the loop's `source` variable, so flea aggregation lands as `source= 'flea'` and curated aggregation continues landing as `source='curated'`. API path unchanged — `_load_invocation_stats('flea')` filters `parent_plugin = ''` already picks up the new aggregated row alongside standalone skill/agent rows. Rollup `name` field carries the synthetic_name keyspace; no collision between standalone entity synthetic and plugin entity synthetic (global suffix uniqueness enforced by `_suffixed_already_taken`). `USAGE_PROCESSOR_VERSION` bumped 8 → 9 to force a reprocess pass so historic nested-invocation data fills the new plugin-level rows on the next tick (instead of waiting for the next live invocation). Tests: - New `test_flea_plugin_row_aggregates_children` mirrors the existing `test_curated_plugin_row_aggregates_children`: seeds a flea plugin entity, three nested events (one user invoking two skills, a second user invoking one) → asserts the aggregated plugin row carries count=3, distinct_users=2 (union, not sum), plus the child rows survive alongside. Verified locally: 43 passed (session_processor_usage + usage_rollups) + 82 passed/2 skipped broader (+ marketplace_filter_store + marketplace_api). * refactor(marketplace): phase-7 — unify Details sidebar across detail surfaces Five marketplace detail surfaces (curated plugin, flea plugin, curated inner skill/agent, flea inner skill/agent, flea standalone skill/agent) had drifted on which Details rows they show and what order — the same field landed in different positions, some fields duplicated hero info, and the flea plugin Owner row leaked the kebab-case `owner_username` slug instead of the user's real name. This commit aligns all five surfaces on a single scan order driven by UX priority: identity → life-stage → telemetry → debug-tier Concretely: 1. Curator / Owner (first scan signal — trust) 2. Parent plugin (inner skill/agent only) 3. Released (top-level only — plugins + flea standalone) 4. Last used (recency) 5. Active days (engagement consistency) 6. Version (flea standalone only — content hash) 7. Bundle size (debug-tier) Dropped: - Slug field on plugin detail surfaces (`marketplace_id` for curated, `entity_id` for flea). Pure debug info, never user-relevant; URL already carries it. - Category + Installs on flea standalone skill/agent detail. Category is already shown as a hero badge; install count is in the hero telemetry chip — sidebar duplication added noise. Owner display: - Flea plugin Owner row now reads `d.owner_display` (resolved through `users.name → users.email → owner_username` by `_resolve_owner_display` in `app/api/marketplace.py:1491`) instead of the raw `d.author_name` (which is `owner_username`, the kebab-case slug). API field already populated from phase 2; templates just consume it. - Curated Curator row continues to read `d.author_name` from marketplace-metadata.json; `owner_todo` placeholder behavior preserved. Files: - app/web/templates/marketplace_plugin_detail.html — rewrote the Details render loop (lines 1364-1427 area). Slug row removed, rows reordered, Owner branch reads `d.owner_display`. - app/web/templates/marketplace_item_detail.html — both branches of the Details sidebar (inner skill/agent + flea standalone) re-laid around the same scan order. Telemetry helper unchanged, just repositioned. Category + Installs rows removed from the standalone branch. No new tests — no existing test asserts the precise order of Details rows or references the dropped fields in a sidebar context (grep confirmed). API surface unchanged. Verified locally: 84 passed / 2 skipped on `test_marketplace_api.py` + `test_store_api.py`. * fix(flea): post-review hardening — N+1, v50 UNIQUE, docs, test cleanup Addresses 5 critical findings from PR #342 code review: 1. N+1 query in `_flea_to_item` — owner-display resolution previously ran one `SELECT … FROM users WHERE id = ?` per item in the listing comprehension. Now batched via `_load_users_display` IN-query prefetch; 50 items drops 51 user queries to 2. Regression-guarded by `TestFleaOwnerDisplayBatched` (spies `_resolve_owner_display` and asserts it's not called inside the list path). 2. Misleading comment in `src/marketplace_filter.py` claimed the attribution layer accepts both `agnes-store-bundle` and `flea` prefixes — it doesn't (clean cut per CHANGELOG). Rewrote to match reality. 3. CHANGELOG `[Unreleased]` had two `### Changed` blocks. Merged into one (BREAKING bullet first). 4. New v49→v50 migration adds `UNIQUE INDEX idx_store_entities_synthetic_name`. v49 made `synthetic_name` the canonical attribution key but uniqueness was only app-enforced; v50 promotes the invariant to the DB layer. Migration pre-checks for existing duplicates and raises `RuntimeError` listing them rather than letting `CREATE UNIQUE INDEX` fail mid-way. v48→v49 migration gained an `is_nullable='YES'` guard on its `SET NOT NULL` ALTERs so re-runs on a fully-migrated DB don't trip DuckDB's "cannot alter entry … entries depend on it" block (the new index counts as such an entry). Index is created by the migration only — keeping it out of `_SYSTEM_SCHEMA` preserves fresh-install ordering (CREATE TABLE → v49 ALTERs → v50 CREATE INDEX). 5. Deleted three redundant version-pinned schema asserts whose names lied about their bodies (`test_schema_version_is_42` asserting `== 49`, etc.). Canonical assert lives in `test_db_schema_version.py`, renamed to `test_schema_version_matches_constant`. * fix(db): gate v34→v38 store_entities ALTER COLUMN steps on column state CI on Linux failed `test_v17_to_v18_drops_*` after the v50 UNIQUE INDEX landed. Root cause: those tests open a DB at the full target version, seed fixtures, then reset `schema_version` to 17 and reopen — forcing the ladder to re-run from 17 → current. With the v50 index now in place, DuckDB blocks intermediate `ALTER COLUMN` steps on `store_entities` ("Cannot drop this column: an index depends on a column after it!" / "Cannot alter entry because there are entries that depend on it"), because `synthetic_name` (the indexed column) sits positionally after the columns those steps touch. Fix: convert the three SQL-list migrations that hit store_entities into defensive Python functions: - `_v34_to_v35_migrate` short-circuits when `synthetic_name` already exists (post-v49 shape — the visibility_status rebuild is moot and the DROP COLUMN would be blocked by the index). - `_v35_to_v36_migrate` gates the `visibility_status SET NOT NULL` + `SET DEFAULT` on `is_nullable='YES'` so it's a true no-op when the column is already constrained. - `_v37_to_v38_migrate` gates the `version_no SET NOT NULL` step the same way. Forward-roll path (real installs that never reset schema_version) is unchanged: the gates fire `YES` → ALTERs run. The fix only changes behavior for the "DB is already at v50 shape but version row says 17" scenario the tests construct. --------- Co-authored-by: Minas Arustamyan <arustamyan.minas@gmail.com>	2026-05-19 02:32:41 +02:00
minasarustamyan	302cf58ccd	feat(marketplace): telemetry v46 + flea inner parity + listing polish (#329) ... * feat(telemetry): marketplace item rollup refactor (schema v46) Replace the v42 attribution layer with prefix-split + live lookup against marketplace_plugins / store_entities. The v42 design had a latent bug — AttributionLookup keyed on bare skill names while Claude Code writes `<plugin>:<local>` in JSONL, so lookups never matched and usage_plugin_daily stayed empty in every deployment. Schema (v46 migration): - Drop usage_attribution_skills / _agents / _commands (mapping tables, derivable from marketplace_plugins + plugin tree). - Drop usage_plugin_daily (always empty in production due to the bug above). - Create usage_marketplace_item_daily — per-day fact (count, distinct_users, error_count), composite PK on (day, source, type, parent_plugin, name). - Create usage_marketplace_item_window — sliding-window snapshot with true cross-window distinct user counts; period_label='last_7d' refreshes every tick, 'last_30d' refreshes hourly (tracked via session_processor_state). - Mark usage_tool_daily as candidate for removal (no product-UI consumer). Attribution flow: - MarketplaceItemLookup replaces AttributionLookup. Preloads marketplace_plugins.name + store_entities.name into memory once per UsageProcessor tick, then per-event splits identifier on ':', matches prefix, writes resolved source / parent_plugin into usage_events. agnes-store-bundle prefix routes to flea entities. Slash commands with `plugin:` prefix count as type='skill' in rollup. API: - BREAKING: MarketplaceItem.unique_users_30d renamed to distinct_users_30d (now a true distinct count from the window snapshot, not sum-of-daily). - InnerDetailResponse gains a telemetry field — invocations_30d + distinct_users_30d surfaced on curated inner skill / agent detail pages. - Card chip hidden pending UX finalisation; data stays in the response. Backfill: scripts/backfill_marketplace_rollup.py — one-shot rebuild over historic usage_events after deploy, idempotent. USAGE_PROCESSOR_VERSION bumped 4 → 5 so the reprocess loop re-attributes existing events to the new source/ref_id semantics on the next tick. Tests rewritten: test_session_processor_usage, test_usage_rollups, test_marketplace_telemetry, test_api_admin_usage_reprocess, test_db_schema_version, test_home_stats, test_schema_v42_migration. New: test_backfill_marketplace_rollup. * fix(marketplace): refresh Most Popular on search + category changes `loadMostPopular()` early-exits when `state.q` or `state.category` is set, but the search + category handlers only called `loadItems()` — so once the section was visible, typing a query or filtering by category didn't re-run the hide check and the cards stayed on screen out of scope. Tab + sort handlers already chained the call. Add the call to runSearch + category pill click handlers (All + per-category) so the visibility contract holds for every state mutation that can flip the early-exit condition. * feat(marketplace): All-plugins section + 7-day Most Popular Listing layout: - Always-visible "All plugins" / "All items" / "Your stack" section header (label swaps per tab) wrapped in `#mp-all-section` so its margin-collapse mirrors the sibling `#mp-popular-section` and the spacing from the filter row stays consistent in both layouts. - Sort dropdown moved from the filter row into the All-* header, pinned right via `margin-left: auto`. Anchored to its section so the relationship between sort + grid is obvious. - `.mp-section-header` gets `min-height: 32px` + `align-items: center` so the bare-text Most Popular row matches the dropdown-bearing All-* row. - `.mp-section-header` margin tightened 24px → 20px on top. Most Popular: - Capacity reduced 8 → 4 cards. - Now reflects a 7-day window (was 30-day). Backend surfaces `invocations_7d` + `distinct_users_7d` on `MarketplaceItem` alongside the existing 30d fields; the loader pulls a wider page (server still sorts by 30d) and re-sorts + filters client-side on `invocations_7d > 0` so the strip stays "hot right now". - Section label updated to "Last 7 days". - Section now renders on both `curated` and `flea` tabs (was curated-only). Hidden on `my` and whenever search / category filter is active. Refresh hooks wired into search + category click handlers so visibility flips immediately on state change. Backend (`_load_invocation_stats`): - Single SELECT pulls both `last_30d` and `last_7d` rows from `usage_marketplace_item_window`; the result dict carries invocations + distinct_users for both windows. - Trend (recent_7 vs prior_7) kept on the daily fact table so it stays independent of the window snapshot's freshness. * feat(marketplace): Most adopted sort + hide Trending when no trend data Add a fourth sort option to the All-items dropdown — "Most adopted (30d)", keyed on `MarketplaceItem.distinct_users_30d` (true 30d distinct user count from `usage_marketplace_item_window`). Protects the listing from power-user skew that `most_used` is susceptible to: one user × 100 invokes can't beat 10 different users × 1 invoke under adoption sort. Hide Trending option when the response has no trend data. User reported `sort=trending` returning an empty grid because every plugin's `trend_pct` was None (prior-week threshold of >= 3 invocations didn't clear anywhere). Empty grids on a user-selected sort are worse UX than just not offering the sort — surface what works, hide what doesn't. Backend (`app/api/marketplace.py`): - `_apply_sort` gains a `most_adopted` branch (DESC distinct_users_30d, ties by name ASC). - `sort` Literal extended. - `ItemListResponse.available_sorts` lists the sort keys the UI should expose for this response. recent/most_used/most_adopted always; trending only when at least one item in the tab's stats carries a non-null trend_pct. - `_available_sorts(stats_dicts)` helper centralises the rule — curated and flea branches pass one stats dict, my-tab passes both (option is available when either source has trend data). Frontend (`app/web/templates/marketplace.html`): - New `<option value="most_adopted">Most adopted (30d)</option>` between Most used and Trending. - URL state allowlist extended so `?sort=most_adopted` round-trips. - `applyAvailableSorts(available)` runs after each list fetch: hides options not in the response's available_sorts; if the user is on a now-unavailable sort, resets to 'recent' and re-fetches. Search-mode fan-out unions availability across the curated + flea responses so a hit on either side keeps the option visible. * feat(marketplace): funnel chip on cards + deterministic Most Popular sort Card chip — funnel telemetry between description and footer: [stack-icon] N installed · [user-icon] N active · [bolt-icon] N calls · ↑/↓ N% - stack_count (new MarketplaceItem field): for curated it's COUNT(*) on user_plugin_optouts (post-v28 row PRESENCE = subscribed; system plugins are fanned out to every user via fanout_system_for_user so the count includes them naturally). For flea it reuses the existing store_entities.install_count (bumped on install/uninstall). - distinct_users_30d (existing) — active users in the 30d window. - invocations_30d (existing) — call volume. - trend_pct (existing) — week-over-week, both directions: green ↑ / red ↓, magnitude only (sign in the arrow). Hidden when null. Backend additions in app/api/marketplace.py: - MarketplaceItem.stack_count field. - _load_curated_stack_counts() — one SELECT per render, GROUP BY (marketplace_id, plugin_name). Wired into the curated + my-tab branches; flea reads install_count off the entity row directly. Frontend (app/web/templates/marketplace.html): - Heroicons solid 24×24 inlined (one helper per icon, all fill="currentColor" so per-segment colour tokens apply): rectangle- stack (mirrors the My Stack tab icon), user, bolt, arrow-trending- up/down. - Per-segment colour: installed=amber #F59F0A (My Stack accent), active=green #0e9b6a, calls=orange #f97316. Text stays neutral so the chip still reads as metadata, the leading glyph carries the visual cue. Trend pill keeps the full-segment green/red colour. - Zero state: chip hidden when stack_count == 0 AND invocations_30d == 0 — brand-new cards aren't visually penalised by a "0·0·0" row. - Tooltips on every segment via title="…" so hover explains the number's meaning to anyone uncertain about the icon. Most Popular section — deterministic ordering: Previously sorted by invocations_7d DESC with no tie-breakers, so several cards with identical 7d call counts would swap places on refresh (JS stable sort fell back on backend order, and the backend's own tie-breaker for `most_used` was just name ASC — six `grpn` plugins from six test marketplaces collapse to the same name and became indeterminate via list_with_filters' created_at order). New cascading hierarchy (chosen primary now matches what "most popular" really means — wide adoption, not power-user volume): 1. distinct_users_7d DESC ← adoption / social proof 2. invocations_7d DESC ← volume at equal adoption 3. distinct_users_30d DESC ← broader adoption fallback 4. invocations_30d DESC ← broader volume fallback 5. name ASC ← deterministic textual order 6. marketplace_slug ASC ← splits duplicate plugin names across marketplaces Six levels guarantee any two items end at a different sort key, so the strip is stable across refreshes. * fix(marketplace): unify Most Popular on 30d + right-align installed chip Most Popular section was sorting on the 7d window while its cards rendered 30d numbers — header label promised one thing, cards showed another. Unified everything on 30d so a card means the same data everywhere on the page. - Dropped the "Last 7 days" meta from the Most Popular header. - Sort cascade now starts on distinct_users_30d, then invocations_30d, with 7d adoption/volume as recency-aware fallbacks before the name + marketplace_slug deterministic tail. Six levels guarantee identical sort keys never produce indeterminate order across refreshes. - Filter switched from invocations_7d > 0 to invocations_30d > 0 to match the new horizon. - Most Popular now only renders on page 1 of the listing. Past initial discovery, a top-of-list popularity strip on page 2+ would shadow the results the user paged into. Pager click handler refreshes the section so navigating back to page 1 re-mounts it. Chip layout — split engagement vs adoption visually: [user] N active · [bolt] N calls · [↑/↓] N% [stack] N installed └────────── LEFT (time-bounded engagement) ────┘ └── RIGHT (all-time) ──┘ - Installed (stack_count) is all-time, decremented on uninstall. Alone it says little ("12 people installed it") without the engagement context next to it ("…but did anyone actually use it?"). Visually separating the two groups makes that distinction obvious — left group answers "is it used", right answers "does anyone have it". - Implemented via flex with margin-left:auto on .seg-installed so installed drifts to the trailing edge. - Installed tooltip now reads "Currently installed by N users" — the count is a real-time net (uninstall drops it), and saying "currently" makes that explicit. Helps when a card shows 0: signals "nobody has this in their stack right now", not "data missing". * feat(plugin-detail): telemetry chip in hero, derived rows in sidebar Surface the same telemetry funnel the listing card carries on the curated plugin detail page, so clicking through from /marketplace keeps a single mental model — figures match, semantics match. The detail sidebar drops the two raw numbers that used to live there (Invocations 30d / Users 30d — duplicated by the chip now) and replaces them with two *derived* signals only the daily series can provide: Active days + Last used. Backend (app/api/marketplace.py): - PluginDetailResponse.stack_count — curated reads via _load_curated_stack_counts(), flea reuses install_count. Frontend treats both sources uniformly. - _build_telemetry() always returns a dict (never None). Frontend decides chip visibility from stack_count + invocations_30d the same way the listing card does. daily_series is always 30 entries (zero-padded) so "Active days" and "Last used" derivations on the sidebar are trivial array filters. Frontend (app/web/templates/marketplace_plugin_detail.html): - New .hero-telemetry slot at the bottom of the hero meta column, between the pills row and the action buttons. Renders the four funnel segments — active · calls · trend · installed — joined by ` · `. No left/right split: the hero has space, so a single coherent metadata strip reads cleaner than the card's split layout. - Heroicons solid inlined (user / bolt / arrow-trending-up,-down / rectangle-stack) recoloured against the dark hero — icons in lighter tokens (mint #6ee7b7, peach #fdba74, cream #fde68a), trend pill keeps the saturated green/red because direction-coding earns its own colour. - Tooltip on installed reads "Currently installed by N users" — the count is a real-time net (drops on uninstall), and "currently" makes that explicit when a card shows 0. - fmtNum helper added so 1.2k / 14M renderings match the card's format exactly. - Sidebar swap: Invocations + Users rows removed, replaced by Active days → "N of 30" Last used → fmtRelative of the latest non-zero day Both derived from telemetry.daily_series — engagement consistency + recency, neither of which the hero chip exposes on its own. * feat(item-detail): telemetry chip in hero for curated skill/agent Bring the funnel chip the plugin detail page got in 4cf38d40 to the curated inner skill/agent detail page — clicking through from the listing card now keeps the same metadata strip from grid to plugin page to inner item page. Backend (app/api/marketplace.py): - _load_inner_item_stats() rewritten: * always returns a dict (never None) so the frontend can decide chip visibility client-side, same contract as _build_telemetry * adds trend_pct, computed the same way as plugin level (recent_7 vs prior_7 from usage_marketplace_item_daily, ≥3 prior-week threshold) * adds daily_series (30 entries, zero-padded) so the sidebar can derive Active days + Last used - InnerDetailResponse.parent_stack_count — new field. Skills/agents don't have a per-item subscription model, so the hero shows the *parent plugin's* stack count under a "Plugin:" prefix. The funnel: "12 installed plugin → 2 actually use this skill". - curated_skill_detail + curated_agent_detail handlers load _load_curated_stack_counts() once and pass the parent's value. Frontend (app/web/templates/marketplace_item_detail.html): - New .item-detail .hero .hero-telemetry slot beneath the badges row. CSS mirrors plugin-detail's colour tokens (mint/peach/cream Heroicons solid + saturated trend pill) so the two surfaces read as one visual family. - Installed segment uses a "Plugin:" label rendered with reduced opacity to signal the metric describes the parent, not the item itself. Tooltip: "Parent plugin (<plugin_name>) currently installed by N users". - Sidebar Invocations + Users rows removed (chip carries them). Active days + Last used derived from telemetry.daily_series replace them; only rendered when activeDays > 0 so a brand-new skill doesn't show "0 of 30" / "Last used —". - "Type" row dropped from the sidebar — duplicates the hero badge. - fmtNum helper added (matches listing card + plugin detail). Plugin detail (app/web/templates/marketplace_plugin_detail.html): - Hero "Curator: …" line removed. The Details sidebar already carries that info; duplicating it under the h1 was visual noise. - Sidebar "Owner" row renamed to "Curator" — for curated plugins it's a person who curates inclusion in this Agnes instance, not the upstream code owner. "Owner" was a hold-over label. * feat(item-detail): unify hero with plugin detail — pills + breadcrumb + cleaner sidebar - Inner skill/agent hero now uses the same `.pills` / `.pill.cat / .curated / .flea / .muted` class names + CSS as the plugin detail page; the only item-only addition is `.pill.type` (Skill / Agent uppercase, plugin detail has no kind axis). - Hero `Updated` moved out of the meta-row into a muted pill (mirrors the plugin detail hero), removed from the Details sidebar to avoid duplication. - Details sidebar slimmed: dropped Marketplace, Path, Updated rows; Parent plugin now shows the curator-friendly display name (`parent_display_name || manifest_name || slug`) instead of the slug. - Breadcrumb extended to full path: Marketplace > <marketplace_name> > <plugin display name> > <self>, mirroring the plugin detail breadcrumb. - Backend: new `InnerDetailResponse.parent_display_name` field, populated via `_curated_plugin_enrichment` from marketplace-metadata.json — same source plugin detail hero already uses. * feat(marketplace): flea inner skill/agent detail + breadcrumb polish - Flea inner skill/agent detail page parity with curated: * GET /api/marketplace/flea/{id}/skill/{name} + /agent/{name} returning InnerDetailResponse (mirror of curated_skill_detail). * /marketplace/flea/{id}/skill|agent/{name} web routes that render marketplace_item_detail.html with source='flea' + innerName context. * Frontend apiURL grows a third branch for flea-inner; breadcrumb grows to 4 segments (Marketplace > Flea Market > <plugin display name> > <self>) when innerName is set. * Telemetry attribution: MarketplaceItemLookup resolves <flea_plugin>:<inner> prefixes to (source='flea', parent_plugin=<plugin name>) so nested invocations land in the same rollups curated nested skills use. USAGE_PROCESSOR_VERSION bumped 5 -> 6 so the reprocess loop re-attributes historic events. - Breadcrumb 2nd segment is now a generic clickable "Curated Marketplace" / "Flea Market" link to /marketplace?tab=... instead of the opaque per-instance marketplace_name. Applied on both plugin detail and inner item detail. - Inner item hero telemetry chip works for both sources: installedCount branches on parent_stack_count (curated) vs install_count (flea), installed segment drops the "Plugin:" prefix for flea standalone / inner items. - Updated row dropped from Details sidebar on item detail — the hero pill already carries the value, sidebar row was duplicate. * feat(item-detail): block stack-install on flea inner items (mirror curated) Inner skills/agents nested inside a flea plugin can no longer be added to a user's stack on their own — adoption only happens at the plugin level, same rule curated nested items have followed since launch. - Hero action: when innerName is set (curated nested OR flea nested), render "Open parent plugin →" link + helper text instead of the install/remove buttons. Flea standalone entities (no innerName) keep the normal install UX. - Meta-row: same branch now serves curated + flea inner — "part of <parent plugin display name> · by <author>" with the parent link pointing at the right detail page per source. No API gate change needed: POST /api/store/entities/{id}/install only accepts existing entity ids (plugin-level), inner items have no entity id of their own so the endpoint cannot target them directly. * feat(marketplace): telemetry chip on inner cards + fix flea hero chip visibility Inner skill/agent cards on the plugin detail page now carry the same four-segment funnel chip the marketplace listing cards show (N active . N calls . trend . N installed), for both curated nested skills and flea nested skills. Plus two fixes that were keeping the hero chip hidden on flea plugin / flea inner detail pages. - Backend `_load_inner_items_stats_by_parent(conn, source, parent_plugin)` bulk loader: one query per plugin against usage_marketplace_item_window + one against _daily, returning {(name, type): stats}. Avoids N+1 per-card lookups. - `InnerItemSummary` gains invocations_30d / distinct_users_30d / trend_pct / parent_stack_count fields. `curated_detail` and `flea_detail` (in the entity.type=='plugin' branch) enrich the skills / agents lists after the existing cover-photo enrichment loop. - `marketplace_plugin_detail.html`: new `.plugin-detail .inner-card .inv-chip*` CSS lifted from marketplace.html with the listing-card rules, new buildInnerCardChip() helper, buildCardSection appends the chip to each card body. Same gate as the listing card (hidden on parent_stack==0 && calls==0). - fix(flea): flea_detail forgot to populate PluginDetailResponse.stack_count from entity.install_count (listing card does this on line 851; detail endpoint didn't). Hero chip gate `stackCount===0 && calls===0` then always hid the chip even when the entity had installs. Now mirrors listing card semantics: stack_count == install_count for flea. - fix(flea inner): renderInnerHeroTelemetry was reading `d.install_count` for any non-curated source. InnerDetailResponse has no install_count field — it has parent_stack_count (populated server-side from the parent flea plugin's install_count). Gate + label now read parent_stack_count for both curated nested AND flea nested scenarios; install_count remains the flea standalone path. * fix(marketplace): Owner label on flea + parent-centric sidebar for flea inner - Plugin detail Details sidebar — authorship row label now tracks the source: curated bundles get `Curator` (existing behaviour), flea bundles get `Owner`. The `owner_todo` reminder placeholder stays on the curated branch only; flea falls through silently. - Inner item detail Details sidebar — flea-inner (skill/agent nested inside a flea plugin) now shares the curated nested layout: Parent plugin / Bundle size / Active days / Last used / Owner. Drops the flea-standalone shape's `Category`, `Version`, `Installs`, `Released` rows that didn't apply to a nested item. Active days + Last used were already wired (telemetryRows) — they just weren't on the flea-inner branch. * fix(tests): bump SCHEMA_VERSION assertions 47 -> 48 post-rebase The marketplace telemetry migration was renamed _v46_to_v47 -> _v47_to_v48 during the rebase onto main (collision with #326 FTS BM25 migration that took the v47 slot). Two test files still asserted the pre-rebase value: - tests/test_home_stats.py::test_schema_version_constant_is_46 (CI red) - tests/test_schema_v46_migration.py::test_schema_version_is_46 Renames the helper fn name + bumps the assertion. The other two test files (test_db_schema_version.py, test_schema_v42_migration.py) were already updated in the rebase resolution. * fix(telemetry): _build_telemetry returns None when invocations_30d == 0 The follow-up commit that introduced the always-return-dict shape broke the test contract from the original v46 PR (commit b603e998): tests/test_marketplace_telemetry.py::TestDetailTelemetry:: test_detail_endpoint_telemetry_absent_when_no_data AssertionError: assert {'daily_series': [...], ...} is None Both `PluginDetailResponse.telemetry` and `InnerDetailResponse.telemetry` are declared `Optional[Dict] = None`, the frontend renders are None-safe (`d.telemetry || {}` guard + `if (!d.telemetry || ...)` on daily_series), so dropping the dict on zero activity is the cleaner default. * release: 0.54.21 — marketplace telemetry refactor (schema v48) + flea inner detail parity + listing UX polish --------- Co-authored-by: Minas Arustamyan <arustamyan.minas@gmail.com> Co-authored-by: ZdenekSrotyr <zdenek.srotyr@keboola.com>	2026-05-15 20:58:03 +02:00
ZdenekSrotyr	c5d67faad2	feat(memory): DuckDB FTS BM25 search for knowledge items (#121) (#326) ... * feat(memory): DuckDB FTS BM25 search for knowledge items (#121) Replaces `title ILIKE '%q%' OR content ILIKE '%q%'` ranked by insertion order with BM25 relevance ranking via the DuckDB `fts` extension. Czech queries like `cesky` match documents containing `česky` (`strip_accents=1` + `lower=1`). Architecture: - src/fts.py — ensure_fts_loaded / ensure_knowledge_fts_index helpers. The extension is per-connection (INSTALL persisted at engine level, LOAD per-conn). Both helpers are idempotent and soft-fail on unavailability with a logged WARNING. - Schema v47 (_v46_to_v47) — builds the initial BM25 index over knowledge_items(title, content) keyed by id. Migration is best-effort against ANY exception (not just duckdb.Error) so the schema bump cannot get stuck on v46 if a non-DuckDB error escapes the helper. - KnowledgeRepository.search — FTS-or-ILIKE dichotomy with execute- time fallback. Same filter surface (statuses / category / domain / source_type / personal / audience / dismissed) either way. ensure_fts_loaded() returning True only guarantees the extension is loadable, NOT that the index exists — migration soft-fail or a concurrent overwrite=1 rebuild's drop-then-create window leaves the extension loaded but the index missing. The BM25 execute is wrapped in try/except duckdb.Error → ILIKE retry so transient failures cannot 500 the /api/memory?search= endpoint. - KnowledgeRepository.count_items — mirrors the same FTS-or-ILIKE decision tree plus the execute-time fallback so the count always matches the paginated result set. - Per-mutation rebuild — create and title-or-content update rebuild the index via overwrite=1 PRAGMA. Status flips skip (token stream unchanged). - app/main.py lifespan rebuilds once at boot as a safety net for instances already on v47 across restarts. - bm25_score column shape: ILIKE fallback now selects `NULL AS bm25_score` so the result column set matches the FTS path. Consumers can read the score uniformly; absence of relevance ranking is signalled by the column being None everywhere, not missing. Tests in tests/test_knowledge_fts_search.py (9 tests): - BM25 multi-term match set + adversarial-review fix asserting higher-density doc ranks first (skipped if extension unavailable). - bm25_score column attached when extension available. - ILIKE fallback path on search + count_items via patched ensure_fts_loaded → False; bm25_score is None on this path. - Adversarial-review fix: search and count_items also fall back when the extension is loaded but the index is missing (simulated via drop_fts_index PRAGMA — the exact production failure mode the fallback guards against). - Index rebuild on create (new item searchable immediately). - Title update re-surfaces row under new term, drops old. - Czech-diacritic round-trip (cesky query → česky doc). Pinned schema-version asserts bumped 46 → 47 (test_db_schema_version, test_home_stats, test_schema_v42_migration, test_schema_v46_migration). Closes #121. * release: 0.54.20 — Corporate Memory BM25 search + All-Items bulk-edit batch bar	2026-05-15 20:10:59 +02:00
ZdenekSrotyr	9e948abc9c	release(0.54.18): Curated Memory restructure + per-user Dismiss + bundled adversarial-review fixes (#316/#320/#322) (#324) ... * feat(web): Curated Memory restructure + per-user Dismiss + filter-state utility Squashed from cvrysanek/zsrotyr's 4-commit PR branch + rebased onto current main + CHANGELOG bullets spliced into [Unreleased] (preserves existing #316/#320/#322 entries that landed on main since the branch was authored). Routes + access: - /corporate-memory now user-facing (get_current_user), in primary nav next to "Data Packages" — same gate as /api/memory/*. - /admin/corporate-memory is the new admin review queue location (was /corporate-memory/admin); reached via Admin dropdown. Template renamed: corporate_memory_admin.html → admin_corporate_memory.html. Visual chrome: - Both pages migrate to shared _page_hero.html blue hero band. Per-user Dismiss (new feature, schema v46): - knowledge_item_user_dismissed(user_id, item_id, dismissed_at) + index. - POST /api/memory/{id}/dismiss + DELETE (idempotent). - Mandatory items can never be dismissed — enforced at 2 layers. - GET /api/memory: hide_dismissed=false default + dismissed_by_me flag. - GET /api/memory/bundle: always excludes dismissed for the caller. - UI: Dismiss/Undismiss button per item (hidden for mandatory), gray-out + line-through for dismissed rows, Hide-dismissed toggle. Admin edit modal: - Category as <select> + "Add new category…" reveal. - Audience as <select> with (unset)/all/group:<name> from RBAC. - Tags: full tag-input widget (pills, ×-remove, Backspace pop, Enter/comma to add, ↑/↓ typeahead from EXISTING_TAGS). Bulk-edit modal pickers (closes #128): - Move-to-category / Add-tag: <select> + add-new. - Set-audience: <select> (no more typo-able 'gourp:eng'). - Remove-tag: closed-set picker. FilterState utility: - app/web/static/js/filter-state.js — save/load/clear/bindInputs for per-page localStorage filter state. Adopted on /corporate-memory. E2E verified live on a real VM through the API + browser flow. * release: 0.54.18 — Curated Memory restructure + 4 adversarial-review fixes Bundles together: - #316 fix(store): surface review failures + harden publish gate (BREAKING fail-CLOSED guardrail, override v2+ promote, restore guard, retry/rescan staged-bundle, banner widening, LLM truncation retry) - #320 fix(store): C2 bundle export RBAC + H2 per-entity write lock + H3 update_status compare-and-swap with bg_verdict_skipped audit - #322 fix(store): M1 prompt sentinel filename escape + M2 atomic promote_to_version helper + L1 admin forensic download per-version - #324 Curated Memory restructure + per-user Dismiss + FilterState utility Bump from 0.54.17 → 0.54.18 (patch — pre-1.0 policy: every cycle is patch).	2026-05-15 18:51:05 +02:00
ZdenekSrotyr	3e19caa975	fix(security): RBAC filter uses stable user_id instead of mutable email local-part (#293) (#299) ... * fix(security): RBAC filter for agnes_sessions matches both email local-part and user_id The upload API (POST /api/upload/sessions) stores session files under user_sessions/{user_id}/ (UUID), while the session collector uses the OS username (email local-part). The session pipeline writes the directory name verbatim into usage_session_summary.username, so the column can contain either value depending on the ingestion path. The RBAC filter in build_filter_clause previously only matched the email local-part, missing sessions uploaded via the API. The fix adds an OR condition so non-admin users see rows where username matches either their email local-part or their user_id. Closes #293 Co-Authored-By: zdenek.srotyr <zdenek.srotyr@keboola.com> * fix(security): RBAC filter uses stable user_id instead of mutable email local-part Closes #293 Previous fix used OR condition matching both email local-part and user_id in the username column. This was fragile: email changes would break filtering. This commit introduces a dedicated user_id column populated by the session pipeline via resolve_user_id(), and switches the RBAC filter to use it exclusively. Changes: - Schema v45: add user_id column to usage_session_summary and usage_events - UsageProcessor: accept and store user_id in both tables - runner.py: resolve_user_id() maps directory name to users.id UUID (exact match for UUID dirs, email LIKE for local-part dirs) - INTERNAL_TABLES: agnes_sessions/agnes_telemetry filter on user_id column - build_filter_clause: simplified to WHERE user_id = '<uuid>' (no OR) - me.py/admin_user_sessions.py: query by user_id OR username for backward compatibility during transition - USAGE_PROCESSOR_VERSION bumped 2→3 to trigger reprocessing/backfill - Tests updated: 27 pass including new email-change resilience test Co-Authored-By: zdenek.srotyr <zdenek.srotyr@keboola.com> * fix(tests): bump schema version assertions 44→45 Co-Authored-By: zdenek.srotyr <zdenek.srotyr@keboola.com> * fix(docs): correct resolve_user_id docstring, add TypeError comment Co-Authored-By: zdenek.srotyr <zdenek.srotyr@keboola.com> * fix(security): address review — backward-compat OR, LIKE escaping, narrower TypeError Co-Authored-By: zdenek.srotyr <zdenek.srotyr@keboola.com> * fix(security): address code review — eliminate TypeError hack, add resolve_user_id tests Co-Authored-By: zdenek.srotyr <zdenek.srotyr@keboola.com> * fix(db): create user_id indexes in _v44_to_v45, not _SYSTEM_SCHEMA _SYSTEM_SCHEMA runs before the migration ladder. On an upgrade from v42/v43/v44, usage_events / usage_session_summary already exist without the user_id column (CREATE TABLE IF NOT EXISTS is a no-op), so the CREATE INDEX ... (user_id) lines in _SYSTEM_SCHEMA failed to bind and aborted _ensure_schema — the app would not start post-upgrade. Move the index creation to _v44_to_v45, which ADDs the column first. Same pattern as the v41 audit_log indices. * fix(usage): bump USAGE_PROCESSOR_VERSION 3→4 for user_id backfill #303 shipped USAGE_PROCESSOR_VERSION=3 (release 0.54.12) for its <command-name> slash extraction. This PR's 2→3 bump collided with it on rebase, so the reprocess loop would not re-trigger to backfill the new user_id column on deployments already running v3. Bump to 4. * release: 0.54.13 — RBAC filter uses stable user_id (#293) --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>	2026-05-14 14:12:54 +00:00
Vojtech	37ad39c8a3	feat(home): status frame on /home (operator-gated, onboarded-only) (#297) ... * feat(home): status frame on /home — last sync, sessions, prompts, tokens, projects Adds the homepage status frame: a 5-card row above the install-hero / offboard-strip on /home showing the calling user's Last sync (their last `agnes pull`), Sessions, Prompts, Tokens used, and Projects worked on, with a 24h/7d pill toggle. Backed by `GET /api/me/home-stats?window=` (one DuckDB CTE joining `users` + `usage_session_summary` + `usage_events`) and SSR'd from the same `compute_home_stats` helper on initial paint so there's no spinner. The window toggle is the only JS-driven path. Side surfaces: - `GET /api/sync/manifest` now stamps `users.last_pull_at` so `agnes pull` (and the Claude Code SessionStart hook that wraps it) imprints the analyst's last sync time for the new card. - `usage_session_summary` gains four BIGINT token counters (input_tokens, output_tokens, cache_read_tokens, cache_creation_tokens) summed from JSONL `message.usage.*` per assistant turn. - `USAGE_PROCESSOR_VERSION` bumps 1 → 2 so the session-pipeline reprocess loop invalidates stale summaries and backfills tokens on the next tick. Schema migration v43 → v44 is idempotent ALTERs (last_pull_at + 4 token columns) — fresh installs receive them from `_SYSTEM_SCHEMA`, upgrade path runs `_v43_to_v44`. Defaults (NULL / 0) backfill existing rows cleanly. 9 new tests in tests/test_home_stats.py cover the migration, endpoint shapes (24h/7d/unknown/empty/missing-user), and the manifest-side last_pull_at bump. * docs(CHANGELOG): homepage status frame entries under [Unreleased] The post-rebase release-cut now belongs to whichever PR lands next after main rolled to 0.54.9. This PR logs its bullets under [Unreleased] (Added: homepage status frame, per-user pull tracking, token counters; Changed: schema v43 → v44 migration) so they ride out with the next release-cut. * fix(tests): bump test_schema_v42_migration asserts to v44 CI failed because tests/test_schema_v42_migration.py hardcoded `assert SCHEMA_VERSION == 43` and `assert v == 43` after init. v44 (homepage stats frame backing columns) was introduced in the preceding feat commit; this aligns the existing v42-era migration tests with the new schema version. * feat(home): gate status frame on operator flag + user.onboarded Two gates on the homepage status frame: 1. **Operator master switch** — `get_home_status_frame_visibility()` in app/instance_config.py mirrors the existing `get_home_automode_visibility()` shape: env var `AGNES_HOME_SHOW_STATUS_FRAME` > yaml `instance.home.show_status_frame` > default `True`. Cautious-rollout instances can disable the frame without forking; the yaml example documents both knobs. 2. **Onboarded gate** — the template only renders the frame when the caller's `users.onboarded` is true. First-day users see a clean install-hero before all-zero stat cards; the frame appears automatically on the next render after `agnes init` POSTs `/api/me/onboarded`. Router skips the `compute_home_stats` DB read entirely when either gate is closed; `home_stats` arrives at the template as None in that branch and the `{% if %}` shortcuts the include. Why both gates: PostHog feature flags evaluated and rejected — this codebase uses PostHog for analytics capture only, not feature gating; adding a per-user feature_enabled() call on the /home critical path would couple the homepage render to a remote eval and still require an admin master switch. The onboarded gate is a UX coherence rule layered on top of the operator switch, not an A/B test signal. 3 new tests in test_home_stats.py cover the env-var resolution (falsey values + default-true). The yaml example gets a `home:` block documenting both `show_automode` (pre-existing flag, was undocumented in the example) and `show_status_frame`.	2026-05-14 09:28:47 +00:00
ZdenekSrotyr	b4d3c576af	Activity Center: audit log + telemetry + sessions + agnes_* tables (#278) ... * docs(spec): admin observability spec + Activity Center MVP plan Parent spec (480 lines) + executable plan (2295 lines, 14 TDD tasks). Covers Activity Center rebuild (/admin/activity), with /admin/sessions and /admin/feedback deferred to follow-up plans. Already incorporates reviewer-pass revisions across three angles (security, production resilience, code architecture): - _get_db import path corrected to app.auth.dependencies - Test fixtures aligned with seeded_app / admin_user / get_system_db - All new audit writes wrapped in try/except + logger.exception - Filename sanitization on session uploads - DuckDB DESC index behavior documented; upgrade window flagged - Migration idempotency + evolved-DB test cases - reveal_raw + shared-cache multi-worker explicitly deferred Targets schema v40 (audit_log gains params_before, client_ip, client_kind, correlation_id + 3 indices). * feat(db): schema v40 — audit_log gains params_before, client_ip, client_kind, correlation_id + 3 indices * chore(test): clean up Task 1 — drop unused import, rename stale test * feat(audit): AuditRepository.log() accepts params_before/client_ip/client_kind/correlation_id * test(audit): strengthen params_before assertion to round-trip JSON content * feat(audit): AuditRepository.query() rich filters + keyset cursor pagination * feat(sync): SyncStateRepository.list_recent() cross-table feed * feat(audit): POST /api/sync/trigger writes audit_log row * feat(audit): POST /api/scripts/run-due writes audit_log row * feat(audit): POST /api/upload/sessions writes audit_log row + sanitizes filename * feat(audit): GET /api/data/{table_id}/download writes audit_log row * feat(activity): /api/admin/activity timeline + /health + /sync endpoints * feat(ui): /admin/activity rebuilt — health pulse, timeline, sync grid; /activity-center → 308 redirect BREAKING: removed demo executive-pulse / maturity-roadmap content from activity_center.html. The page now reflects real audit_log + sync_history data. * feat(ui): admin nav + dashboard widget point at /admin/activity * feat(activity): recursive-audit suppression for AC read endpoints (60s window per actor+filter) * feat(activity): emit PostHog events when integration enabled (no-op default) * fix(audit): move v40 indices out of _SYSTEM_SCHEMA + update test_repositories to unpack query() tuple _SYSTEM_SCHEMA CREATE INDEX on audit_log(timestamp) failed when migration tests hand-roll a bare audit_log (id, action) without the timestamp column. Fix: remove indices from _SYSTEM_SCHEMA; add ADD COLUMN IF NOT EXISTS guards for timestamp and other pre-v40 columns in _v39_to_v40() so the upgrade path is safe on any hand-rolled schema; call _v39_to_v40 explicitly in the fresh-install (current==0) path to restore index creation there. Also unpack the (rows, next_cursor) tuple from AuditRepository.query() in the three TestAuditRepository tests that still treated it as a list. * docs: CHANGELOG entry for Activity Center MVP * chore: refresh stale module docstring in app/api/activity.py * feat(cli): agnes admin activity — terminal access to Activity Center (timeline + health + sync) * fix(db): _v39_to_v40 — add IF NOT EXISTS guard for 'action' column The v39→v40 ladder step adds defensive ADD COLUMN IF NOT EXISTS for every audit_log column so a hand-rolled bare audit_log (id only) is safe through the ladder. 'action' was missing from the guard list, causing CREATE INDEX idx_audit_action_time to fail on tests that stub audit_log with only an id column (tests/test_e2e_extract.py:: TestSchemaMigration::test_migration_preserves_and_extends). Local 6/6 schema tests + the previously-failing CI test pass. * docs(spec): platform telemetry epic — Boss directive + Activity Monitoring plan rebased onto v40 (stacked on zs/spec-activity-center) * feat(db): schema v41 — 7 usage_* tables for telemetry (events, summary, rollups, attribution) * chore(db): tighten v41 — usage_session_summary.session_id NOT NULL + upgrade test asserts all 7 tables * feat(usage): UsageAttributionRepository — replace/delete/lookup over usage_attribution_* tables * refactor(marketplace): extract list_inner_skills/agents/commands to src/marketplace_listing.py for reuse * feat(usage): explode plugin attribution on marketplace sync + store entity write; backfill script * refactor(marketplace): finish src/marketplace_listing.py extraction — drop duplicate _list_inner_* + _parse_frontmatter from app/api/marketplace.py * feat(usage): promote attribution helpers to src/usage_attribution_helpers.py; hook update_entity rename + bundle-swap; clarify best-effort semantics * feat(usage): UsageProcessor real extraction + rollup rebuild + 10 fixture-driven tests * fix(usage): include tool_id in event hash + executemany + rollup transaction (critical multi-tool-turn drop fix) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(marketplace): popularity stats — invocations_30d + trend + sort=most_used|trending + Most Popular section * feat(admin): /admin/users/<id> Sessions section — list + single-file + bulk-zip downloads (audit-logged) * feat(usage): admin export endpoint + CLI — csv/json/parquet streaming, filters, audit-logged * feat(usage): agnes admin ask — LLM Text-to-SQL over usage_events with SELECT-only validator (audit-logged) * feat(usage): reprocess + prune endpoints + scheduler daily prune job + CLI * docs: PLATFORM_SETUP.md operator playbook + HOWTO/ cookbook (5 guides + index) Adds docs/PLATFORM_SETUP.md as a consolidated operator playbook covering bootstrap, TLS, marketplaces (curated + flea), scheduler env vars, telemetry extraction/export/ask/prune, privacy posture, and daily routine. Adds docs/HOWTO/ with 5 analyst cookbook guides: first query, snapshots for remote tables, private sessions, feedback + admin ask, and customizing skills. Existing setup docs (QUICKSTART, DEPLOYMENT, ONBOARDING, HEADLESS_USAGE) get a one-line cross-reference at the top pointing to PLATFORM_SETUP.md. * docs(changelog): platform telemetry epic — usage_* foundation + surfaces + admin access + docs Comprehensive [Unreleased] entry covering: usage_events/session_summary/ tool_daily/plugin_daily tables (v41), attribution lookup tables, backfill script, marketplace Most Popular + invocation chips + sort, admin Sessions section, export/ask/reprocess/prune endpoints + CLI mirrors, Activity Center (v40), PLATFORM_SETUP.md + HOWTO/ docs, and operations notes for v41 upgrade. * fix(security): block DuckDB read_*/http_*/glob functions in usage_ask validator + symlink escape guard in session zip + clarify mark-private semantics * fix(admin): parquet export tempfile cleanup on COPY failure + correct processed-first sort on /admin/users/<id>/sessions * feat(audit): close 8 production audit gaps — query (local/remote/hybrid), catalog/schema/sample, snapshot estimate/create, check-access * feat(ui): /admin/usage summary dashboard + per-user activity tab on /admin/users/<id> * fix(audit): cap error messages at 200 chars + audit user_activity reads + recursion guard on usage.summary * fix(audit): catalog.list audits on error path + clean up deferred json import * fix(ux): client_kind=cli for PAT auth + timeline empty state + email-instead-of-uuid + nav reorder + help text + loading indicators + ask doc * feat(observability): unify /admin/activity into single page with saved views - KPI cards (events, users, error rate, p95) clickable as quick-filters - Faceted filter dropdowns populated from audit_log in the current window - Sortable audit table, cursor pagination, per-row JSON side panel - Saved views (schema v43: user_observability_views) — per-user state - Top bar: window selector + 30s Live toggle + saved views dropdown - /admin/scheduler-runs → 308 redirect (source=scheduler filter) - New endpoints: /api/admin/observability/{facets,kpis,views} * test: update activity + scheduler-runs tests for unified page - test_admin_activity_page_renders asserts new structural anchors - test_admin_scheduler_runs_page_admin_only asserts 308 redirect * fix(observability): respect [hidden] on modal + side panel CSS `display: flex` on .obs-modal beat the [hidden] attribute's UA display:none, so the save-view modal rendered on page load and Cancel clicks couldn't dismiss it. Gate the modal's flex layout on :not([hidden]); add the same display:none guard prophylactically to .obs-panel and .obs-views-panel. * feat(observability): user enrichment in audit + interactive /admin/usage Activity: - /api/admin/activity now joins users for user_email + user_name per row - User column renders "name (id-prefix)" or "email (id-prefix)" instead of an opaque truncated UUID; falls back to id when the user record is missing Usage: - /admin/usage rewritten as the same filter/group-by/search pattern as /admin/activity. Faceted dropdowns (User / Tool / Source / Event type) populated from usage_events; debounced free-text search across tool_name / skill_name / subagent_type / command_name - New endpoints /api/admin/usage/{facets,kpis,query}; the query endpoint supports group_by in {day, username, tool_name, source, ref_id} with sort + offset pagination, plus an ungrouped raw-events mode - 4 KPI cards (events, distinct users, distinct tools, error rate) are clickable quick-filters; clicking a grouped row applies the bucket as a filter - Old static `?window=7d|30d|all` server preload removed; all state is client-side via since_minutes + group_by + filters in the URL * fix(observability): clearer labels, all-column sort, drop saved views UI - Rename page titles: "Activity" → "Server activity", "Usage" → "Tool usage" with a one-line subtitle on each explaining what the page covers and linking the other one. The two pages source different data (audit_log vs usage_events) and the previous labels conflated them. - Drop the saved-views dropdown + save modal from /admin/activity. The modal pop-open bug was the trigger; the value wasn't there yet. The /api/admin/observability/views CRUD + DuckDB table stay in place. - Rename "Live (30s)" to "Auto-refresh (30s)" with a tooltip clarifying that it's the re-fetch rate, not the time range. Time range now labeled "Time range" instead of "Window". - All audit-table columns are sortable (User, Source, Action, Resource, Result added); sort is page-local with a Jinja comment explaining the trade-off. Same for raw usage rows. - Fix duplicate sort-arrow bug — the literal "▼" in the Time th HTML was rendering alongside the CSS ::before arrow. Removed the literal; CSS is the single source of truth. * feat(observability): global Sessions browser + transcript viewer + CLI Web: - /admin/sessions — list every collected session JSONL across all users with time-range, user, model, errors-only and free-text filters. Default sort surfaces error-heavy sessions first. KPI cards (sessions, distinct users, sessions w/ errors, tool error rate) clickable as quick-filters. - /admin/sessions/<username>/<file> — transcript viewer rendering the JSONL chronologically: user prompts, assistant text, tool calls (with JSON input) and tool results (with flattened output). Errors get a red border + chip and a "Next error" navigation button at the top. - Admin dropdown gains a "Sessions" link. API: - GET /api/admin/sessions/{list,kpis,facets} — filtered cross-user reads off usage_session_summary - GET /api/admin/sessions/{username}/{file}/transcript — parses JSONL via the existing services.session_pipeline.lib, returns chronological events - GET /api/admin/sessions/{username}/{file}/download — JSONL stream, same path-safety guards as the per-user endpoint, audit-logged CLI: - `agnes admin sessions list [--user X] [--errors] [--since 7d]` — table output with `!` prefix on rows that hit a tool error - `agnes admin sessions show <username> <file>` — transcript dump, with `--errors` to print only the failed tool_result blocks - `agnes admin sessions download <username> <file> [-o path]` - `agnes admin sessions kpis` — top-level numbers * feat(internal): expose telemetry tables to agnes query with row-level RBAC Three new registered tables backed by system.duckdb, queryable through the same /api/query plumbing analysts use for Keboola / BigQuery / local sources: agnes_sessions → usage_session_summary (filter: username) agnes_usage → usage_events (filter: username) agnes_audit → audit_log (filter: user_id) RBAC is per-row, not per-table: admins see every user's rows; non-admins see only their own. The filter is built server-side from the auth user dict; non-admin filter values are regex-validated before SQL interpolation. Implementation: - new connector connectors/internal/ with access (filter+exec) + registry (idempotent table_registry seed at startup) - /api/query detects internal table refs and short-circuits to a CTE wrapper that prepends "WITH agnes_x AS (SELECT * FROM <src> WHERE …), …" then "SELECT * FROM (<user_sql>) AS _q". DuckDB cursor on the shared system.duckdb handle — opening parallel handles / ATTACH on the same file is blocked process-wide. - mixing internal + BQ / registered local tables in one SELECT is rejected (v1 limitation) - src.rbac.can_access_table waves internal tables through for all authenticated users; row scoping is the actual security control - /api/v2/schema and /api/v2/sample gained internal branches; sample intentionally skips its cache because rows are RBAC-scoped per caller - audit row written as action='query.internal' with is_admin flag Tests: connectors/internal/access — RBAC, filter clause, schema, CTE wrapper coexistence with user-supplied aggregations, unsafe-username rejection. 16/16 passing. Motivating queries this enables: SELECT tool_name, COUNT(*) FROM agnes_usage WHERE is_error GROUP BY 1 ORDER BY 2 DESC -- analyst self-introspection: which tools fail for me? SELECT user_id, COUNT(*) FROM agnes_audit WHERE action = 'session.transcript_view' GROUP BY 1 -- admin: who's been looking at whose session transcripts? * feat(admin): group dropdown into 5 named sections + internal tables in /catalog Admin dropdown gains section headers so admins can land on the right page without re-reading the full menu: Activity Center Server activity / Tool usage / Sessions Users & Access Users / Groups / Resource access / Tokens Data Tables Agent Experience Curated Marketplaces / Flea Submissions / Agent Setup Prompt / Agent Workspace Prompt Server Server config "Agent Experience" frames the curated content + prompts as one cluster — it's all admin-controlled material that shapes what an analyst's AI agent encounters. "Configuration" → "Server" since only one item lives there now. Renamed the section's first two items: "Activity" → "Server activity" (matches page H1) "Usage" → "Tool usage" Also fixes /catalog visibility of the internal tables (agnes_sessions / _usage / _audit) for non-admin users: ``app.auth.access.can_access`` short-circuits to True for resource_type='table' + an internal-table id. Without this, non-admins saw the tables in /api/v2/catalog (which uses the same RBAC bypass) but not on the /catalog HTML page (which calls can_access directly, requiring a resource_grants row internal tables don't have). CSS for `.app-nav-menu-section`: small caps, muted, non-clickable; first section trims top padding so the panel doesn't open with an awkward gap. * refactor(admin): move corporate memory into Admin > Agent Experience Memory link was the only admin-only entry in the primary nav (gated by session.user.is_admin). Moves it into the Admin dropdown under Agent Experience, alongside Curated Marketplaces / Flea Submissions / Prompts — all admin-curated content that shapes what an analyst's AI agent encounters. Renamed the nav label to "Shared Knowledge" to match what the page actually is (admin-curated organisational knowledge from session verification, surfaced to agents). URL stays at /corporate-memory; the route still gates on require_admin per the existing comment. Side effect: primary nav (Home / Marketplace / Data Packages) is now uniform for every authenticated user — no conditional admin-only entry. * ui: rename admin entries to Curated Knowledge / Init Prompt / Workspace Prompt - "Shared Knowledge" → "Curated Knowledge" (parallel with "Curated Marketplaces" in the same Agent Experience section; "curated" tells the admin what they do there — review + approve) - "Agent Setup Prompt" → "Init Prompt" (matches the `agnes init` flow it actually drives) - "Agent Workspace Prompt" → "Workspace Prompt" (the "Agent" prefix was redundant — every item in the section is agent-facing) Renames page titles + H1s on /admin/agent-prompt and /admin/workspace-prompt to match. * refactor: rename Usage → Telemetry across user-facing surfaces External surfaces all switch; internal Python module / file names and the physical DB tables (usage_events, usage_session_summary, usage_tool_daily, usage_plugin_daily) stay — renaming them would force a schema migration + a redo of the LLM Text-to-SQL prompt for no analyst-visible win. Changes: - Admin dropdown: "Tool usage" → "Telemetry" - Page H1 / <title>: same - URL: /admin/usage → /admin/telemetry; old URL 308-redirects - API prefix: /api/admin/usage/* → /api/admin/telemetry/* - CLI: primary command `agnes admin telemetry …`; `agnes admin usage` kept as a deprecated alias so existing operator scripts keep working - Internal data-source table id: agnes_usage → agnes_telemetry. The registry seed now evicts any stale internal-source row whose id no longer matches INTERNAL_TABLES, so the old `agnes_usage` row is removed from table_registry on next app boot - All tests + JS endpoint paths updated * test(rbac): include auto-appended internal tables in expectations get_accessible_tables now appends agnes_sessions / agnes_telemetry / agnes_audit to every authenticated user's accessible-tables list so the internal data source shows up in /catalog. The two existing rbac tests asserted hardcoded list shapes that pre-dated the change. Rewritten to assert "granted tables + the canonical internal-table set" instead of literal lists, so the test stays correct if the internal table roster changes again later. * ui: visual dividers between admin-dropdown sections Adds a 1px top border + 6px top margin to every section header except the first, so the five named groups (Activity Center, Users & Access, Data, Agent Experience, Server) read as visually separated clusters. The header itself stays small-caps + muted as before — the border is additive. * ui(memory): match obs-topbar visual on /corporate-memory The Curated Knowledge page (linked from the admin dropdown's Agent Experience section) opened straight into the stats bar — no title, no subtitle, no shared chrome with the other admin pages. Adds an obs-topbar-style header at the top of .container-memory: - H1 "Curated Knowledge" - subtitle explaining what the page is + how AI agents pull from it The `.ck-*` class set duplicates the inline obs-* styles from /admin/activity etc. for this one page; promoting the obs-* class set to style-custom.css for shared reuse is the obvious next step (4 pages already inline the same CSS), tracked as a follow-up. Page <title> also renamed from "Corporate Memory" → "Curated Knowledge". * ui(tables): list Agnes internal tables in /admin/tables + group in /catalog /admin/tables previously rendered three per-source-type listings (BQ / Keboola / Jira) and dropped any row whose source_type didn't match — so the agnes_sessions / agnes_telemetry / agnes_audit rows seeded into table_registry were invisible. Adds a fourth read-only section "Agnes internal tables" that filters source_type === 'internal' and renders the same registry-table layout the other sections use, with two changes: - no Register button (these rows are seeded on every app boot from connectors/internal/registry.py) - Edit + Delete actions hidden (any change would be reverted on the next start). Manage access stays so admins can still inspect. Mode badge picks up a new mode-internal CSS class (teal accent) so the display doesn't lie and call it "local". In /catalog, internal tables now group under an "agnes" accordion section (bucket="agnes" on seed) instead of falling into the catch-all "default". Single source of truth for which tables exist; admins find them where they expect. * ui(tables): Agnes internal as a 4th tab next to BQ/Keboola/Jira Previous iteration mounted the internal-table listing as a separate standalone card under the tab strip. Reshapes it to a proper tab-content section so admins switch between data sources via one consistent nav (BigQuery / Keboola / Jira / Agnes internal). - New tab button "Agnes internal" in the tab-nav. - The listing card becomes <section id="tab-content-internal" class="tab-content">; switchTab() already routes by id so no JS change beyond extending the hash allowlist for direct #internal links. - Tab content keeps the read-only treatment from the previous commit (no Register button, no Edit / Delete in renderRegistryListing). * ui: rename Curated Knowledge → Curated Memory Settles the naming back on "Curated Memory" — parallel structure with "Curated Marketplaces" in the same Agent Experience section, and zero rename ripple: URL (/corporate-memory), API (/api/memory/*), CLI (agnes admin memory), and Python modules all stay on "memory" so the admin label finally lines up with the underlying surfaces. The "Curated" prefix still tells admins what they do on the page (review pending → approve / mandate / reject) and reads as a sibling of "Curated Marketplaces" right next to it in the dropdown. Touches: admin dropdown label, page <title>, page H1. DB tables stay on knowledge_* (already the canonical naming for the data shape). * ui: rename "Server activity" → "Audit log" "Audit log" is what the page actually is — server-side audit_log table rendered with KPI cards + filter bar + sortable table. The "Server activity" label confused the term with Claude Code session telemetry (Telemetry page) and didn't make the source/concept clear. Touches: - Admin dropdown nav label - /admin/activity page H1 + subtitle - /admin/telemetry subtitle cross-link - test_activity_api page-renders assertion URL (/admin/activity) and API (/api/admin/activity/*) stay — the "activity" name has stuck at the route layer for a year; rerouting those would churn dashboards/bookmarks for zero analyst-visible win. * ui(admin-nav): gray band on each section header for clearer separation Previous iteration used a 1px top border between section labels — the labels still blended into the items above/below at a glance. Switches to a light gray background band per section header, extended edge-to- edge inside the panel via negative horizontal margins. Bolder font-weight (700) reinforces the separation; bumping the font color isn't needed because the band itself does the work. First section's header tucks into the panel's top border-radius so the band reaches the corners without a gap. * ui(catalog): rename internal-table category to "Agnes Internal" `bucket` is what /catalog renders as the accordion category header verbatim — "agnes" lowercase didn't read as a real category name and got confused with a system identifier. Bumps to "Agnes Internal". Seed re-applies on every app boot so existing rows pick up the new bucket value via `ON CONFLICT (id) DO UPDATE`. * ui(catalog): split Agnes Internal into its own card on /catalog Previously the three internal tables landed inside the "Core Business Data" card under an "Agnes Internal" accordion alongside Keboola / BQ buckets — readers conflated system telemetry with business datasets, and the data_stats header counter ("3 tables · ~X rows total") only ever counted synced rows so internal tables looked invisible. Split the catalog page into two cards: - Core Business Data: only non-internal source_types (Keboola, BQ, Jira). Accordions group by bucket as before. Stats counter reflects this card's tables. - Agnes Internal: a dedicated card with its own visual treatment (teal accent matching the mode-internal badge in /admin/tables). Flat list (no accordion — only 3 rows, never grows here), each row carries the canonical `agnes query` snippet. Read-only — no profiler click, no In-stack toggle, no sync metadata. Route adds `internal_card` context object; template renders the new card only when it's non-None. * fix(rbac): hide internal tables from /admin/access + drop "my" framing Two related cleanups for the Agnes-internal tables: 1. /admin/access (resource grants) no longer lists them. The `can_access` check has a hardcoded internal-table bypass — security is row-level (per-request view filter), so a table-grain `resource_grants` row would do nothing. Surfacing them in the UI let admins set up grants that silently no-op. Filter at the `_table_blocks` projection so the UI tree never sees them. 2. Display names drop the analyst-perspective "my" framing: "Agnes — my sessions" → "Agnes sessions" "Agnes — my telemetry events" → "Agnes telemetry events" "Agnes — my audit log" → "Agnes audit log" The "my" only makes sense from the querying analyst's seat (`SELECT … FROM agnes_sessions` returns *their* rows); on /admin/* pages where admin sees / configures them across users, the pronoun was misleading. Description text now spells out the row-level RBAC contract explicitly. Display names update via TableRegistryRepository.register's ON CONFLICT UPDATE on next app boot; no manual cleanup needed. * ui: subtitle notes about agnes_* tables on each Activity Center page The recursive observability story — Agnes serves its own audit / telemetry / session data through the same `agnes query` plumbing analysts use for business data — wasn't surfaced anywhere on the admin pages that show that data. Three pages get a one-liner with the canonical `agnes query` snippet + the RBAC contract (analysts see their own rows, admin sees all): - /admin/activity (Audit log) → agnes_audit - /admin/telemetry (Tool usage) → agnes_telemetry - /admin/sessions → agnes_sessions Sets up the discovery moment for admins: they're reading the page, they see "you can query this from Claude Code", they remember it when an analyst asks "how do I find my own failed tool calls?". * ui(tables): explain "Show log" empty-state on /admin/tables Cache warmup log <pre> renders with a dark background and is only populated by the SSE stream during a Re-warm all run. Opening the page cold + clicking Show log just revealed a black bar with no context — admins couldn't tell what they were looking at. Adds an inline paragraph above the <pre> explaining what the log is, the row format, when it fills in, and where to find the historical audit trail (/admin/activity). The actual <pre> stays empty until SSE events arrive, but the surrounding copy carries the meaning. * ui(tables): auto-open cache-warmup log on Re-warm all click A Re-warm all run takes ~24s per remote BQ row. With the <details> collapsed by default, operators saw the button disable, watched a quiet ~24s pass, and assumed nothing had happened — the streaming log was hidden behind a closed disclosure. Two small JS tweaks: - cacheWarmupRun() opens the details on click, so streamed lines appear without an extra interaction - cacheWarmupOnStart() hides the inline hint paragraph the moment real log content lands, so the dark log block isn't competing with redundant context Hint paragraph also clarifies that only `query_mode='remote'` BQ rows are warmed — operators with only materialized/internal tables would see total=0 and the page would "do nothing" by spec. * ui: trim Agnes internal copy across surfaces Descriptions had grown to explain the extraction pipeline ("parsed out of session JSONLs"), the underlying table ("Backed by usage_session_summary"), the RBAC mechanic ("row-level RBAC at query time — analysts see their own; admin sees all"), and the SQL snippet. Every implementation detail meant another rewrite on the next iter. Strips to one stable line per surface: what the data is, plus "Also available locally for analysis". Mechanics live in code + docs; the page copy says what the user needs to know. Touched: - connectors/internal/access.py: INTERNAL_TABLES descriptions - activity_center.html / admin_usage.html / admin_sessions.html subtitles - catalog.html Agnes Internal card description + row strip - admin_tables.html "Agnes internal" tab hint * fix(internal): is_user_admin arity bugs + + saved-view payload cap Round-1 code review (PR #278) caught two blocking bugs and three nits. Blocking — both `is_user_admin(user)` (single dict arg) calls raised TypeError. is_user_admin signature is `(user_id, conn)`. Affected: - app/api/query.py:_run_internal_query — every POST /api/query that references agnes_sessions / agnes_telemetry / agnes_audit blew up with a 500. The headline analyst-facing feature of this PR was unusable through the API. - app/api/v2_sample.py — same shape; `GET /api/v2/sample/agnes_*` returned 500. Both fixed to call `is_user_admin(user.get("id"), conn)`. Added two FastAPI-level tests in test_internal_data_source.py that go through the TestClient — the existing unit tests on `execute_internal_query` and `build_filter_clause` skipped the request-handler layer where the bugs lived, which is why this landed. Nits also closed: - connectors/internal/access.py: `+` allowed in _USERNAME_RE / _USER_ID_RE so RFC 5321 email local-parts (alice+test@x) resolve correctly without hitting InternalAccessError. - app/api/observability.py: saved-view payload capped at 64 KiB to prevent an admin from bloating system.duckdb with a malformed save. * fix(security): close non-admin data-leak via underlying-table refs PR #278 R2 review surfaced a non-admin-exploitable bypass: SQL whose string literal contains 'agnes_sessions' routed into the privileged internal-query path, then queried the underlying physical table (usage_session_summary / usage_events / audit_log) directly, escaping the CTE wrapper's row filter. Two reinforcing defenses: 1. find_internal_refs() now strips single-quoted string literals before scanning for alias names — a literal alone no longer routes the request into the privileged code path. 2. execute_internal_query() rejects non-admin SQL that references the underlying physical tables (usage_*, audit_log). The CTE wrapper only scopes the agnes_* aliases; a direct FROM on the base table — or a shadowing inner WITH that still has to read the base table — bypasses RBAC. Block before execution with an actionable error pointing to the agnes_* alias. Admins are unaffected (god-mode short-circuit on the filter clause). 3. tests/test_internal_data_source.py — three new negative tests covering literal-only matches, direct-table refs, and CTE shadow attempts. Also tightens usage_ask.py's SELECT-only validator: pragma_table_info, pragma_storage_info, pragma_database_*, and duckdb_tables / columns / views / indexes / schemas are reflection functions that leak metadata the analyst question shouldn't reach. \bPRAGMA\b in _FORBIDDEN never matched the function-call form (word-boundary between `A` and `_`). * fix(security): dynamic denylist for non-admin internal queries R3 review (PR #278) caught a wider data-leak than R2: the underlying- physical-table guard listed only the 7 usage_* + audit_log tables, but system.duckdb has 30+ other sensitive tables — users (emails + ids), personal_access_tokens, resource_grants, user_groups, user_observability_views, store_*, marketplace_*, knowledge_*, etc. A non-admin SQL like SELECT * FROM agnes_sessions UNION ALL SELECT email, id, … FROM users LIMIT 1 would leak every user's row. Replaces the hardcoded denylist with a **dynamic allowlist** — non-admin SQL may reference ONLY the registered agnes_* aliases. Every other table in `information_schema.tables` (main schema) is rejected. Future migrations that add a new sensitive table are automatically covered without re-editing this module. Also strips SQL comments (`/* */` and `--`) before the identifier scan so a comment-wrapped table name (`/**/users/**/`) can't slip past the regex. Four new negative tests pin: `users`, `personal_access_tokens`, block-comment wrap, line-comment wrap. Plus: per-user view-count cap (100) on /api/admin/observability/views so an admin can't fill system.duckdb with thousands of saved views. * release: 0.54.0 — Activity Center + Telemetry + Sessions + internal datasource Cuts the work shipped across this PR (Activity Center build, recursive internal data source) into a versioned release. Bumps pyproject.toml to 0.54.0; renames the top of CHANGELOG.md from [Unreleased] to [0.54.0] — 2026-05-12 with a header summary; opens a fresh [Unreleased] section for the next round. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-05-12 22:41:19 +02:00
ZdenekSrotyr	99b9379ba3	Merge remote-tracking branch 'origin/main' into worktree-catalog-bq-hotfix ... # Conflicts: # CHANGELOG.md	2026-05-12 11:56:49 +02:00
minasarustamyan	dc5e0e0d11	Marketplace UX overhaul: rich plugin/skill/agent detail + filename rename (#251) ... * Rename agnes-metadata.json to marketplace-metadata.json Curated marketplace enrichment file (.claude-plugin/agnes-metadata.json) becomes marketplace-metadata.json. Clean cut, no fallback — curators of upstream marketplace repos must rename the file on their side. Python API renames mirror the file rename: read_agnes_metadata → read_marketplace_metadata, AGNES_METADATA_REL → MARKETPLACE_METADATA_REL, AGNES_METADATA_MAX_BYTES → MARKETPLACE_METADATA_MAX_BYTES. Synth Claude Code marketplace strip rule (.agnes/** + the metadata file) follows the new filename. * Marketplace detail polish: window cover + 715:310 aspect + helper alignment - Plugin & item (skill/agent) detail hero: 160x160 square cover replaced with a macOS-style window frame (3 traffic-light dots + titlebar label showing the entity name). Body is constrained to 715:310 so curator- uploaded covers no longer crop to a square. Window is 380px wide; meta column and absolutely-positioned top-right install/remove actions stay put. Fallback when no cover_photo_url (translucent gradient + PL/SK/AG initials) is unchanged, just inside the window body. - Inner skill/agent cards in the plugin detail's Internal structure section adopt the same 715:310 aspect (was fixed 78px tall). No window chrome on inner cards — just the matching proportions so covers read consistently across hero, grid tiles, and listing cards. - Curated nested item helper text ("This skill is part of ... — add the bundle to your stack to use it") now stacks UNDER the "Open parent plugin" button instead of being a side-by-side flex sibling in the actions-row. Added align-self: flex-end so the 260px helper box anchors at the right edge of the 300px actions column, matching the button's right edge. * Marketplace My tab: surface the same category + type filters as Flea - Frontend: mp-cat-row and mp-type-row now show on tab=my (previously hidden — type was flea-only, category was flea/curated-only). Curated browse stays plugin-only and continues to hide the type pills. fetchOne() sends the `type` param for tab=my too, so the items endpoint's existing my-branch filter actually receives it. - Backend categories endpoint, tab=my branch: when the type filter is set to skill/agent, skip counting curated subscriptions. Curated plugins are always type='plugin', so they wouldn't survive the items endpoint's type filter; including them in the category counts made the pill numbers overstate what users could actually see in the grid. type=None or type='plugin' keeps the previous behaviour. - CHANGELOG entry under [Unreleased]. * Marketplace plugin detail: render rich content from marketplace-metadata.json Adds five optional plugin-level fields to marketplace-metadata.json and renders them on the curated plugin detail page + listing card: * display_name — friendly h1 / listing-card name / mac-window titlebar label (overrides the technical plugin id) * tagline — punchy 1-line value prop for the hero subtitle and the listing card description (replacing the verbose marketplace.json description on cards) * description — multi-paragraph markdown body, server-side rendered through markdown-it-py and sanitized through nh3 with a description-scoped allowlist (no iframes / no raw HTML / no javascript: links). Powers the "What it does" panel. * use_cases[] — {title, description, prompt} entries that render as a 3-column "When to use it" card grid; each card shows the literal prompt as a code chip so users can copy-paste into Claude Code. * sample_interaction — {user, assistant} dialog rendered in a Claude Code-style dark Catppuccin Mocha transcript panel: monospace user row with a green ">" prompt indicator + sans-serif assistant body with markdown formatting (peach bold, yellow italic, pink inline code, mantle-dark fenced code blocks). All five fields are optional; UI sections only render when populated, so plugins without enrichment look identical to before. Fields are read on-demand from the working tree (cached by mtime per marketplace slug) so curator edits land at the next request without waiting for a sync cycle — same pattern as the existing inner-skill/agent enrichment path. No DB schema bump. Skill / agent rich-content rendering is deferred to a later phase (needs a source-of-truth decision: extend plugin.yml? LLM-generate from SKILL.md / agent.md?). The schema accepts the same fields at skill/agent level today for forward compatibility but the UI ignores them for now. Also: stripped a stale `background-color: var(--bg)` from the global `code` rule in style.css (was making inline code visually disappear on the page background). * Skill / agent detail: render rich content from marketplace-metadata.json Brings the skill/agent detail pages to parity with the plugin detail page. Same rich-content schema (display_name, tagline, description as markdown, use_cases[], sample_interaction) plus two per-item additions: * invocation — curator-provided literal command string. When set, overrides the computed "<manifest_name>:<inner_name>" chip and cleanly supports both "/" skill prefix and "@" agent prefix (the hardcoded "/" in the chip markup is hidden when the curator provides the invocation, so /grpn-eng:query <q> and @grpn-eng:cto-architect both render correctly). * when_to_use — markdown disambiguation block ("Use this for X. For similar Y, see /other-skill") rendered into a new "When to use this" panel below the Example section. Skill / agent category is now per-item overridable in marketplace-metadata.json. When absent, the API keeps the parent plugin's category as the badge so existing items don't lose their category until curators opt in to per-item categorization. The new "Example" Q&A panel uses the same Claude Code-style dark Catppuccin Mocha transcript treatment as the plugin detail — monospace user row with a green ">" prompt indicator + sans-serif assistant body with markdown formatting. All new fields are optional and read on-demand from the working tree. Skills / agents whose marketplace-metadata.json doesn't carry rich content render exactly the same way they did before (frontmatter description + computed slash command + cover from existing v32 enrichment). No DB schema bump. * Fix TypeError in skill / agent detail when curator sets per-item category `curated_skill_detail` and `curated_agent_detail` were passing both `**parent` (from `_curated_inner_parent_fields`, which returns the parent plugin's category as a fallback) and `**enrichment` (from `_curated_inner_enrichment`, which returns the per-item category override when the curator set one) into `InnerDetailResponse(...)`. Python function-call kwargs unpacking with overlapping keys raises `TypeError: got multiple values for keyword argument 'category'` — it doesn't merge like a literal dict does. The bug only surfaced when the marketplace-metadata.json carried a `category` field at skill / agent level (curator opting into per-item categorization); items without that override hit the endpoint cleanly because only parent provided the key. Fix: build `merged = {**parent, **enrichment}` first (literal-dict syntax DOES merge, with the right-hand-side winning) and unpack the merged dict. Curator override still wins via the merge order, and the same pattern is future-proof for any other field that lands in both layers later. Plus a regression test in test_marketplace_metadata.py asserting that the inner-resolver carries `category` for downstream merging. * Marketplace detail: tolerate partial curator JSON Server constructed UseCase / SampleInteraction via raw dict indexing (uc["title"], sample["assistant"]), so a curator commit missing any required Pydantic field crashed the whole plugin / skill / agent detail endpoint with a 500. Route both constructions through _safe_use_case / _safe_sample_interaction helpers — partial input silently drops the malformed card / section instead of breaking the page. Regression test in test_marketplace_api.py covers the three shapes: use_case missing a key, use_case with an empty string, and sample_interaction with only user (no assistant). Sibling rich fields still render. * Address PR-251 review (must-fixes + S2/S3 polish) + release-cut 0.50.0 Five must-fixes from the review pass (3 from @cvrysanek's two-stage review, 2 from my independent pass), plus the 0.50.0 release-cut as the last commit on this PR per CLAUDE.md (CLAUDE.md "Release-cut belongs to the PR" rule added in v0.49.1). Must-fixes ---------- 1. Cache eviction: bounded LRU instead of per-marketplace predicate. The previous predicate (`k[0] == marketplace_id and k[1] != mtime_ns`) only swept stale entries for the CURRENT marketplace; with N>100 distinct marketplaces each holding one mtime key, the cap silently failed and memory grew linearly. Replaced with OrderedDict-backed bounded LRU at cap=256, drop oldest insert on overflow. Cache stress test pinned in test_marketplace_metadata.py. 2. Render CPU cap: per-field byte cap on description / when_to_use / sample_interaction.assistant via MARKETPLACE_METADATA_FIELD_MAX_BYTES (= 64 KiB). Without this, a 1 MiB curator markdown body × QPS = curator-controlled CPU burn through pure-Python markdown-it-py. Truncation respects UTF-8 boundaries and logs a warning so the curator sees the cap fire on the next sync. Test for cap + UTF-8-boundary preservation. 3. Inner-detail bypassed the metadata cache. _curated_inner_enrichment, _curated_inner_cover, and curated_detail all called read_marketplace_metadata directly, defeating the mtime cache the plugin listing already shared. Routed all three through _read_metadata_cached so skill/agent detail hits are O(1) re-parses per marketplace per mtime instead of O(QPS). 4. Truthy-vs-presence trap in plugin/inner enrichment merge. API-layer writers used `if resolved.get(k):` which silently dropped any future falsy-but-valid resolver field (bool featured=False, int priority=0, str category=''). Switched to presence check (`if k in resolved`) so the resolver is the authority on field presence; `{**parent, **enrichment}` merge respects whatever the resolver decided to ship. 5. Vendor-agnostic OSS cleanup. Removed operator-specific token references (/grpn-eng:, @grpn-eng:, .foundryai/) from src/marketplace_metadata.py docstring, app/web/templates/ marketplace_item_detail.html JS comment, docs/curated-marketplace- format.md, and tests/test_marketplace_metadata.py fixtures. Replaced with generic /my-plugin:tool / @my-agent:role / .example/ placeholders. CHANGELOG --------- - New "### Fixed (PR #251 follow-ups)" section documenting all 4 code-side must-fixes - New "### Internal" section noting the vendor cleanup + new tests - BREAKING bullet for the file rename now covers operator-side migration: running instances see plugin enrichment disappear from the UI until upstream curator renames + nightly sync overwrites the working tree; POST /api/marketplaces/{id}/sync forces refresh sooner - Stripped /grpn-eng: leaks from the existing skill/agent rich-content bullet Tests ----- 128 targeted tests pass (test_marketplace_metadata, test_marketplace_api, test_marketplace, test_markdown_render, test_marketplace_synth_strip, test_marketplace_filter). New tests added: - 6 XSS regression tests on render_safe (javascript:/data:/vbscript: schemes via autolink, reference link, and mixed-case + positive http/https/mailto + noopener noreferrer rel) - 3 byte-cap tests (truncation + UTF-8 boundary + under-cap pass-through) - 1 cache eviction stress test (>256 marketplaces -> bounded at cap) - 1 truthy-vs-presence resolver-contract test Release-cut ----------- - pyproject.toml 0.49.1 -> 0.50.0 (minor; BREAKING file rename per pre-1.0 CHANGELOG note: "breaking changes called out under Changed or Removed with the BREAKING marker") - CHANGELOG [Unreleased] -> [0.50.0] - 2026-05-12, new empty [Unreleased] on top. --------- Co-authored-by: Minas Arustamyan <arustamyan.minas@gmail.com> Co-authored-by: ZdenekSrotyr <zdenek.srotyr@keboola.com>	2026-05-12 08:38:39 +00:00
ZdenekSrotyr	27863f88e2	test(db_schema_version): bump locked SCHEMA_VERSION assertion to 40 + add v40 changelog comment	2026-05-11 21:04:11 +02:00
minasarustamyan	9de679c714	System plugins (schema v39) + marketplace UX polish + drop legacy pages (#241) ... * System plugin tier with mark/unmark fanout (schema v39) Adds a mandatory plugin tier so admins can pin a small set of curated plugins into every user's stack from day one. Marking a plugin via the new toggle on /admin/marketplaces materializes resource_grants for every group and user_plugin_optouts subscriptions for every user, so the existing resolver pulls the plugin into every served set without a new filter layer. Hooks on user-create (Google OAuth, magic-link, admin POST, scheduler) and group-create propagate the same materialization to new principals. UI locks: /admin/access disables the checkbox with a SYSTEM pill; /marketplace cards swap the "In stack" green pill for an amber "Required" badge with shield icon; the plugin detail install button reads "Required by your org"; /my-ai-stack toggle is disabled. Bypass paths return 409 (DELETE /api/admin/grants for system grants, PUT /api/my-stack/curated/.../{enabled:false}, DELETE /api/marketplace/curated/.../install). Unmark only flips the flag — materialized rows persist so admins curate cleanup at their leisure through the now-unlocked /admin/access checkboxes. * Marketplace UX polish + drop legacy /store and /my-ai-stack pages Two-part cleanup post-v39: (1) Page deletion. /store and /my-ai-stack were already replaced by /marketplace?tab=flea and /marketplace?tab=my respectively, but the standalone routes lingered. Hard delete in dev mode — no redirects, stale bookmarks 404. The /store/new upload wizard, the flea detail/edit pages, the admin queue, and all /api/store/* + /api/my-stack endpoints (CLI consumers) stay. Internal hardcoded hrefs in the upload wizard's Cancel button and the advanced-setup page repointed to the marketplace tabs. (2) Detail-page install button rework. The single button that morphed between "+ Add to my stack" and "✓ In your stack" did not communicate uninstall affordance. The installed state now renders an inline white status label *before* a separate red-bordered "✕ Remove from stack" button on the same row, both at identical height to avoid layout shift. System plugins keep their locked amber "✓ Required by your org" pill (no Remove button — API refuses 409). The post-action hint panel now fires on remove too with the title flipped to "✓ Removed from your stack" — Claude Code needs the same /update-agnes-plugins refresh either way. Also: /admin/marketplaces Details modal "Mark as system" toggle redesigned. The button was near-invisible (matched neutral row metadata). It's now a balanced amber-toned chip with shield icon and a structured confirm modal replacing the native confirm() dialog that summarizes fanout consequences before commit. * Move stack-hint inside hero with glass-on-gradient styling The post-action hint card ("✓ Added to your stack" with the /update-agnes-plugins recipe) used to live below the hero in panel-what (gray card on white page body). Clicking add/remove inserted/removed it between the hero and content, shifting the panels below — a noticeable scroll jump. The hint is now anchored inside the hero's top-right corner alongside the install/remove buttons, both as flex children of an absolutely positioned .actions container. The card uses a translucent white-on-glass treatment that adopts the hero's kind color (blue for plugin, green for skill, purple for agent) without per-kind branching. Hero is always tall enough (160px photo) to contain the action+hint stack without overflow, so toggling the hint visibility doesn't grow the hero or shift body content. The hero-head grid reserves a third 300px column for the absolute actions overlay so meta gets the proper 1fr free space instead of being squeezed by a padding-right hack. Responsive breakpoint at 1100px reflows the actions stack below hero-head when the viewport isn't wide enough to keep meta + actions side-by-side comfortably. * Add optional -DataPath bind mount to run-local-dev.ps1 When the operator wants to inspect DuckDB files (system.duckdb, extracts, marketplaces, store/, …) directly from Windows Explorer, the named volume inside the Docker Desktop WSL VM isn't reachable. The new -DataPath param generates a transient compose override that rebinds /data on app, scheduler, extract (and Caddy's /srv:ro mirror) to a Windows host folder. Fully additive — when -DataPath is omitted everything behaves exactly as before: no override file is generated, $composeFiles array is unchanged, finally cleanup is a no-op. Existing positional invocations (.\run-local-dev.ps1 up | down | logs) keep binding to $Action because $DataPath is a named-only parameter with no Position attribute. The override is written via [System.IO.File]::WriteAllText so the YAML is BOM-less across PS 5.1 / 7+ — Compose rejects BOM-prefixed YAML on Windows. The override file is unique per PID and removed in the script's finally block so concurrent invocations and crashes don't leak files. * factor mark_system fanout into UserCuratedSubscriptionsRepository The endpoint imported UserCuratedSubscriptionsRepository, ignored it (noqa: F841), then duplicated the user-side fanout SQL inline. Adds fanout_system_for_plugin() symmetric to the existing fanout_system_for_user() and routes mark_plugin_system through it — removes the dead import + 14 lines of inline SQL, returns the same `affected_users` delta count, no behavior change. * drop customer-specific path from .ps1 example Per CLAUDE.md vendor-agnostic OSS rule: replaced C:\\Business\\Groupon\\Agnes\\agnes-data with the generic C:\\Users\\<you>\\agnes-data placeholder so the docstring example reads cleanly on any reviewer's box. * release: 0.48.0 + parallelize Release-workflow pytest Cuts the release shipped via #228 #230 #231 #232 #233 #234 #236 #237 #238 #239 #240 plus this PR (#241). Major changes: - System plugin tier (schema v39) — admins mark a plugin mandatory; fans out RBAC grants + subscriptions to every existing user/group plus hooks for new principals - BREAKING: removed standalone /store + /my-ai-stack page routes (replaced by /marketplace?tab=flea + /marketplace?tab=my) - Setup-prompt + bootstrap recovery fixes (#240) - DuckDB CHECKPOINT-on-shutdown + 60s compose grace (#235) - Marketplace + flea-market UX polish, agnes-metadata.json enrichment Bonus: switch release.yml test step to `-n auto` (matches ci.yml). Single-threaded was 15-20 min and frequently the bottleneck on PR mergeability — now ~6 min. --------- Co-authored-by: Minas Arustamyan <arustamyan.minas@gmail.com> Co-authored-by: ZdenekSrotyr <zdenek.srotyr@keboola.com>	2026-05-10 19:15:41 +00:00
Vojtech	929520f5e1	Flea-market edit feature with version history (schema v37) (#239) ... * feat(store): flea-market entity edit feature with version history (schema v38) Owner + admin can now edit a store entity from a real Edit page at /marketplace/flea/{id}/edit, replacing the prior "coming soon" placeholder. Editable: display name, description, category, video URL, cover photo, and an optional new bundle. Type is locked (400 type_locked). Display-name change renames the on-disk slug for both live plugin/ and version dirs (reuses rename-on-archive helper). Schema v38 (originally drafted as v37; renumbered after rebase onto main where v37 was taken by the curated marketplace enrichment). Versioning model: * Each bundle update bakes into ${DATA_DIR}/store/<id>/versions/v<N+1>/plugin/ and runs the standard guardrails pipeline. * DEFERRED PROMOTION: live plugin/ + entity.version_no stay at the prior approved version through the LLM review window so existing installers keep receiving the previously approved bundle. Live swap + version_no/version/file_size bump happen only on LLM approval. Blocked verdicts leave the prior version serving forever. * store_entities gains version_no INTEGER + version_history JSON. Each version_history entry carries hash, sha256, size, submission_id, created_at, created_by. * Existing entities backfill to v1 with a single-entry history seeded from the row's current `version` hash. Initial create also seeds versions/v1/plugin/ so future restore can copy v1 bytes forward. Concurrency: * Block-while-pending: an in-flight LLM review blocks any further edit with 409 prior_version_pending. Owner waits 5-30s; Edit button on detail page renders disabled in the same window via the new edit_in_flight flag (decoupled from quarantine_sub since the deferred-promotion flow keeps visibility='approved'). Rollback: * New endpoint POST /api/store/entities/{id}/versions/{n}/restore (owner + admin). Copies vN bundle forward as v<max+1> and re-runs guardrails (rules tighten over time; pre-approved bundles re-validate). Forward-only history. Same deferred-promotion semantics — live stays at prior version until LLM approves the restored copy. UI: * New /marketplace/flea/{id}/edit page (owner + admin gated). * Versions card on plugin + item detail templates (owner/admin only) via shared _flea_versions.html partial. * Admin queue gains v# column with current badge + separate Hash column. Submission detail surfaces Version + Bundle hash rows. * Activity timeline split into per-submission + entity-wide cards; entity-wide rows render vN chips when audit row params reference a specific version. * Section headers (Manifest / Static / Quality / LLM review) tag with vN chip via shared macro. * Reviewed-by-model field surfaces explanatory text per status. * Banner upload-failure now redirects to detail page on submission_blocked instead of staying stuck. Tests: 24 in tests/test_store_entity_versions.py covering metadata- only edit, bundle-edit version bump, type lock, block-while-pending, name change disk rename, restore flow + 404/400/403 paths, edit page 404 for non-owner, versions card visibility gating, admin queue v# column, admin detail Version/Hash rows, deferred-promotion installer contract (pending review doesn't break installer / blocked verdict keeps prior / approved promotes), admin can edit/restore non-owned, restore deferred promotion, audit log per-version params. 214 tests green across guardrails + edit + admin + repo + schema suites. * docs(store): refresh update_entity docstring to match deferred-promotion + submission-status gate Bring the docstring in sync with the actual fixes from the prior commit. The pre-fix wording said the gate read visibility_status='pending' AND submission status — under deferred promotion that would never fire for v2+ edits. Now describes: - Block-while-pending gates on submission.status DIRECTLY, independent of visibility (so v2+ deferred-promotion edits don't slip through). - Display-name + bundle change defers the live rename to promotion; metadata-only renames stay immediate. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-10 00:14:33 +04:00
minasarustamyan	6fe67d5279	Curated marketplace enrichment via agnes-metadata.json + curator metadata (#234) ... * Curated marketplace enrichment via agnes-metadata.json + curator metadata Adds a second well-defined metadata file `.claude-plugin/agnes-metadata.json` that upstream marketplace repos can opt into, providing per-plugin (and per-skill / per-agent) cover photo, demo video URL, doc links, and category override. The Claude Code marketplace contract is untouched — agnes-metadata.json + the convention `.agnes/` directory are stripped from the synthetic Claude Code marketplace served via /marketplace.zip and /marketplace.git/*, so user instances see a clean Claude Code repo with no Agnes-only metadata. Highlights: - DB schema v32 — adds curator_name + curator_email on marketplace_registry, cover_photo_url + video_url + doc_links on marketplace_plugins. - Mandatory curator at marketplace registration, editable later through the admin UI; surfaces on cards + detail pages in place of owner_todo. - External-asset mirror cache at ${DATA_DIR}/marketplace-cache/<slug>/ with conditional GET, 60s timeout, 10 MB body cap, SSRF guards, and Wikipedia-policy-compliant User-Agent. - Strict drop semantics — anything Agnes can't deliver as a real PDF / Markdown / plain text doc, or a real PNG / JPEG / WebP cover, is dropped from the served metadata; UI looks identical to no-entry case (gradient placeholder for missing covers, no row in the doc list). - Doc allowlist + image allowlist enforced on both the curated mirror flow and the Flea upload flow (/store/new); shared module src/marketplace_assets.py. - New /api/marketplace/curated/{mp}/{plugin}/{asset,doc,mirrored}/... endpoints with path-traversal guards + RBAC + Content-Disposition attachment for docs. - Curator-focused format guide at /marketplace/format-guide; canonical source is docs/curated-marketplace-format.md, also linked from the admin /admin/marketplaces page next to + Add Marketplace. See CHANGELOG.md under [Unreleased] for the full breakdown. * Fix format-guide test assertion to match shortened disclaimer The 'Flea Market' phrase was trimmed out of the disclaimer in docs/curated-marketplace-format.md after the curator-focused rewrite. Update the rendered-HTML test to assert the channel-scoping phrase that's actually present ('Curated Marketplace channel only') rather than the 'Flea Market' contrast that's no longer in the doc. * Drop unused 'version' field from agnes-metadata.json schema The parser never read it; it was a YAGNI placeholder for future schema evolution. Curators don't need to wonder what to put there when adding the file for the first time. Will be re-added if and when we actually introduce a backwards-incompatible schema change. * Harden asset mirror against SSRF via redirect + DNS rebinding The pre-flight _is_safe_url check validated only the initial URL; urllib.request.urlopen then followed redirects and re-resolved DNS for the actual connection — both bypassable. Attacker-controlled origin could 302 to http://169.254.169.254/... and exfil cloud metadata; attacker-controlled DNS could return public IP first / 127.0.0.1 second. Replace urlopen call with a shared OpenerDirector wired through three custom handlers: _SafeRedirectHandler re-runs SSRF allowlist on every redirect Location (max 5 hops, down from urllib's 10), and _PinnedHTTPHandler / _PinnedHTTPSHandler connect to the IP that passed validation rather than re-resolving the hostname. TLS SNI + cert verify stay bound to the original hostname. _resolve_safe returns the validated IP (the existing _is_safe_url 2-tuple wrapper stays for backwards compatibility) and rejects round- robin DNS that mixes a public + private record. _UnsafeRedirectError is a typed exception so _fetch_url can map redirect blocks to terminal 'rejected' status (not transient 'failed'). _http_open is the single call site so tests can mock at one well-defined seam. Tests cover redirect blocking (link-local, loopback), redirect-error unwrapping inside URLError, pinned-IP connection target, and the end-to-end DNS-rebinding scenario. Existing tests that mocked urllib.request.urlopen are migrated to mock _http_open. * Harden /asset/ endpoint against stored XSS The endpoint served any file in the cloned marketplace repo with stdlib-detected Content-Type, so a curator who landed evil.html (or a renamed evil.png carrying HTML bytes) in the working tree got a same-origin XSS — the response shares cookie scope with /admin and /api/me/*. The asset endpoint is image-only by contract (cover photos referenced from agnes-metadata.json + inner skill / agent cards), so applying the same allowlist + magic-bytes pattern that /doc/ already uses closes the gap without breaking any legitimate use case. Three layered checks: extension in IMAGE_EXTENSIONS (.png/.jpg/.jpeg/.webp; SVG excluded — <script> inside SVG executes), validate_image_file magic bytes (defeats rename-extension attack), Content-Type pinned from the validated extension (never stdlib mimetypes). Defense-in-depth: X-Content-Type-Options: nosniff stops browser MIME sniffing; Content-Security-Policy: default-src 'none' blocks script / iframe execution even if a future regression let HTML through. Tests cover the .html extension reject, the renamed-HTML-as-PNG magic- bytes reject, the .svg reject, and the happy-path PNG with security headers attached. The pre-existing path-traversal test seeds a real PNG instead of ok.txt now that the endpoint is image-only. * Enforce mandatory curator on marketplace PATCH The POST handler enforced curator_name + curator_email at create time, but PATCH treated empty / missing curator inputs as 'no change'. Legacy rows that pre-date v32 (curator_name=NULL) could be edited indefinitely without ever filling the curator gap, and OWNER_TODO_PLACEHOLDER lingered on every /marketplace card. Reject the PATCH with 400 when the post-merge row would persist with empty curator. The check fires after the existing field-merge logic, so once-filled rows that don't touch curator still pass through (their existing values fall through from the DB row). DB column stays nullable so untouched legacy rows continue to coexist — the gate fires only the moment an admin opens the edit modal. Existing PATCH semantics preserved: empty-string input still means 'leave existing value alone', and once-filled curator can't be cleared (those test cases pass unchanged). New test seeds a legacy row directly via the repository, then exercises url-only PATCH (rejected), partial-fill PATCH (rejected), and full-fill PATCH (succeeds); a follow-up no-curator PATCH on the now-formed row also passes. * Drop unused curated-marketplace helpers (PR #234 review) * build_db_payload — imported by src/marketplace.py but never called. The strict-drop semantics it would have implemented were re-written inline in _refresh_plugin_cache (see the comment block there). The standalone helper still carried the old fall-back-to-original-external- URL-on-mirror-failure behaviour, which contradicts the documented drop-when-can't-deliver contract — a future contributor who re-wired it would have introduced a silent regression. Delete with the helper + the import + the comment that referenced it. * _resolve_marketplace_name — one-line shim with no remaining call sites. Callers use _resolve_marketplace_meta which returns name + curator together, avoiding the double DB hit the shim exists to hide. * '# noqa: F401 Optional kept for forward-compat' was wrong — Optional IS used in src/marketplace.py (line 70 and line 238). Drop the noqa comment so a future ruff run doesn't try to remove a real import. Removing build_db_payload also drops the only remaining use of Optional in src/marketplace_metadata.py, so the import comes out there too. * Cap agnes-metadata.json size + catch RecursionError on parse The reader is invoked once per marketplace per sync and the file is curator-controlled. Two failure modes were unguarded: * Multi-GB JSON: path.read_text() pulled the whole file into memory before json.loads even ran. A curator with commit access to an upstream repo could OOM the sync worker. * Deeply-nested JSON under any size cap: cpython's recursive object / array parser raises RecursionError at ~1000 levels of depth. RecursionError is a RuntimeError, not ValueError, so the existing catch let it propagate up and abort the entire sync — every other marketplace in the same pass got skipped. Add AGNES_METADATA_MAX_BYTES = 1 MiB (a real metadata file with covers, docs, categories for ~50 plugins fits in <100 KB so the cap is generous) and gate the size check on path.stat().st_size before the body read. Broaden the parse except to (ValueError, RecursionError) with a unified log line. Both failure modes degrade to the same empty-dict fall-back the malformed-JSON path already used, so one bad upstream never aborts the rest of the sync. Tests cover the size cap firing before json.loads (whitespace-padded valid JSON exceeding the cap) and the recursion path (5000 nested arrays — past cpython's default recursion limit but well under the size cap). * Persist asset-mirror manifest per body write, before unlink sync_assets wrote each body atomically (tmp + rename) but persisted the manifest only at the end of the batch. A kill -9 mid-Phase 2 left on-disk files the manifest never referenced. Once a curator dropped that URL from agnes-metadata.json, Phase 3's cleanup had no record of the file and the orphan stayed forever — there's no GC pass walking the cache dir today, so disk would slowly bloat. Phase 2 (body-write iteration): after the in-memory manifest mutation, persist BEFORE unlinking the previous body. The crash window narrows from 'all of Phase 2' to 'between persist and unlink' (microseconds). A persist failure mid-batch keeps the previous body on disk — the on- disk manifest still references it, and a stale-but-existing file beats a 404. Cost: one extra tmp+rename per body write; manifest is a few KB so the overhead is negligible vs. the HTTP fetches. Phase 3 (curator-removed URLs): same discipline. Collect the to-delete relpaths, persist the manifest with the entries already gone, THEN unlink. A crash mid-cleanup leaves at most a microsecond window where files exist despite the manifest no longer naming them. The next sync reads the (correct) manifest and the orphan stays orphaned, but the served state is consistent. Tests cover per-body persist call count, the post-update on-disk manifest content, and Phase 3 ordering verified by reading the on-disk manifest from inside Path.unlink. * Consolidate marketplace video embeds + format-guide CSS The YouTube nocookie / Vimeo / <video> / link-fallback detection logic was duplicated verbatim in marketplace_plugin_detail.html and marketplace_item_detail.html (~40 JS lines each, with subtly-different inline styles). Both templates now {% include %} a single _marketplace_video_embed.html partial inside their IIFE so the regex, the nocookie attribute set, and the unknown-host link fallback live in ONE place — future tweaks (new host, new attribute, fixed sandbox flag) no longer need to be applied twice in lockstep. The .video-wrap selectors (one inline <style> rule in plugin_detail, one inline style='...' attribute in item_detail) are replaced by the existing .video-embed 16:9 wrapper in style-custom.css, with new .video-embed video / .video-embed a child rules added so the wrapper handles all four embed shapes uniformly without per-template positioning. The 60-line inline <style> block in marketplace_format_guide.html moves verbatim to style-custom.css under a new 'Marketplace format guide page' section, scoped to .format-guide so other pages aren't affected. No user-visible behaviour change: the rendered HTML for valid YouTube / Vimeo / mp4 / external links is byte-identical to before, and the format-guide page renders the same. * Maintainability cleanup batch (PR #234 review) #10: drop _path_under from app/api/marketplace.py — it was a byte- equivalent clone of _safe_join (same Path.resolve(strict=True) + relative_to() containment check). The three v32 endpoint handlers (/asset, /doc, /mirrored) now share the existing helper. #14: rename src/marketplace_assets.py → src/marketplace_asset_validation.py so the file's purpose is obvious from the name and the previous overlap with src/marketplace_asset_mirror.py is gone. Six call-site imports updated in lockstep; CHANGELOG references under [Unreleased] updated to track the new path. #11: consolidate the URL builders that resolve /api/marketplace/curated/<slug>/<plugin>/{asset,doc,mirrored}/... paths. _internal_asset_url / _internal_doc_url / _mirrored_asset_url lived in src/marketplace.py, while a copy named _mirrored_url lived in app/api/marketplace.py with a 'must stay aligned' comment. New module src/marketplace_urls.py is the single source of truth — both call sites import from it and a future URL-format tweak only needs to change one file. The _ROUTE_PREFIX constant collapses the per- function f-string repetition. The route-handler endpoints themselves still own the path string literals (keeping the builders identical to the route declarations remains a checklist item, not a runtime guarantee). * Re-key asset-mirror manifest by (plugin, url) + dedup HTTP fetches The manifest used to be keyed by URL alone, so two plugins in the same marketplace referencing the same external image (a shared CDN icon, a common cover) collided on entry.plugin_name — last writer won. The DB row for the losing plugin then stored a served URL pointing under the winning plugin's tree, and require_resource_access denied legitimate access on one side and let the other plugin's user reach the wrong asset. In-memory: Dict[Tuple[str, str], MirrorEntry] keyed (plugin_name, url). On disk: format flips from {url: entry} dict to [entry, ...] list of self-describing entries (each carries plugin_name + url + the previous fields). JSON keys can't be tuples; encoding 'plugin::url' would just shift the parsing burden. Phase 1 of sync_assets deduplicates fetches by URL — three plugins sharing one URL share one HTTP request. The conditional-GET prior is picked from any owning plugin's prior entry; if their etags diverge (rare) we miss one 304 and pay for a full re-download instead. Phase 2 still creates a per-(plugin, url) manifest entry pointing under the plugin's own subdir, and Phase 3 cleanup is keyed the same way so dropping a URL from one plugin's metadata doesn't disturb another plugin still referencing it. Body files stay per plugin (RBAC-clean isolation: deleting plugin A's cache can't strand plugin B). Bandwidth saved by fetch dedup. Consumer code re-keyed: src.marketplace._refresh_plugin_cache rebuilt served_url_for / mirror_status as composite-keyed maps; app.api.marketplace._resolve_external_via_mirror / _curated_inner_cover / _curated_inner_enrichment look up by (plugin_name, url). Tests cover per-plugin manifest entries with shared URL, the single HTTP fetch for N plugins, and Phase 3 drop-one-keep-other. All existing tests migrated to composite key access; v2 list format assertions verify on-disk shape. * Migrate asset mirror from urllib.request to httpx The asset mirror was the only HTTP call site in Agnes still using urllib.request; every other module (CLI, Jira / OpenMetadata / OpenAI connectors, scheduler, Telegram bot) already used httpx. The asset mirror was added in this PR's base commit, so this is the only chance to bring it into convention before someone copies it as 'the pattern for HTTP fetches in Agnes'. Three concrete benefits beyond consistency: * SSRF defence collapses from five urllib classes (_PinnedHTTPConnection, _PinnedHTTPSConnection, _PinnedHTTPHandler, _PinnedHTTPSHandler, _SafeRedirectHandler) into one _SSRFGuardTransport. httpx invokes handle_request() on every redirect hop, so re-validation is free — we don't need a custom redirect handler at all. * DNS-rebinding defence: the transport rewrites request.url.host to the SSRF-validated IP before delegating to super().handle_request(). httpcore connects to whatever URL.host says, so this pins the connection without subclassing HTTPSConnection. The original hostname goes into the Host header + the sni_hostname extension so TLS / vhost routing still bind to the curator-supplied hostname. * Error handling: one httpx.HTTPError catch-all for transport errors, plus specific httpx.TimeoutException / httpx.TooManyRedirects branches for clearer diagnostics. Matches the _translate_transport_error shape in cli/client.py. The shared httpx.Client is built lazily at module load (same pattern as cli/client.py:_get_shared_client) with follow_redirects=True, max_redirects=5, timeout=HTTP_TIMEOUT_SEC, and our custom transport. Externally observable behaviour is unchanged: same FetchOutcome statuses, same manifest format, same conditional GET semantics, same body-size cap. Tests migrated from urllib-shaped fakes to httpx-shaped (status_code, iter_bytes, context manager). Five urllib-specific tests replaced with httpx equivalents — three transport unit tests + one DNS-rebinding integration test that verifies host rewrite via monkey-patched super().handle_request. One test deleted without replacement (unwrap-URLError-wrapping-an-_UnsafeRedirectError — urllib-specific, not applicable to httpx). * Surface curated agnes-metadata enrichment on My Stack tab GET /api/marketplace/items?tab=my built each curated row from the on-disk marketplace.json by way of resolve_allowed_plugins, which doesn't carry the agnes-metadata enrichment columns (cover_photo_url, video_url, category override, doc_links). The handler then hard-coded cover_photo_url=None on the synthetic row. Result: once a user clicked '+ Add to my stack' on a curated card, the same plugin in tab=my rendered with the gradient placeholder instead of its cover photo — confusing parity break vs. the curated tab where the same row goes through MarketplacePluginsRepository and gets the enriched columns. Pre-load the enriched marketplace_plugins rows for every marketplace the user is subscribed to, then look each granted+subscribed plugin up by (marketplace_id, plugin_name). Fall back to the on-disk synthetic shape only when the DB row is missing — happens during the rare race where RBAC is granted before the first sync cycle ingests the plugin. RBAC gating (granted set from resolve_allowed_plugins) is unchanged so this fix can't widen visibility; it just upgrades the data shape behind cards the user was already going to see. Per-marketplace list_for_marketplace beats N gets — typical user is subscribed to <5 marketplaces, so this is at most a handful of queries vs. one per subscribed plugin. Regression test seeds a plugin with cover_photo_url + category override, subscribes the user, hits /api/marketplace/items?tab=my, and asserts photo_url + category come through. The misleading 'fall through to gradient until the user re-visits the curated tab' comment is gone. --------- Co-authored-by: Minas Arustamyan <arustamyan.minas@gmail.com>	2026-05-09 17:01:37 +02:00
minasarustamyan	a6b647dda9	Make v34 to v35 store_entities migration idempotent (#236) ... The original list-form _V34_TO_V35_MIGRATIONS ran four ALTER statements in sequence: ADD _vis_v35 → UPDATE _vis_v35 = visibility_status → DROP visibility_status → RENAME _vis_v35 TO visibility_status If the RENAME failed for any reason after the DROP succeeded — DuckDB lock contention at startup, scheduler-vs-app race opening system.duckdb, container kill mid-migration, etc. — the DB was stranded with _vis_v35 populated and visibility_status missing. The schema_version row never bumped because the UPDATE at the bottom of the migration ladder runs only when every step succeeded. Subsequent restarts then hit DROP visibility_status again with no IF EXISTS guard and looped on the same error; the only recovery was hand- editing the DB. Replace the list with a Python function _v34_to_v35_migrate that inspects the table's columns up front and dispatches into one of three paths: * clean v34 (visibility_status present, _vis_v35 absent) — run the full rebuild * partial v35 (_vis_v35 present, visibility_status absent) — finish the RENAME alone, data is already in _vis_v35 from the prior UPDATE * both columns present (rare; aborted before DROP) — drop the temp and keep visibility_status The audit columns (archived_at, archived_by) ship first behind IF NOT EXISTS so they're safe in all states. Operators stranded by the original bug now recover automatically on next startup. Tests cover the three direct paths plus an end-to-end scenario where _ensure_schema walks a schema_version=32 DB with the half-applied state up through to v36. Co-authored-by: Minas Arustamyan <arustamyan.minas@gmail.com>	2026-05-09 16:38:29 +02:00
Vojtech	d6ad08f107	Flea-market upload guardrails + soft delete + JOIN-based admin queue (#233) ... * feat(store): flea-market upload guardrails + soft delete + JOIN-based admin queue Adds an end-to-end guardrails pipeline for store uploads (manifest + static-security + LLM review), persists blocked bundles for forensics, introduces soft-delete (Archive) semantics, consolidates the legacy /store/{id} surface into /marketplace/flea/{id}, and reworks the admin queue so lifecycle filters read live entity visibility via LEFT JOIN rather than a denormalized submission column. Schema v29 → v35: * v29 store_submissions table + store_entities.visibility_status * v30 file_size, bundle_sha256, bundle_purged_at on submissions * v31 reshape store_submissions (drop legacy unique on entity_id) * v32 store_entities.archived_at/by + 'archived' visibility value * v33 drop store_submissions.retry_count (unused) * v34 ensure idx_store_submissions_entity exists post column-drop * v35 broaden visibility_status enum + JOIN architecture cutover Pipeline (src/store_guardrails/): * Inline checks: manifest_check, static_scan, quality_check * LLM review configurable haiku|sonnet|opus (default haiku) * BackgroundTasks-driven async path with structured-output JSON * Per-submitter daily quota (default 50) * 30-day TTL purge job (POST /api/admin/run-blocked-purge) * Bundle SHA256 + size persisted; sha256 survives purge for forensics Visibility model: * pending | approved | hidden | archived * _enforce_visibility returns 404 (no leak) for non-owner non-admin * Owner sees own non-approved entries via include_owner_id widening * Install refused with 409 entity_not_approved when not approved Soft-delete (DELETE /api/store/entities/{id}): * Default = soft (visibility_status='archived'); existing installs keep getting served the bundle so users don't lose the plugin * ?hard=true admin-only: drops bundle + cascades user_store_installs * Hard-delete preserves entity_id on submission as tombstone so audit_log linkage survives for the activity timeline Admin queue lifecycle (the JOIN refactor): * Verdict (store_submissions.status) is immutable forensic record * Lifecycle (store_entities.visibility_status) is live state * /admin/store/submissions Archived chip translates to `e.visibility_status='archived'` via LEFT JOIN — any path that flips visibility surfaces in the queue immediately * Detail page renders Status (verdict) and Entity lifecycle side by side so admins see "approved at review, now archived" at a glance URL consolidation: * /store/{id} deleted (no redirect, stale bookmarks 404) * /marketplace/flea/{id} is the canonical detail surface * Three in-tree callers (upload-success, my-stack card, store listing card) updated to point at the new URL * Quarantine banner extracted to _quarantine_banner.html partial, self-guarded, included from both flea detail templates * Banner JS auto-refreshes when the verdict lands by polling /api/marketplace/flea/{id}/detail (visibility_status + submission_status — the latter is needed because blocked_llm keeps the entity at visibility_status='pending') Audit log resource format: * runner.py emits prefixed `store_submission:{id}` (post-fix) * Detail-page timeline query handles three patterns: prefixed submission, helper-emitted `store_entity:{sub_id}`, and bare-id legacy rows — all surface in the activity timeline UX fixes: * Owner sees Under review / Quarantined / Hidden banner with status * Install button gray-disabled (not blue) when non-approved * Owner cannot delete quarantined entries (403); admin can * Admin queue: filter chips, sortable columns, paging, page-size * Auto-refresh queue every 5s while pending rows are visible * Store upload page file picker no longer opens twice (label → input default action collided with explicit JS handler) Tests: 168 passed across the guardrails suites (admin submissions, store API, inline / LLM / purge guardrails, store repositories, marketplace filter, schema version). New regression coverage includes: archive surfaces via JOIN even when API path is bypassed; deleted submission renders activity timeline (tombstone); flea detail surfaces submission_status only for owner/admin; detail page renders Entity lifecycle row; audit log resource format covers both helper and runner paths. * fix(store-guardrails): PR #233 follow-up — prompt injection, atomic PUT, BG race, schema, reaper, sort whitelist Addresses 9 of the 23 findings from the PR #233 review (spec at docs/superpowers/specs/2026-05-09-pr233-guardrails-fixes-spec.md). Merge-gate items #1-#6 plus high-value mediums #7, #9-#12, #23. Architectural items (#8 enum split, #14 factory) and pure maintainability (#15-#22) deferred to follow-ups. Security: * #1 prompt injection — SYSTEM_PROMPT now passed via the SDK's dedicated system= parameter; bundle wrapped in <bundle>...</bundle> sentinels declared data-only by the system prompt; literal sentinel strings in user content are escaped so an adversarial README can't forge a close tag. * #6 static scan honesty — module docstring + admin copy + docs declare static scan as signal not gate; .md/.txt/.rst/.html/.json/ .yaml/.yml/.toml skipped to avoid false positives on prose. AST mode for Python deferred (separate flag, FP comparison work). Correctness: * #2 PUT atomicity — bundles bake into plugin.staging-<rand>/ alongside live, atomic-rename on success; failed checks leave live tree byte-for-byte intact. * #3 BG-task race — set_visibility_if_pending guards verdict flips to the (pending, hidden) review window; admin archives during review survive; skipped flips audit-logged. * #4 v35 NOT NULL/DEFAULT — schema v35→v36 re-applies them on store_entities.visibility_status. CHECK constraint enforced application-side (DuckDB ADD CHECK on existing column unsupported). * #7 stuck-review reaper — reap_stuck_llm_reviews flips pending_llm rows older than guardrails.stuck_review_grace_seconds (default 1800) to review_error. Scheduler runs every 15 min via new /api/admin/run-reap-stuck-reviews. Set knob to 0 to disable. * #9 quota counter — count_blocked_for_submitter_since now counts blocked_inline + blocked_llm + review_error so a submitter triggering only LLM-blocked verdicts is bounded. * #10 missing risk_level — surfaces as review_error with error='missing_risk_level' instead of silently defaulting to 'medium' (which looked like a model-decided block). * #11 archived_at clear — set_visibility nulls archived_at + archived_by when transitioning out of 'archived' so a future read doesn't show stale archive forensics on an approved row. Maintainability: * #12 FSM doc comment — accurate insert/transition/lifecycle description in src/db.py near store_submissions schema. * #23 sort-key whitelist — admin queue rejects unknown sort keys with 400 invalid_sort_key; substring-replace footgun removed. Deferred (separate PRs): * #5 quota race — proper fix requires asyncio.Lock spanning the full pipeline; threading.Lock blocks event loop, DuckDB MVCC doesn't help. API-level slowapi bounds worst case for now. * #6 part 3 (AST static scan), #8 (enum split), #13 (import bundle docs), #14 (factory consolidation), #15-#22 (maint). Tests: * New: tests/test_store_guardrails_prompt_injection.py (corpus + trust-boundary invariants), tests/test_store_put_atomic.py, tests/test_store_guardrails_reaper.py. * Extended: test_store_guardrails_llm.py (system param, missing risk_level, BG race), test_admin_store_submissions.py (quota counter widening, sort whitelist 400), test_store_repositories.py (un-archive metadata clear), test_db_schema_version.py (v36). * Full suite: 3738 passed; 17 pre-existing baseline failures unchanged (db migration tests, cli binary rename, catalog export, user mgmt v5 backfill — confirmed by stash + rerun on clean tree).	2026-05-09 17:32:53 +04:00
minasarustamyan	e26236fdc1	Extract session-pipeline framework + UsageProcessor skeleton (#232) ... * Extract session pipeline framework, refactor verification, add UsageProcessor skeleton Pluggable framework under services/session_pipeline/ (contract + lib + per-processor runner) so multiple processors can read /data/user_sessions/<key>/*.jsonl on their own cadence with full failure isolation. Verification flow becomes the first plugin; a no-op UsageProcessor reserves the second slot pending a separate brainstorm on extraction logic + storage shape. Schema v28→v29: rename session_extraction_state → session_processor_state with composite PK (processor_name, session_file). Existing rows copied over with processor_name='verification'; legacy table dropped. Migration is idempotent and no-ops the copy step on fresh installs that came up at the new schema. Endpoint: /api/admin/run-verification-detector replaced by parametrized /api/admin/run-session-processor?processor=<name>. Audit action format follows. Scheduler JOBS: verification-detector entry split into session-processor:verification + session-processor:usage. SCHEDULER_VERIFICATION_DETECTOR_INTERVAL retained for operator compatibility (drives both cadence and health-check grace window); SCHEDULER_USAGE_PROCESSOR_INTERVAL added. * Address PR #232 review: scan dead branch + per-processor lock - `SessionProcessorStateRepository.scan_unprocessed_for` dead else: both branches surfaced every jsonl, the SELECT was unused, runner MD5-rehashed every stable session per tick. Replaced with an mtime precheck — stable sessions (mtime <= processed_at) are filtered at scan; modified files still surface for the runner's authoritative `file_hash` invalidation. Naive-local comparison matches the existing health-check idiom (DuckDB TIMESTAMP strips tz on storage). - Per-processor advisory lock around `_run_processor` in `/api/admin/run-session-processor`. Scheduler tick + manual admin POST could otherwise both run, both call create_evidence on overlapping detections, and accumulate duplicate verification_evidence rows (the dedup short-circuit only covers create+contradiction, not evidence per ADR Decision 3). Non-blocking acquire → 409 Conflict on concurrent invocation; release in finally so a runner exception doesn't wedge the processor. Tests: two new scan unit tests (mtime filter + post-mark mtime bump), 409 endpoint test, lock-released-on-exception test. Two existing tests updated for the new "filtered at scan" stat shape (previously asserted skipped == 1, now scanned == 0). * Address PR #232 review #2: parallel scheduler tick + last_run on terminal state Two pre-existing scaffold bugs in services/scheduler/__main__.py amplified by adding more session-pipeline jobs: 1. Serial for-loop over jobs with synchronous httpx.post(timeout=900) — a 10-minute verification run blocked every other job (data-refresh, health-check, usage, corporate-memory) for the whole window. The PR's stated isolation guarantee held inside the runner but broke at the scheduler dispatch layer. 2. last_run advanced only when _call_api returned True. Permanent-failure jobs hot-looped on every tick (30s) instead of cadence (15min). Fix: ThreadPoolExecutor.submit per due job + per-job in_flight set so a long-running job can't be re-launched on subsequent ticks. last_run advances unconditionally in finally; errors still surface via _call_api logging + audit_log on the receiving side. _run_job extracted to module-level for unit testing. New tests: - TestRunJobBookkeeping: advances on success / failure / unhandled raise - TestRunLoopParallelism: in_flight protection prevents duplicate launches across ticks for a single slow job --------- Co-authored-by: Minas Arustamyan <arustamyan.minas@gmail.com>	2026-05-08 19:47:46 +02:00
Vojtech	2e2e1a1eca	feat(home): state-aware /home + /setup-advanced + schema v26 (#228) ... * feat(home+news): state-aware /home + /news + admin-edited news section Squash of the vr/home-page feature work for clean rebase onto main. Original 18-commit history preserved in branch backup/vr-home-page-pre-rebase. What's in this PR: **State-aware /home page** - New `/home` route with hero + auto-mode + connectors (Asana / GWS / Atlassian) + lookarounds. Onboarded vs not-onboarded state-machine branches a single template (`home_not_onboarded.html`); the install steps, "Setup a new Claude Code" CTA (90-day PAT mint), and per- connector setup prompts hide once `users.onboarded=TRUE`. A completion badge replaces them. - "Mark me as offboarded" button reverses the flag without an SQL UPDATE. - `users.onboarded BOOLEAN` column added; default FALSE; flipped by the CLI's `agnes init` post-success POST and the `/admin/users` API. - Connector setup prompts pre-check whether the tool is already installed/connected before re-running setup. - GWS scope set widened to include Google Chat (`chat.spaces`, `chat.messages`). **Single template + design tokens** - `dashboard.html` now extends `base.html` via the new `{% block layout %}` opt-out (full-width pages skip the 800px `.container`). Net: every page shares one shell. - `style-custom.css` `:root` extended with `--space-{7,9,10,12}`, `--radius-2xl`, `--shadow-{card,elevated}`, `--text-{muted,disabled}`, `--focus-ring`, `--transition-*`, `--width-{narrow,app,wide}` so inline page styles can migrate incrementally. **Auth redirects honor AGNES_HOME_ROUTE** - `safe_next_path` resolves the configured home route when no `default=` is passed; OAuth callbacks, magic-link clicks, password form, and LOCAL_DEV_MODE shortcuts now land on `/home` (or whatever the operator picked) instead of always /dashboard. **News section + /news permalink + /admin/news editor** - Schema-bumped `news_template` table (single versioned entity, draft + publish gate). `published BOOLEAN` distinguishes draft from public; monotonically-increasing `version` per save; rows >30d pruned on save except the currently-displayed published version. - `/home` bottom-of-page renders the latest published intro with a "Read more →" link to `/news` (which renders the full body). - `/admin/news` editor with sandboxed live preview, versions table, per-row Unpublish, Format-help cheatsheet. - `agnes admin news show / draft / edit / publish / unpublish / versions / export` (CLI). Talks to the live server via the `/api/admin/news/*` endpoints (PAT-authed) — no direct DB access so it coexists with a running uvicorn. - **Optimistic-lock guard**: `agnes admin news publish --version N` and PUT/PATCH endpoints accept `expected_version` and 409 with structured `{error: "version_conflict", expected, actual, actual_by}` when a concurrent admin replaced the draft. Edit refuses to overwrite a draft authored by someone else without `--force` or `--expect-version`. - nh3 (Rust-backed ammonia) HTML sanitizer; iframe pre-pass strips any iframe whose src is not on the YouTube/Vimeo/Loom allowlist; javascript:/data: schemes blocked everywhere. - Author CSS vocabulary: `.news-hero` (blue gradient hero block), `.callout`/`.callout-{info,warn,success,danger}`, `.video-embed`, `.news-section`, `.news-grid-{2,3}`, `.news-cta` — all consolidated in `style-custom.css` under "News content vocabulary (shared)" so /home perex, /news body, and /admin/news preview share one source of styling. - Code-inside-`<pre>` contrast fix (was unreadable amber-on-silver). - `.news-content` table styling (border, header band, row-hover). **`scripts/dev/run-local.sh`** — local uvicorn launcher. Pulls Google OAuth client id/secret from GCP Secret Manager (`AGNES_OAUTH_GCP_PROJECT`-driven, no vendor defaults), points `AGNES_CLI_DIST_DIR` at `./dist` so the wheel endpoint resolves, and `--dev` flips `LOCAL_DEV_MODE=1` + `AGNES_HOME_ROUTE=/home` for one- command iteration. `LOCAL_DEV_MODE=1` also enables the FastAPI debug toolbar. **CLAUDE.md "Run tests before every push" section** codifies `pytest tests/ -n auto -q` as non-negotiable before each push. **Tests**: 51 + 14 + 8 = 73 new tests across news-template repo, sanitizer, API, web, CLI; plus updated home/auth/template tests for the new shared-shell architecture. Origin docs (gitignored, customer-fork content): docs/brainstorms/home-page-requirements.md, docs/plans/2026-05-07-001-feat-home-page-plan.md. * feat(cli): agnes onboarded {on,off,status} — self-scoped flag toggle User-facing equivalent of the in-page "Mark me as (off)boarded" button on /home. POSTs /api/me/onboarded with {onboarded, source}; --source overrides the audit-log marker so flips made from the CLI vs the web button vs agnes init automation stay distinguishable. `status` reads via /api/me/profile (when present); falls back to a quick body-marker scan of /home so the read path doesn't write an audit_log row. PAT-authed via cli.client.api_post — same convention as agnes admin news / agnes admin add-user etc. Tests: 5 covering on/off/status round-trip, idempotency, and audit-log source recording. Full suite holds at 12 pre-existing failures (same set as before). * ui(nav+home): primary nav reorg + green What's new band + /marketplace link fix Primary nav (post-rebase audit + per-user feedback): - Items: Home → Marketplace → Data Packages → Memory. Admin dropdown for admins only. The "Dashboard" label was renamed Home — point still resolves through `home_route` so customer instances on /dashboard still land there. - Activity Center moved into the Admin dropdown. Per-team adoption analytics is admin-consumed in practice; the route still allows any authed user for direct deep-links so existing /home tile + bookmarks keep working. - Memory link added (→ /corporate-memory) — was previously buried in the /home "Look around" tiles. - Setup local agent + My Stack dropped from main nav. Setup is the /home install flow's home now; My Stack lives as a tab inside /marketplace. /home tweaks: - Plugin marketplace tile now points at /marketplace (was /store — legacy from before the marketplace rebrand landed in #230). - "What's new" section header gets a green band (success-flavored D1FAE5 background, A7F3D0 border, darker green title) so the bottom-of-page news block visibly distinguishes from the blue install-hero at the top. Header strip only — body stays white. Test fix: test_home_route_resolution renamed `dashboard_link_uses_home_route` → `home_link_uses_home_route` and asserts `href="/home">Home` instead of `href="/home">Dashboard` after the label change. * fix(home): decouple Step 3 + Connect-tools collapse from server onboarded flag The server-side `users.onboarded` flip happens through two paths: 1. Explicit user click on "Mark me as onboarded" or `agnes onboarded on`. 2. Implicit `agnes init` POST → /api/me/onboarded on success. Path 2 produced a UX surprise: an analyst running `agnes init` mid-flow reloaded /home and saw Step 3 (auto-mode) + Connect-your-tools auto- collapse to summary bars. They were actively working through those sections — the install POST never signalled "I'm done with the rest of setup", just "Agnes itself is installed". Decouple the section-collapse decision from the server flag: - Step 1 + Step 2 install blocks: still hidden on `onboarded=TRUE` (their completion is a hard server signal — Agnes IS installed). - Step 3 + Connect-your-tools: render flat by default in BOTH states. Wrapped in `<details class="setup-collapsible" open>` so the browser's native disclosure handles per-section toggle without JS, but the `<summary>` is CSS-hidden until the page-level `data-setup-minimized="1"` attribute is set on `.home-mock`. - New "Minimize setup view" toggle inside the blue install-hero, rendered only when onboarded. Click flips the data-attr on `.home-mock` AND removes the `open` attribute from each `<details>`. State persists in `localStorage["agnes_home_setup_minimized"]` so the choice survives reloads but is per-device. - "Show full setup view" (the same button when minimized) re-opens both `<details>` and clears localStorage. When minimized, each `<details>` still has its own native expand/ collapse — click the gray summary bar to peek at one section without toggling the page-level minimize off. Tests: - test_step3_and_connectors_render_flat_when_onboarded_by_default — asserts `<details class="setup-collapsible" ... open>` for both sections post-onboarding and the absence of any server-rendered `data-setup-minimized` attribute on the `.home-mock` root. - test_minimize_toggle_visible_only_when_onboarded — toggle button rendered only when onboarded. Full pytest holds at 12 pre-existing failures (same set).	2026-05-08 18:28:47 +02:00
minasarustamyan	4fb2818a19	Add /marketplace browse page + Model B opt-in stack composition (#230) ... * Add /marketplace browse page + Model B opt-in stack composition New /marketplace browse surface unifies the curated marketplaces (admin-managed git mirrors) and the community Flea Market behind three tabs — Curated / Flea / My Stack — with per-tab category filter, search across both sources with scope checkboxes, and numeric pagination, all driven by URL query state. Plugin detail at /marketplace/curated/<slug>/<plugin> and /marketplace/flea/<id>; nested skill / agent detail at /marketplace/curated/<slug>/<plugin>/ {skill,agent}/<name> and the flea-side single-page detail. Model B opt-in: an RBAC grant on a curated plugin is now only *eligibility*. The user must click "Add to my stack" for it to enter their served Claude Code marketplace. Composition flips from (rbac ∖ opt_outs) ∪ store_installs to (rbac ∩ subscriptions) ∪ store_installs. The legacy user_plugin_optouts table is renamed user_curated_subscriptions (schema v27) — same table shape, inverted semantic, repository methods become subscribe / unsubscribe / is_subscribed. UX vocabulary: Install → Add to my stack, Installed → In your stack, card "Installed" badge → "In stack" (amber pill), tab "My Subscriptions" → "My Stack". Bridges the two-step model (server-side bookmark vs. on-laptop install) the previous label hid. Click triggers an inline post-add hint panel under the description with the agnes refresh-marketplace recipe + Copy chip, dismissible per-browser via localStorage. Per-tab info blocks above the filter row: - Curated: trust signal — "Each plugin here has a named curator accountable for it." (blue accent + See-all-curators link) - Flea: open-shelf signal — "Anyone in the company can upload here." (purple accent + Tips-for-sharing link) - My Stack: personal-shelf orientation — "Your AI stack — everything you've added." (slate accent, no link) Tabs carry per-tab Heroicons (shield-check / building-storefront / rectangle-stack) tinted to match each tab's accent; flips white when the tab is active for contrast. Hero illustration anchored to the right of the blue hero panel (absolute, 47% wide, behind the search row content). Hidden under 900px viewport. Action-row CTAs realigned to publication intent: curated "How to add new content" → "Submit a plugin" (links to the guide page); flea button removed since +Upload sits next to it. Empty-state CTAs match. /marketplace/guide/{curated,flea} routes now host publication-flow guide pages with placeholder ledes — full copy to be authored separately. Categories: Heroicons-based icons mapped per category in src/category_icons.py (zero new dependencies; SVG path strings inlined). Marketplace cards, filter pills, and detail pages read from the same source. API endpoints under /api/marketplace: - GET /items per-tab listing (curated / flea / my) - GET /categories per-tab non-zero counts - GET /curated/{slug}/{plugin} plugin detail - POST/DELETE /curated/{slug}/{plugin}/install subscribe toggle - GET /curated/{slug}/{plugin}/{skill,agent}/{name} inner item The tab=my branch reads directly from user_curated_subscriptions ∪ user_store_installs (not resolve_user_marketplace, which bundles flea skills/agents into a single store-bundle synthetic entry useful for serving the Claude Code marketplace ZIP/git but wrong for browsing where each item should appear as its own card). Detail pages: plugin detail surfaces inner skills/agents as clickable nested cards; commands/hooks/MCPs render as plain name lists. Skill/agent detail mirrors the plugin layout with kind-tinted accents (skill = green, agent = purple), Description + Details sidebar, Files + Docs sections, and the "How to call it" copy-able invocation chip showing /<plugin>:<inner-name> exactly as Claude Code namespaces it post-install. Curated nested has no install button — links back to the parent plugin. Navbar: standalone "My AI Stack" relabelled "My Stack" and points at /marketplace?tab=my; "Store" link removed (Store flow is reachable via the Flea Market tab's +Upload button). The standalone /my-ai-stack and /store routes still work for old bookmarks. Tests cover the new browse / categories / install / RBAC paths under tests/test_marketplace_api.py; existing marketplace and store tests updated for Model B (explicit subscribe in fixtures). Schema bumped v26 → v27 with idempotent migration that wipes existing user_plugin_optouts rows on flip and adds marketplace_plugins.created_at with registered_at backfill. * Fix v28 migration + post-rebase test fallout v28 ALTER TABLE marketplace_plugins ADD COLUMN created_at conflicted with _SYSTEM_SCHEMA's earlier CREATE that already includes the column on fresh installs (test fixtures starting at any pre-v28 version trip on it). Switch to ADD COLUMN IF NOT EXISTS — same idiom as the upstream v27 Keboola sync-strategy migration on the same ladder. Two test patches needed after the rebase bumped SCHEMA_VERSION 27 → 28: - test_keboola_v27_migration.py: test_schema_version_constant_is_27 was pinning ==27. Loosened to >=27 (the test's purpose is to verify the v27 Keboola migration, not to pin the current SCHEMA_VERSION). - test_setup_page_unified.py: was monkeypatching resolve_allowed_plugins but compute_default_agent_prompt now reads from resolve_user_marketplace (Model B-aware). Stub the right function so the test exercises the v28 served-set path. * Harden curated skill/agent inner endpoints against path traversal `_read_inner`, the `skill_dir` walk in `curated_skill_detail`, and the `agent_path.stat` in `curated_agent_detail` joined URL path-params onto `plugin_root` without verifying the resolved candidate stayed inside it. Starlette's `[^/]+` on `{skill_name}` / `{agent_name}` blocks the direct URL exploit (encoded `/` 404s before the handler), but a curator-planted symlink inside a curated marketplace's git mirror could still dereference outside the plugin tree on read. Adds `_safe_join(plugin_root, *parts)` doing `Path.resolve(strict=True)` + `relative_to(plugin_root.resolve())`, used by all three call sites so the boundary is enforced once and consistently. Tests cover the helper directly (normal path resolves, escaping `..` returns None, escaping symlink returns None, missing file returns None) plus an end-to-end check that the symlink case actually 404s on the HTTP endpoint. Symlink tests skip on Windows where symlink creation needs elevated permissions; they run on Linux CI. --------- Co-authored-by: Minas Arustamyan <arustamyan.minas@gmail.com>	2026-05-08 14:22:19 +02:00
ZdenekSrotyr	506a378c3a	release: 0.47.1 — Keboola connector v27 (incremental, partitioned, where_filters, typed parquet) (#217) ... ## Summary Brings the Keboola connector to feature parity with the legacy internal data-analyst's per-table sync strategies. Closes the four documented gaps from the spec branch (`zs/keboola-connector-specs`): - **Typed parquet** in the legacy SDK extraction path — column types from Keboola Storage metadata (provider cascade `user > ai-metadata-enrichment > keboola.snowflake-transformation`) survive the CSV → parquet roundtrip; invalid date strings (`'0000-00-00'`) and invalid numeric strings (`'Non-Manager'`) become NULL while keeping the column's typed schema. Pre-fix everything was VARCHAR. - **Incremental sync** via Storage API `changedSince` — opt-in per table; pulls only delta rows, merges into the existing parquet by `primary_key` (drop_duplicates with keep='last'). Cuts daily extraction from O(full table) to O(delta). - **Partitioned sync** — flat per-partition layout `data/<table>/<key>.parquet` (e.g. `2026_05.parquet`), per-affected-partition merge for daily updates, chunked initial load with 1-day overlap and 2-empty-chunk stop heuristic. - **`where_filters`** — server-side row filter with date placeholders (`{{today}}`, `{{last_3_months}}`, `{{start_of_3_months_ago}}`, etc.) resolved at sync time. Force the SDK path; reject `incremental + where_filters` combination at API layer (changedSince already filters temporally). ## Architecture - **Schema migration v25 → v26**: 7 new columns on `table_registry`. Existing `sync_strategy` column reused (pre-v26 it was inert catalog metadata; post-v26 the extractor dispatches off it). - **Per-table dispatcher** in `extractor.run()` routes to one of `_extract_via_extension` (full_refresh + extension), `_extract_via_legacy` (full_refresh + filters or extension fallback), `extract_incremental`, or `extract_partitioned`. - **API conflict policy**: `incremental + where_filters` → 422; `partitioned + query_mode='remote'` → 422; `partitioned ⇒ partition_by required`. - **Admin UI**: third "Direct extract (Storage API)" radio in the Keboola Register / Edit modals, alongside existing "Whole table (extension)" and "Custom SQL". When selected, exposes a v26 sync-strategy panel with conditional fields per strategy. ## Test plan - [x] **Unit + module** — 134 v26 tests covering migration, repo, parquet_io, where_filters, incremental (compute_changed_since + merge_parquet + extract_incremental E2E), partitioned (key derivation + merge_partition + chunked windows + extract_partitioned E2E), extractor dispatcher, admin API validators, PUT field clearing, registry-shape → dispatcher bridge - [x] **HTML form structure** — all v26 inputs + visibility classes + JS payload fields verified in rendered template - [x] **Real Keboola roundtrip** — registered a small test table as `sync_strategy='incremental'` against a test Storage project, triggered two syncs: - Sync 1: `changedSince=None` → full pull → 9 rows typed parquet - Sync 2: `changedSince=last_sync - 1d window` → 9 delta rows merged with 9 existing → 9 after dedup on primary_key (PK merge confirmed) - [x] **Browser UX** — agent-browser session against a local uvicorn: login → admin/tables → register modal → switch radios → verify field visibility per strategy → submit → edit existing row → switch to Direct/Incremental → save → confirm DB persistence - [x] **Regression** — no regressions in the broader 3252-test suite (3 pre-v26 tests updated for the deprecation-marker removal + schema-version bump; 2 pre-existing environment-sensitive test failures unrelated to this change) ## Bugs caught + fixed during E2E The browser + real-Keboola roundtrip exposed four bugs the unit tests missed: 1. **JS visibility race** — two competing `forEach` loops set `display=''` then `display='none'` on form elements sharing `kb-strategy-incremental kb-strategy-partitioned` classes (window_days + max_history_days are reused across strategies). Fix: single-pass selector with class-based visibility resolver. 2. **PUT cannot clear field** — pre-v26 `updates = {k: v ... if v is not None}` collapsed "omitted from body" and "sent as null" into the same case, so admin couldn't switch a partitioned row back to full_refresh and have stale `partition_by` clear. Fix: `model_dump(exclude_unset=True)`. 3. **Subprocess DB lock conflict** — `_read_last_sync` reopened `system.duckdb` while the parent server held the write lock (subprocess contract at `app/api/sync.py:_run_sync` line 260). Fix: parent injects `__last_sync__` into table_config before subprocess spawn. 4. **Wrong KBC table_id** — `extract_incremental` / `extract_partitioned` built the Storage API table_id from the registry row's slugified `id` (`circle_inc`) instead of `bucket.source_table` (`in.c-finance.circle`), producing 404s. Fix: prefer `bucket+source_table`; fall back to `id` only when bucket empty. ## Operator notes - Existing tables stay on `full_refresh` after migration; admins opt individual tables in via `agnes admin register-table --sync-strategy ...`, the Keboola Edit modal, or `POST/PUT /api/admin/registry`. - `merge_parquet` and `merge_partition` use `pd.concat + drop_duplicates`, loading both existing and delta into pandas RAM. For tables in the multi-million-row range this may OOM — switch to `partitioned` strategy for those (per-partition merge keeps memory bounded). Documented in `### Internal` of the changelog entry. - Date placeholders are resolved at **sync time**, not register time — a typo'd `{{lasst_week}}` is accepted at register and surfaces only when the next sync runs. By design (rolling windows need late-binding). ## Spec source The four corresponding plans on the `zs/keboola-connector-specs` branch under `docs/superpowers/plans/2026-05-07-0[1-4]-*.md` capture the design rationale and link back to internal repo references for each subsystem. <!-- devin-review-badge-begin --> --- <a href="https://app.devin.ai/review/keboola/agnes-the-ai-analyst/pull/217" target="_blank"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1"> <img src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1" alt="Open in Devin Review"> </picture> </a> <!-- devin-review-badge-end -->	2026-05-07 19:01:27 +02:00
ZdenekSrotyr	28430ced09	Keboola cutover: native parquet path + sync correctness + auto-discover protection (#190) ... * fix: cutover regressions + parallel Keboola legacy fallback Bundled fixes from a fresh-deploy run on a Keboola Storage backend with the block-shared-snowflake-access feature flag — DuckDB Keboola extension's per-table scan can't access bucket schemas, so the legacy kbcstorage Storage-API client is the only working path. CUTOVER REGRESSIONS - agnes pull hash mismatch on every Keboola local-mode table — src/orchestrator.py:_update_sync_state stored md5(mtime+size)[:12] while the CLI compares against full 32-char content MD5. Now stores the same content MD5 the materialized SQL path already used. - Trailing-slash sanitization in connectors/keboola/access.py and extractor.py — DuckDB Keboola extension's ATTACH fails when the URL ends in / (canonical form). - src/profiler.py:TableInfo.description becomes optional — two call sites instantiated without it, crashing the profiler pass. - scripts/ops/agnes-auto-upgrade.sh: chown on UID change — older images ran as root, current runs as agnes (uid 999). Reads target uid:gid from /etc/passwd inside the new image and chowns ${STATE_DIR}, /data/extracts, /data/analytics when the digest moves. - POST /api/sync/trigger is now singleton per process — two near-simultaneous trigger calls each forked an extractor subprocess, fought for extract.duckdb's file lock, starved uvicorn, flipped the container to unhealthy. Trigger now returns 409 (sync_already_in_progress) when held; _run_sync acquires non-blocking. PARALLEL LEGACY FALLBACK - Process pool fan-out for the _extract_via_legacy queue (default 8 workers, override via AGNES_KEBOOLA_PARALLELISM). Process pool, not thread pool, because connectors/keboola/client.py:export_table does os.chdir(temp_dir) — process-global, so threads raced and slice files landed in the wrong directory ("[Errno 2] No such file or directory: '<job_id>.csv_X_Y_Z.csv'"). - Extractor subprocess timeout 1800s -> 3600s (configurable via AGNES_EXTRACTOR_TIMEOUT_SEC). 28+ tables × multi-minute Keboola export jobs need the headroom on telemetry-class projects. - Process group cleanup on timeout — Popen(start_new_session=True) puts the extractor in its own group. On timeout the parent SIGTERMs the group (10s grace) then SIGKILLs stragglers. Without this, the pool workers were reparented to PID 1 and continued holding open Keboola Storage export jobs. Inline extractor script also installs a SIGTERM -> sys.exit(143) handler so the with ProcessPoolExecutor(...) block __exit__ runs cleanly. Tests: existing tests that patched subprocess.run updated to patch subprocess.Popen with a _FakePopen stand-in (same exit-code-injection contract). Two tests that exercised the parallel path forced AGNES_KEBOOLA_PARALLELISM=1 to keep mocks alive (mocks don't ride into ProcessPoolExecutor subprocesses). Squashed onto current main (was 7 commits + multi-commit CHANGELOG + agnes-auto-upgrade.sh conflicts; squash avoids per-commit conflict resolution against main's flat-mount STATE_DIR refactor and 0.38.0 release cut). * feat(keboola): Storage API direct extract path; drop extension data path The DuckDB Keboola extension's COPY routes through Keboola QueryService, which is unreliable on linked-bucket projects (extension v0.1.6 fixes that case but isn't yet in the community CDN, and pre-fix any project with the block-shared-snowflake-access feature flag couldn't see bucket schemas at all). Move the extract path off the extension entirely and talk to the Storage API directly via signed-URL download — works on any project, regardless of extension state. connectors/keboola/storage_api.py (NEW) Lightweight client built on requests.Session. Three endpoints: - POST /v2/storage/tables/{id}/export-async (kicks off job) - GET /v2/storage/jobs/{id} (poll until done) - GET /v2/storage/files/{id}?federationToken=1 (signed URL detail) - GET <signed_url> (download bytes) Supports sliced exports (manifest + per-slice signed URLs) and gzipped payloads. ExportFilter dataclass mirrors the Keboola filter spec (whereFilters / columns / changedSince / limit) and handles JSON round-trip with the registry's source_query column. Token redaction in error messages. Bounded exponential backoff on job polling. No cloud-SDK dependency on the data path; thread-safe. connectors/keboola/extractor.py - materialize_query() rewritten: takes bucket/source_table/source_query (JSON filter spec), exports via KeboolaStorageClient, converts CSV to parquet via DuckDB, atomic os.replace. Same return shape so sync.py downstream code stays uniform with the BQ branch. - _extract_via_legacy() also moved to Storage API direct (kept the name for caller compatibility with _legacy_worker / the parallel batch extractor). Per-call temp directories — no os.chdir, threads don't race. app/api/sync.py _run_materialized_pass for source_type='keboola' rows now constructs a KeboolaStorageClient (replaces KeboolaAccess) and passes bucket/source_table/source_query to materialize_query. Reuses one client across rows for HTTP keep-alive. Sources keboola URL from env too (KEBOOLA_STACK_URL) when instance.yaml doesn't have stack_url configured. cli/commands/admin.py discover-and-register defaults Keboola rows to query_mode='materialized' (NULL source_query = full table), matching the v26 migration's unification of the local/materialized split for Keboola. BigQuery and Jira keep their per-source defaults. src/db.py Schema bump 25 → 26. Migration: UPDATE table_registry SET query_mode='materialized' WHERE source_type='keboola' AND query_mode='local'. NULL source_query on those rows means "full table export" — same effective behavior the local mode provided, but now via Storage API instead of the extension. pyproject.toml kbcstorage dep stays (admin-side bucket/table list still uses the SDK in app/api/admin.py / connectors/keboola/client.py); only the data path is migrated off the SDK. Comment updated to reflect the new boundary. tests - test_keboola_storage_api.py (NEW, 19 tests): ExportFilter parsing, HTTP client (token redaction, retry logic, polling), download_file (single, gzipped, sliced), end-to-end export_table_to_csv. - test_keboola_materialize.py rewritten: mocks KeboolaStorageClient instead of FakeAccess; same atomic-write + zero-rows + unsafe-id contracts. - test_sync_trigger_keboola_materialized.py: registry rows now carry bucket+source_table+JSON-shape source_query. 114+ Keboola-impacted tests green locally. * test: schema version assertion bumped to 26 alongside the keboola query_mode migration * fix(keboola): cutover hot-patches surfaced on agnes-dev Five small fixes that were applied as in-container hot-patches during agnes-dev cutover and need to be on the source-of-truth image so a fresh upgrade does not undo them. - app/api/sync.py: auto-discover gate considers the WHOLE registry (any source, any mode), not just rows where source matches and query_mode is local. After the v25→v26 keboola materialized migration an instance can have 30 materialized rows and zero local rows; the previous gate kept re-firing _discover_and_register_tables every scheduler tick, creating duplicate auto-discovered rows with the wrong bucket prefix every time. - app/api/admin.py: _discover_and_register_tables reassembles the bucket as <stage>.<bucket-id> (e.g. in.c-finance) instead of dropping the stage prefix; default query_mode for keboola is now materialized (the v26 contract); validator allows NULL source_query for keboola materialized rows (full-table export via Storage API export-async, no SQL needed). - cli/commands/admin.py: register-table mirrors the server validator (NULL source_query allowed for source_type=keboola); --bucket help text generalized to cover both BQ dataset and Keboola bucket id. - connectors/keboola/extractor.py: max_line_size=64 MiB on read_csv_auto so embedded JSON / SQL cells (kbc_component_configuration in particular) do not trip the default 2 MiB ceiling. - connectors/keboola/storage_api.py: GCP backend support — when the Storage API returns a manifest whose slice URLs are gs:// references with a gcsCredentials block, rewrite to the JSON REST download endpoint and authenticate with the issued OAuth bearer token; redact tokens in any surfaced error string. * test: align with new keboola materialized + auto-discover-gate contracts - test_admin_keboola_materialized: rename test_register_keboola_materialized_rejects_missing_source_query → test_register_keboola_materialized_accepts_missing_source_query. v25→v26 introduced 'keboola materialized with NULL source_query means full-table export via Storage API export-async' as the default registration shape; the rejection case is no longer the contract. - test_sync_filter: add list_all() to _StubRegistry. The auto-discover gate in _run_sync now keys off the WHOLE registry (not just local rows) so materialized-only Keboola instances do not re-trigger discovery on every tick. * feat(keboola): native parquet export — skip CSV roundtrip Storage API export-async accepts fileType={csv,parquet}. Switching the materialized sync to parquet eliminates the CSV → DuckDB COPY → parquet roundtrip that pinned a single uvicorn worker over 4 GiB on multi-GB tables (read_csv with all_varchar + max_line_size=64MB has to materialize the whole CSV in memory before COPY can stream out a parquet). Snowflake UNLOAD on Keboola's side already produces typed, self-contained parquet files; the extractor downloads them and renames into place. Two cases: - **Single-file** export (small table): file_info.url points at one signed URL; download_file streams chunks straight to .parquet.tmp and we're done. No DuckDB. - **Sliced** export (Snowflake UNLOAD respects MAX_FILE_SIZE — 16 MiB default — so anything larger arrives as N parquet slices): each slice is a complete parquet file with its own footer; naive concat would corrupt them. download_file_slices keeps the slices as separate files in a tempdir, then DuckDB COPY (SELECT * FROM read_parquet([slice0, slice1, ...])) merges them into one consolidated parquet. DuckDB streams row groups during this — peak memory bounded to one row group (~1 MiB) regardless of source size. The legacy CSV path stays as the explicit opt-in via source_query= '{"file_type":"csv"}' for projects whose backend can't UNLOAD parquet (none known today; cheap escape hatch). Backward-compat alias KeboolaStorageClient.export_table_to_csv kept. Also fixes a latent bug in download_file's gzip detection: previous heuristic flagged any unencrypted file as gzipped, which would have corrupted parquet downloads at gunzip time. Name-suffix-only now. * fix: tempdir leak cleanup, every 0m schedule, /sync/trigger body shapes Three small self-contained fixes uncovered during agnes-dev cutover. - connectors/keboola/extractor.py: tempfile.TemporaryDirectory now uses ignore_cleanup_errors=True so a worker death mid-write doesn't leave multi-GiB stale slice trees on the boot disk. (12 GiB seen after a disk-full crash where TemporaryDirectory's own cleanup also raised and got swallowed.) - src/scheduler.py: is_valid_schedule accepts 'every 0m' (interval=0 = always due). Force-resync of an errored row no longer requires waiting out the default 'every 1h' interval — admin can flip the schedule, trigger, then flip back. - app/api/sync.py: POST /api/sync/trigger accepts both ['table_id'] (legacy bare-array body) and {'tables': ['table_id']} (matches the response payload shape, more discoverable for clients building requests by hand). Malformed bodies return 422 with a structured detail; null/missing means 'sync everything' as before. Tests cover: tempdir cleanup on raise (sliced parquet path), is_valid_schedule + is_table_due 'every 0m' acceptance, and trigger body parametrized matrix (8 valid shapes + 6 rejection cases). * fix: targeted-trigger filter in materialized pass + auto-upgrade defer Two operational gaps observed during agnes-dev cutover, in the same sync-routing area. - _run_materialized_pass now takes a 'tables' arg and skips rows not in the target set with reason='not_in_target'. POST /api/sync/trigger with a body of tables previously only scoped the legacy extractor subprocess — the materialized pass kept iterating every due materialized row, so an admin asking to re-sync kbc_job re-ran every other due materialized row alongside it. Match on registry id OR name (admins commonly pass either form). tables=None preserves the no-filter behavior. - New GET /api/sync/status (public, no auth) returns {locked: bool} off _sync_lock.locked(). agnes-auto-upgrade.sh probes this before docker compose up -d and exits 0 with a 'deferred recreate' log line if a sync is in flight — the next 5-min cron tick retries. Pre-fix, an auto-upgrade triggered mid-sync would recreate the uvicorn worker and kill the in-flight extractor / Snowflake-UNLOAD download (observed when kbc_job's first 7-day retry got SIGKILLed). Connection failures in the probe fall through to the upgrade — being stuck on a wedged image is worse than interrupting a hypothetical sync. * fix: auto-discover protects admin overrides + surfaces drift Two real-world incidents on agnes-dev drove this: 1. kbc_job was registered manually with the correct (in.c-kbc_telemetry, kbc_job) coordinates. A naive auto-discover re-run would have inserted a SECOND kbc_job row at the slugified id 'in_c-keboola-storage_kbc_job' (where Keboola's discovery places it) — and that row's Storage API export-async 404s. 2. An earlier auto-discover bug stripped the stage prefix from bucket ids ('c-finance' instead of 'in.c-finance'), inserting 137 rows whose syncs all failed. Fix: - _discover_and_register_tables now builds a plan first (_build_keboola_discovery_plan) classifying each discovered table into one of new / existing_match / existing_drift / invalid, then executes only the 'new' bucket. Drift rows are reported with both sides of the disagreement plus drift_kind: - same_id_diff_coords: registry has the same id but different bucket / source_table (admin migrated coords inline). - name_collision: discovery's slugified id differs from any registry id, but the discovered .name matches an existing row's .name (case-insensitive). Catches the kbc_job case. - Bucket detection now prefers the API's authoritative bucket_id field (separate field on the Keboola tables.list response, normalised by KeboolaClient.discover_all_tables). Falls back to id-string parsing only when bucket_id is missing (older fallback path inside discover_all_tables). - Endpoint POST /api/admin/discover-and-register?dry_run=true returns the plan without writing — would_register, drift, invalid lists. Lets an operator audit before merging discovery with a registry that has admin overrides. Removed 'every 0m' from test_register_request_rejects_malformed_sync_schedule — the runtime started accepting it in the previous commit (force-resync override) and the validator follows suit. * feat(keboola): AGNES_TEMP_DIR routes tempfiles off overlayfs /tmp The container's /tmp lives on the boot disk's overlayfs (29 GiB on agnes-dev, shared with /var). Snowflake UNLOAD of a wide table writes slices into per-call /tmp tempdirs that fill multi-GiB / many-slice exports long before the dedicated data disk fills. agnes-dev hit 100% boot-disk while the 20 GiB data disk had 15 GiB free. connectors.keboola.storage_api.get_temp_root() reads AGNES_TEMP_DIR; mkdirs the target on first use; unset / empty / unwritable falls back to None (system tempdir, OSS-pre-fix behaviour). Both materialize_query (parquet path) and _extract_via_legacy (CSV fallback) and the sliced-CSV concat path in storage_api use the helper now. docker-compose.yml defaults AGNES_TEMP_DIR=/data/tmp on app, scheduler, and extract services. The data volume is the dedicated disk in production layouts and a plain docker volume in single-disk dev/laptop setups — same blast radius as the previous /tmp default on the latter, no regression.	2026-05-07 12:12:14 +02:00
Minas Arustamyan	d5a7c9ad79	feat(store): /store + /my-ai-stack — community marketplace + per-user composition ... Adds a community-driven Store where any authenticated user uploads skills/agents/plugins as ZIPs, plus /my-ai-stack as the per-user composition view. The served Claude Code marketplace is now: (admin_granted ∖ opt_outs) ∪ store_installs Skill + agent installs are merged into a single `agnes-store-bundle` plugin in the served marketplace; type=plugin uploads stay standalone. Names are suffixed with `-by-<owner-username>` at upload time so two owners can use the same display name without colliding in Claude Code's flat skill/agent namespace. Schema v23 → v24 adds three tables: - store_entities — community-uploaded skills/agents/plugins - user_store_installs — what each user has chosen to install - user_plugin_optouts — opt-out overlay on top of admin grants Admin grant-delete drops every user's opt-out for that plugin so re-grant resets cleanly to enabled (no sticky personal preference). UI: - /store — e-commerce-style listing with type/category/owner filters, search, pagination, owner-aware [Install] buttons, clickable cards - /store/new — 2-step upload wizard with drag & drop, preview validation (POST /api/store/entities/preview), docs multi-upload, photo + video URL - /store/{id} — detail page with hero, file list, docs, owner actions (Edit/Delete) for the uploader - /my-ai-stack — Granted plugins (toggle opt-out) + From the Store (uninstall) sections - Admin nav: Marketplaces moved into Admin dropdown, renamed to "Curated Marketplaces" Validation hardening: type-mismatch guards reject skill ZIP uploaded as agent (or vice versa), and plugin ZIPs masquerading as skills/agents. Human-readable error messages mapped client-side from machine codes. Cross-source naming: Store entity-id-prefixed dirs (`plugins/store-<id>/`) plus the bundle (`plugins/store-bundle/`) avoid collisions with admin marketplaces (whose `store` slug is reserved by `is_valid_slug`). Bundle composition is content-hashed at serve time — install/uninstall or owner re-upload bumps the bundle's plugin.json `version`, so Claude Code's auto-update toggle picks up changes. Tests: 50+ new tests across naming, repositories, filter (admin ∪ store ∪ bundle), API (upload/install/uninstall/delete/preview/docs), end-to-end marketplace.zip with bundle merging.	2026-05-05 02:53:49 +02:00
ZdenekSrotyr	fac10b29e4	feat(schema): v24 — rewrite materialized BQ source_query to BQ-native ... Materialize now wraps admin SQL into bigquery_query('<billing>', '<inner>') which requires the inner SQL to be BigQuery-flavor (backticked identifiers, native function syntax). v24 migrates existing rows from DuckDB-flavor (bq."ds"."tbl") to (`<project>.ds.tbl`) using the configured BQ project. Idempotent on already-converted rows; logs a warning and skips when the project isn't configured (operator can configure + restart for retry).	2026-05-04 19:15:54 +02:00
ZdenekSrotyr	f01eb4143d	feat(db,repo,renderer): schema v23 + claude_md_template + ClaudeMd renderer ... - Bump SCHEMA_VERSION 22 → 23; add claude_md_template singleton table to _SYSTEM_SCHEMA and _V22_TO_V23_MIGRATIONS; wire migration + fresh-install seed - src/repositories/claude_md_template.py: ClaudeMdTemplateRepository (get/set/reset) mirroring WelcomeTemplateRepository; defensive re-seed in get() - src/claude_md.py: compute_default_claude_md / render_claude_md / build_claude_md_context — rich renderer with RBAC-filtered tables, metrics, and marketplaces; reads override from claude_md_template or falls back to config/claude_md_template.txt; raises TemplateError on broken override - config/claude_md_template.txt: default Jinja2 markdown template restored from PR #167 history (tables, metrics, marketplaces, BQ guidance, corporate memory, directory structure, per-user footer)	2026-05-03 22:43:56 +02:00
ZdenekSrotyr	0ee22f8fb0	docs: add setup-banner.md + rename migration test to test_db_schema_version.py ... - Add docs/setup-banner.md: placeholder table, autoescape semantics, security note on post-render stripping, diff table vs welcome-template (M-9). - Update CHANGELOG.md to reference docs/setup-banner.md. - Rename tests/test_db_migration_v20.py → tests/test_db_schema_version.py (file tested SCHEMA_VERSION==22, not just the v20 step; clearer name) (M-10).	2026-05-03 16:12:13 +02:00

Renamed from tests/test_db_migration_v20.py (Browse further)