agnes-the-ai-analyst/pyproject.toml
minasarustamyan dc5e0e0d11
Marketplace UX overhaul: rich plugin/skill/agent detail + filename rename (#251)
* Rename agnes-metadata.json to marketplace-metadata.json

Curated marketplace enrichment file (.claude-plugin/agnes-metadata.json)
becomes marketplace-metadata.json. Clean cut, no fallback — curators of
upstream marketplace repos must rename the file on their side.

Python API renames mirror the file rename: read_agnes_metadata →
read_marketplace_metadata, AGNES_METADATA_REL → MARKETPLACE_METADATA_REL,
AGNES_METADATA_MAX_BYTES → MARKETPLACE_METADATA_MAX_BYTES. Synth Claude
Code marketplace strip rule (.agnes/** + the metadata file) follows the
new filename.

* Marketplace detail polish: window cover + 715:310 aspect + helper alignment

- Plugin & item (skill/agent) detail hero: 160x160 square cover replaced
  with a macOS-style window frame (3 traffic-light dots + titlebar label
  showing the entity name). Body is constrained to 715:310 so curator-
  uploaded covers no longer crop to a square. Window is 380px wide; meta
  column and absolutely-positioned top-right install/remove actions stay
  put. Fallback when no cover_photo_url (translucent gradient + PL/SK/AG
  initials) is unchanged, just inside the window body.

- Inner skill/agent cards in the plugin detail's Internal structure
  section adopt the same 715:310 aspect (was fixed 78px tall). No window
  chrome on inner cards — just the matching proportions so covers read
  consistently across hero, grid tiles, and listing cards.

- Curated nested item helper text ("This skill is part of ... — add the
  bundle to your stack to use it") now stacks UNDER the "Open parent
  plugin" button instead of being a side-by-side flex sibling in the
  actions-row. Added align-self: flex-end so the 260px helper box
  anchors at the right edge of the 300px actions column, matching the
  button's right edge.

* Marketplace My tab: surface the same category + type filters as Flea

- Frontend: mp-cat-row and mp-type-row now show on tab=my (previously
  hidden — type was flea-only, category was flea/curated-only). Curated
  browse stays plugin-only and continues to hide the type pills.
  fetchOne() sends the `type` param for tab=my too, so the items
  endpoint's existing my-branch filter actually receives it.

- Backend categories endpoint, tab=my branch: when the type filter is
  set to skill/agent, skip counting curated subscriptions. Curated
  plugins are always type='plugin', so they wouldn't survive the items
  endpoint's type filter; including them in the category counts made
  the pill numbers overstate what users could actually see in the
  grid. type=None or type='plugin' keeps the previous behaviour.

- CHANGELOG entry under [Unreleased].

* Marketplace plugin detail: render rich content from marketplace-metadata.json

Adds five optional plugin-level fields to marketplace-metadata.json and
renders them on the curated plugin detail page + listing card:

* display_name — friendly h1 / listing-card name / mac-window titlebar
  label (overrides the technical plugin id)
* tagline — punchy 1-line value prop for the hero subtitle and the
  listing card description (replacing the verbose marketplace.json
  description on cards)
* description — multi-paragraph markdown body, server-side rendered
  through markdown-it-py and sanitized through nh3 with a
  description-scoped allowlist (no iframes / no raw HTML / no
  javascript: links). Powers the "What it does" panel.
* use_cases[] — {title, description, prompt} entries that render as a
  3-column "When to use it" card grid; each card shows the literal
  prompt as a code chip so users can copy-paste into Claude Code.
* sample_interaction — {user, assistant} dialog rendered in a Claude
  Code-style dark Catppuccin Mocha transcript panel: monospace user
  row with a green ">" prompt indicator + sans-serif assistant body
  with markdown formatting (peach bold, yellow italic, pink inline
  code, mantle-dark fenced code blocks).

All five fields are optional; UI sections only render when populated,
so plugins without enrichment look identical to before. Fields are
read on-demand from the working tree (cached by mtime per marketplace
slug) so curator edits land at the next request without waiting for
a sync cycle — same pattern as the existing inner-skill/agent
enrichment path. No DB schema bump.

Skill / agent rich-content rendering is deferred to a later phase
(needs a source-of-truth decision: extend plugin.yml? LLM-generate
from SKILL.md / agent.md?). The schema accepts the same fields at
skill/agent level today for forward compatibility but the UI ignores
them for now.

Also: stripped a stale `background-color: var(--bg)` from the global
`code` rule in style.css (was making inline code visually disappear
on the page background).

* Skill / agent detail: render rich content from marketplace-metadata.json

Brings the skill/agent detail pages to parity with the plugin detail
page. Same rich-content schema (display_name, tagline, description as
markdown, use_cases[], sample_interaction) plus two per-item additions:

* invocation — curator-provided literal command string. When set,
  overrides the computed "<manifest_name>:<inner_name>" chip and
  cleanly supports both "/" skill prefix and "@" agent prefix (the
  hardcoded "/" in the chip markup is hidden when the curator provides
  the invocation, so /grpn-eng:query <q> and @grpn-eng:cto-architect
  both render correctly).
* when_to_use — markdown disambiguation block ("Use this for X. For
  similar Y, see /other-skill") rendered into a new "When to use this"
  panel below the Example section.

Skill / agent category is now per-item overridable in
marketplace-metadata.json. When absent, the API keeps the parent
plugin's category as the badge so existing items don't lose their
category until curators opt in to per-item categorization.

The new "Example" Q&A panel uses the same Claude Code-style dark
Catppuccin Mocha transcript treatment as the plugin detail —
monospace user row with a green ">" prompt indicator + sans-serif
assistant body with markdown formatting.

All new fields are optional and read on-demand from the working tree.
Skills / agents whose marketplace-metadata.json doesn't carry rich
content render exactly the same way they did before (frontmatter
description + computed slash command + cover from existing v32
enrichment). No DB schema bump.

* Fix TypeError in skill / agent detail when curator sets per-item category

`curated_skill_detail` and `curated_agent_detail` were passing both
`**parent` (from `_curated_inner_parent_fields`, which returns the
parent plugin's category as a fallback) and `**enrichment` (from
`_curated_inner_enrichment`, which returns the per-item category
override when the curator set one) into `InnerDetailResponse(...)`.

Python function-call kwargs unpacking with overlapping keys raises
`TypeError: got multiple values for keyword argument 'category'`
— it doesn't merge like a literal dict does. The bug only surfaced
when the marketplace-metadata.json carried a `category` field at
skill / agent level (curator opting into per-item categorization);
items without that override hit the endpoint cleanly because only
parent provided the key.

Fix: build `merged = {**parent, **enrichment}` first (literal-dict
syntax DOES merge, with the right-hand-side winning) and unpack the
merged dict. Curator override still wins via the merge order, and
the same pattern is future-proof for any other field that lands in
both layers later.

Plus a regression test in test_marketplace_metadata.py asserting
that the inner-resolver carries `category` for downstream merging.

* Marketplace detail: tolerate partial curator JSON

Server constructed UseCase / SampleInteraction via raw dict indexing
(uc["title"], sample["assistant"]), so a curator commit missing any
required Pydantic field crashed the whole plugin / skill / agent detail
endpoint with a 500. Route both constructions through _safe_use_case /
_safe_sample_interaction helpers — partial input silently drops the
malformed card / section instead of breaking the page.

Regression test in test_marketplace_api.py covers the three shapes:
use_case missing a key, use_case with an empty string, and
sample_interaction with only user (no assistant). Sibling rich fields
still render.

* Address PR-251 review (must-fixes + S2/S3 polish) + release-cut 0.50.0

Five must-fixes from the review pass (3 from @cvrysanek's two-stage
review, 2 from my independent pass), plus the 0.50.0 release-cut as the
last commit on this PR per CLAUDE.md (CLAUDE.md "Release-cut belongs
to the PR" rule added in v0.49.1).

Must-fixes
----------

1. Cache eviction: bounded LRU instead of per-marketplace predicate.
   The previous predicate (`k[0] == marketplace_id and k[1] != mtime_ns`)
   only swept stale entries for the CURRENT marketplace; with N>100
   distinct marketplaces each holding one mtime key, the cap silently
   failed and memory grew linearly. Replaced with OrderedDict-backed
   bounded LRU at cap=256, drop oldest insert on overflow.
   Cache stress test pinned in test_marketplace_metadata.py.

2. Render CPU cap: per-field byte cap on description / when_to_use /
   sample_interaction.assistant via MARKETPLACE_METADATA_FIELD_MAX_BYTES
   (= 64 KiB). Without this, a 1 MiB curator markdown body × QPS =
   curator-controlled CPU burn through pure-Python markdown-it-py.
   Truncation respects UTF-8 boundaries and logs a warning so the
   curator sees the cap fire on the next sync. Test for cap +
   UTF-8-boundary preservation.

3. Inner-detail bypassed the metadata cache. _curated_inner_enrichment,
   _curated_inner_cover, and curated_detail all called
   read_marketplace_metadata directly, defeating the mtime cache the
   plugin listing already shared. Routed all three through
   _read_metadata_cached so skill/agent detail hits are O(1) re-parses
   per marketplace per mtime instead of O(QPS).

4. Truthy-vs-presence trap in plugin/inner enrichment merge. API-layer
   writers used `if resolved.get(k):` which silently dropped any
   future falsy-but-valid resolver field (bool featured=False, int
   priority=0, str category=''). Switched to presence check
   (`if k in resolved`) so the resolver is the authority on field
   presence; `{**parent, **enrichment}` merge respects whatever the
   resolver decided to ship.

5. Vendor-agnostic OSS cleanup. Removed operator-specific token
   references (/grpn-eng:, @grpn-eng:, .foundryai/) from
   src/marketplace_metadata.py docstring, app/web/templates/
   marketplace_item_detail.html JS comment, docs/curated-marketplace-
   format.md, and tests/test_marketplace_metadata.py fixtures. Replaced
   with generic /my-plugin:tool / @my-agent:role / .example/ placeholders.

CHANGELOG
---------
- New "### Fixed (PR #251 follow-ups)" section documenting all 4
  code-side must-fixes
- New "### Internal" section noting the vendor cleanup + new tests
- BREAKING bullet for the file rename now covers operator-side
  migration: running instances see plugin enrichment disappear from
  the UI until upstream curator renames + nightly sync overwrites the
  working tree; POST /api/marketplaces/{id}/sync forces refresh sooner
- Stripped /grpn-eng: leaks from the existing skill/agent rich-content
  bullet

Tests
-----
128 targeted tests pass (test_marketplace_metadata, test_marketplace_api,
test_marketplace, test_markdown_render, test_marketplace_synth_strip,
test_marketplace_filter). New tests added:
- 6 XSS regression tests on render_safe (javascript:/data:/vbscript:
  schemes via autolink, reference link, and mixed-case + positive
  http/https/mailto + noopener noreferrer rel)
- 3 byte-cap tests (truncation + UTF-8 boundary + under-cap pass-through)
- 1 cache eviction stress test (>256 marketplaces -> bounded at cap)
- 1 truthy-vs-presence resolver-contract test

Release-cut
-----------
- pyproject.toml 0.49.1 -> 0.50.0 (minor; BREAKING file rename per
  pre-1.0 CHANGELOG note: "breaking changes called out under Changed
  or Removed with the BREAKING marker")
- CHANGELOG [Unreleased] -> [0.50.0] - 2026-05-12, new empty
  [Unreleased] on top.

---------

Co-authored-by: Minas Arustamyan <arustamyan.minas@gmail.com>
Co-authored-by: ZdenekSrotyr <zdenek.srotyr@keboola.com>
2026-05-12 08:38:39 +00:00

147 lines
5.8 KiB
TOML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

[project]
name = "agnes-the-ai-analyst"
version = "0.50.0"
description = "Agnes — AI Data Analyst platform for AI analytical systems"
requires-python = ">=3.11,<3.14"
license = "MIT"
readme = "README.md"
dependencies = [
# Core database
"duckdb>=0.9.0",
# Web framework (FastAPI)
"fastapi>=0.115.0",
"uvicorn[standard]>=0.32.0",
"python-multipart>=0.0.27",
"jinja2>=3.1.0",
"starlette>=0.41.0",
# Authentication
"PyJWT>=2.8.0",
"itsdangerous>=2.1.0",
"authlib>=1.6.11",
"argon2-cffi>=23.1.0",
# HTTP client. `h2` enables HTTP/2 multiplexing for the persistent
# CLI client used by `agnes pull` (one TCP connection serves N
# concurrent parquet streams + range chunks). `cli/client.py`
# gracefully falls back to HTTP/1.1 if h2 is missing, so this
# extra is for performance, not correctness.
"httpx>=0.27.0",
"h2>=4.1.0",
# CLI
"typer>=0.12.0",
"rich>=13.0.0",
# Configuration
"python-dotenv>=1.0.0",
"pyyaml>=6.0",
# Data processing
"pandas>=2.0.0",
"pyarrow>=12.0.0",
"pytz>=2024.1",
# SQL parsing — server-side WHERE validator for /api/v2/scan (app/api/where_validator.py)
# Minimum 30.x — older versions had walk() yielding (node, parent, key)
# tuples instead of expression nodes, which would silently bypass the
# WHERE-validator structural checks (isinstance(tuple, exp.Subquery)
# is always False). 30.x yields nodes directly.
"sqlglot>=30.0.0",
# Data source connectors
"google-cloud-bigquery>=3.0.0",
"google-cloud-bigquery-storage>=2.0.0",
# Google Workspace Cloud Identity / Admin SDK (Workspace group membership sync)
"google-api-python-client>=2.0.0",
# Profiler visualizations
"matplotlib>=3.8.0",
"numpy>=1.24.0",
# Claude Code marketplace endpoint — pure-Python git server mounted in FastAPI
"dulwich>=0.22.0",
"a2wsgi>=1.10.0",
# In-process TTL cache for marketplace etag (transitively present via
# google-auth, declared explicitly here because we depend on it directly).
"cachetools>=5.3.0",
# Per-IP rate limiting on auth endpoints (#45). In-process counters by
# default — fine for single-replica deploys. Multi-replica rollouts can
# swap the storage backend via slowapi's `storage_uri` (Redis, Memcached).
"slowapi>=0.1.9",
# LLM provider SDKs — core (not dev) because connectors/llm/*_provider.py
# is imported by services/{corporate_memory, verification_detector} which
# the scheduler drives in production. Promoted from [dev] in #176 to fix
# ModuleNotFoundError boot loops on default Compose deploys.
"anthropic>=0.30.0",
"openai>=1.30.0",
# Keboola Storage API SDK — used by:
# - `connectors/keboola/client.py` for admin-side bucket / table list
# (consumed from `app/api/admin.py` discover-and-register, table
# metadata refresh).
# Extraction itself uses the lightweight `connectors/keboola/storage_api.py`
# module (export-async + signed-URL download) which talks to Storage API
# directly via `requests` — no SDK dependency on the data-path side. The
# SDK stays for the metadata reads.
"kbcstorage>=0.9.0",
"sse-starlette>=2.0",
# Optional observability — pure-Python, no compilation. Lazily initialized
# in src/observability/posthog_client.py and only emits events when
# POSTHOG_API_KEY is set in the environment. With the key unset the
# integration is fully off (no network, no init). See docs/observability.md.
"posthog>=3.7.0",
# Rust-backed (ammonia) HTML sanitizer for admin-edited rich content
# (news intro + body, curated marketplace-metadata.json descriptions).
# Allowlist-based with per-tag attribute scoping; closes the bypass
# shapes the legacy regex sanitizer in src/welcome_template.py was
# vulnerable to. Pre-built wheels published for all supported
# (mac/linux/windows × arm64/x86_64) targets.
"nh3>=0.2",
# CommonMark markdown renderer for curator-authored marketplace-metadata.json
# rich content (plugin description / sample_interaction.assistant). Pure
# Python, no compilation. Rendered output is funneled through nh3 above.
"markdown-it-py>=3.0",
# Cross-platform advisory file locking for the `agnes push` single-instance
# guard. Wraps fcntl.flock on POSIX and msvcrt.locking on Windows behind
# a uniform API; OS releases the lock automatically on process exit (no
# stale-lock detection required). Used by cli/lib/push_lock.py.
"filelock>=3.13,<4",
]
[project.optional-dependencies]
observability = [
# Already in base dependencies — listed here so operators who want to
# be explicit can `pip install -e ".[observability]"` and signal intent.
"posthog>=3.7.0",
]
dev = [
"pytest>=9.0.0",
"pytest-timeout>=2.0.0",
"pytest-xdist>=3.0.0",
"faker>=24.0.0",
# jsonschema validates the corporate-memory extraction-tool golden fixtures
# under tests/test_corporate_memory_v1.py (extraction.json, correction.json,
# confidence_calibration.json). Production code does not depend on it.
"jsonschema>=4.0.0",
# FastAPI debug toolbar — gated behind DEBUG=1 env var in app/main.py.
# Provides per-request panels (headers, routes, timer, profiling, etc.)
# for local development. Never loaded in production (no DEBUG=1 there).
"fastapi-debug-toolbar>=0.6.3",
]
[project.scripts]
agnes = "cli.main:main"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["app", "src", "connectors", "cli", "services", "config"]
[tool.ruff]
line-length = 120
target-version = "py313"
[tool.uv]
dev-dependencies = [
"pytest>=9.0.0",
"pytest-timeout>=2.0.0",
"pytest-xdist>=3.0.0",
"faker>=24.0.0",
"anthropic>=0.30.0",
"openai>=1.30.0",
"fastapi-debug-toolbar>=0.6.3",
]