## Summary
Two bugs in `agnes describe` surfaced from a real analyst session following the CLAUDE.md agent-rails discovery workflow. Together they break `agnes describe` end-to-end for any analyst (or analyst-AI) who follows the documented form.
### A) CLI parsing
`agnes describe TABLE -n 5` failed with `Missing argument 'TABLE_ID'`. Root cause: the command was registered as a `Typer.Typer` subcommand group via `app.add_typer(describe_app, name="describe")` + `@describe_app.callback(invoke_without_command=True)`, and that pattern mis-parses positional + short-int option in some orderings. Same pattern in `cli/commands/schema.py` works only because schema has no INTEGER short option. Fix: switch to flat `@app.command("describe")`.
### B) Server NaN
`/api/v2/sample/<id>` (called by `agnes describe`) returned HTTP 500 with `ValueError: Out of range float values are not JSON compliant: nan` whenever a row contained NaN. Fix: sanitize NaN/±inf to None before JSON serialization.
## Test plan
- [x] `pytest tests/test_cli_describe*.py` — added regression tests pinning `-n` parsing on either side of the positional.
- [x] `pytest tests/test_api_v2_sample*.py` — added regression test for NaN row → JSON `null` (not 500).
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/keboola/agnes-the-ai-analyst/pull/224" target="_blank">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1" alt="Open in Devin Review">
</picture>
</a>
<!-- devin-review-badge-end -->
Adds end-to-end flow for installing and keeping the per-user filtered
Claude Code marketplace in sync with the user's Agnes stack
(admin RBAC grants \ MyAIStack opt-outs U /store installs).
Setup (one-liner in install prompt step 5):
`agnes refresh-marketplace --bootstrap` clones the per-user marketplace
bare repo to ~/.agnes/marketplace, strips PAT from the cloned origin
URL, registers the local path with Claude Code, and installs every
plugin in the served manifest at --scope project. Replaces a 15-line
inline shell sequence that tripped Claude Code's agent-driven `rm -rf`
permission gate.
Auto-refresh (SessionStart hook installed by `agnes init`):
`agnes refresh-marketplace --quiet` runs every Claude Code session,
fetches+resets the clone (server rebuilds as orphan commits, so
pull --ff-only is impossible), and version-aware reconciles:
- missing in workspace -> claude plugin install <name>@agnes --scope project
- version differs -> claude plugin update <name>@agnes
- matches -> skip
Don't auto-uninstall plugins that disappeared from the manifest --
a transient empty manifest from the server would wipe the stack.
Hook output: when --quiet AND something actually changed, emits Claude
Code hook JSON on stdout -- `systemMessage` (transient toast) and
`hookSpecificOutput.additionalContext` (model-side system reminder),
both carrying the change summary plus a "/exit + restart Claude Code"
instruction (Claude only scans plugins at session start).
Windows hook compatibility: the refresh-marketplace hook command is
wrapped in `bash -c "..."` because Claude Code on Windows runs hook
commands directly without invoking a shell, so `2>/dev/null || true`
would otherwise be passed as literal argv tokens.
Cross-cutting:
- cli/lib/marketplace.py: shared CLONE_DIR + MARKETPLACE_NAME constants.
- cli/lib/hooks.py: SessionStart now has two independent entries
(pull + refresh-marketplace) so a failure in one doesn't suppress
the other; legacy `da sync` and prior single-pull layouts upgrade
cleanly on re-init.
- PAT injection on every git fetch via per-invocation credential
helper (token in \$AGNES_TOKEN env, never in argv or .git/config).
- Pre-snapshot of installed plugins captured BEFORE
`claude plugin marketplace update` so silent auto-applied version
bumps still fire notifications.
- scripts/dev/agnes-client-reset.sh: cleans ~/.claude/plugins/marketplaces/agnes,
~/.claude/plugins/cache/agnes, drops uv build cache, documents
workspace-scoped residue that can't be enumerated from the script.
- app/web/setup_instructions.py: legacy AGNES_DEBUG_AUTH path also
uses clone (direct HTTPS marketplace add is broken end-to-end on
every Claude Code distribution -- stores response as single file,
plugin source paths then 404).
28 new tests (test_cli_refresh_marketplace.py) + extended hook + setup
template tests cover bootstrap, fetch+reset ordering, version-aware
reconcile, project-path filtering, hook JSON shape, and the bash-c
Windows wrapper invariant.
Reuses cli.update_check.check() for the version probe — extended with
bypass_disabled=True so explicit user-typed self-upgrade is not silenced
by AGNES_NO_UPDATE_CHECK (which is for the implicit warning loop).
Install path: uv tool install --force when uv is on PATH; otherwise
curl + pip via sys.executable (NOT system python3, NOT --user — both
would land outside the agnes venv and silently no-op the upgrade).
Smoke test execs the binary at the install-resolved path (uv tool dir
joined with agnes-the-ai-analyst/bin/agnes, or sys.executable's sibling
agnes for pip) — never via shutil.which, which can resolve a stale shadow
on PATH and produce a false-positive smoke pass on the OLD version. Smoke
also asserts --version output contains info.latest via PEP 440 Version()
equality (so 0.40.0 does not falsely match 0.40.10).
On smoke fail: rollback to last_known_good.json (written only after a
previous run's smoke passed). Rollback rc is captured and surfaced on
stderr if it also fails. First-ever upgrade or unrecoverable rollback
prints the canonical bootstrap recovery: curl -fsSL <server>/cli/install.sh | bash.
AGNES_SELF_UPGRADE_IN_PROGRESS=1 is set for the duration of the run
and propagated to the smoke-test subprocess. Layer B's _check_version_headers
honors the sentinel and skips the < min hard-stop, so an in-flight
upgrade can never sys.exit(2) itself.
--force invalidates the update_check cache BEFORE probing. --force +
offline = exit 1 with explicit stderr (without --force, offline is silent).
--quiet suppresses progress output but never gags failure stderr.
Three first-try-failure-surface fixes from Pavel's #185 trace + the
template guidance question, all under PR #188's umbrella so they land
together with the file_server / parallel pull / Tier 1 work.
1. CLI clean-error wrapper — new AgnesTransportError raised by the
api_*/stream_download helpers when httpx times out / drops /
refuses, plus a top-level Typer wrapper (cli/main.py) that prints
one-line "Error: …" + actionable hint and exits non-zero. Full
traceback goes to ~/.config/agnes/last-error.log for support
forwarding. Unhandled Exceptions are caught at the same boundary
so no Python traceback ever leaks to the analyst's terminal.
Pavel's #185 Phase 3B: a 30-frame httpx traceback from a slow BQ
--remote query made it look like a CLI bug. Now: clean message +
hint pointing at `agnes snapshot create` / partition-column
guidance.
Entry point in pyproject.toml flipped from `cli.main:app` →
`cli.main:_run_with_clean_errors` so the wrapper actually runs
under the installed `agnes` binary.
2. agnes init / agnes pull --skip-materialize + progress bar.
--skip-materialize omits query_mode='materialized' rows from the
download set so a first init doesn't spend 44 minutes silently
pulling a single 6 GB parquet (Pavel's #185 Phase 1). Rich-driven
per-file progress bar with label/bytes/rate/ETA renders to stderr
when not --quiet and not --json. Aggregates across the parallel
ThreadPoolExecutor workers added earlier in this PR.
3. config/claude_md_template.txt: explicit one-line snippet pointing
at `agnes catalog --json | jq '.tables[] | select(.id=="<id>")'`
for per-table descriptions + restated invariant: "the description
field on each catalog row is the authoritative business-rules
text — re-read live, never copy into this file." Resolves the
regression-or-feature debate between Pavel (wants annotations)
and the user feedback that landed in the prior commit (don't
embed table-specific content; tables change). Catalog command
stays the source of truth.
Ports Minas's PR #172 (against pre-rename `da` CLI on main) and applies
the principle to the post-rename `agnes` CLI. Two distinct failure modes
on Windows consoles whose default codepage is cp1250 (cs-CZ) / cp1252
(en-US):
1. `agnes pull` and other Rich-progress codepaths
UnicodeEncodeError on Braille spinner glyphs. Fix: `cli/main.py`
reconfigures stdout/stderr to UTF-8 with errors='replace' at import
time on `sys.platform == 'win32'` so Rich's legacy-Windows render
path emits decodable bytes. Wrapped in try/except so pytest's
captured streams (which aren't TextIOWrapper) don't break.
2. `agnes skills list` and `agnes skills show`
UnicodeDecodeError when reading skill markdown containing em-dashes /
accented chars. Default `Path.read_text()` uses
locale.getpreferredencoding(False), which is the broken codepage on
Windows. Fix: every call site passes encoding='utf-8' explicitly.
Broader scope than #172 because:
- The bootstrap rewrite renamed/removed several files Minas's PR
patched (`cli/commands/analyst.py` -> rolled into init.py;
`cli/commands/sync.py` -> split into pull/push). Those targets no
longer exist; the equivalent code lives in init.py.
- Other call sites Minas didn't touch (still bare in his branch) are
patched here too — config.py / update_check.py / snapshot_meta.py /
setup.py / skills.py — so the codebase has zero locale-default text
I/O in cli/.
Side cleanup: stale `Run `da`` reference in snapshot_meta.py:88 fixed
to `agnes` while touching the file.
Replaces the BigQuery wrap-view pattern with a discovery + scoped-fetch toolkit driven by the analyst's Claude session. Adds /api/v2/{catalog,schema,sample,scan,scan/estimate}, da catalog/schema/describe/fetch/snapshot/disk-info CLI commands, sqlglot-backed WHERE validator, process-local quota tracker, agent rails skill (cli/skills/agnes-data-querying.md). BREAKING: BQ wrap views off by default — set data_source.bigquery.legacy_wrap_views=true for one cycle. Backward-compat field_validator on primary_key. Catalog cache now matches documented 300s TTL with RBAC fresh per request. Cuts release v0.14.0.
* fix(cli): versioned wheel URL in setup instructions; drop broken /cli/agnes.whl alias (#36)
* fix(cli): inline PEP 427 wheel filename in setup instructions
`uv tool install <server>/cli/agnes.whl` fails with
error: The wheel filename "agnes.whl" is invalid: Must have a version
because uv validates the filename in the URL path *before* fetching — so
the server-side Content-Disposition header (which has the real versioned
filename) is never consulted, and an HTTP redirect does not help either:
uv resolves the filename from the initial URL.
Fix the root cause by inlining the real PEP 427 filename into the setup
snippet the dashboard copies to the clipboard. The wheel filename is
resolved server-side via `_find_wheel()` and substituted into the lines
returned from `setup_instructions.resolve_lines()`, so both the read-only
HTML preview and the JS clipboard renderer get byte-identical output.
Also added `/cli/wheel/{filename}` to serve wheels at their PEP 427 path,
and kept `/cli/agnes.whl` as a 302 redirect for manual/legacy callers —
though that redirect alone is NOT sufficient for `uv tool install` (uv
validates before following redirects) and is there only as defense-in-depth.
Verified locally:
- `uv tool install <server>/cli/wheel/agnes_the_ai_analyst-2.0.0-py3-none-any.whl` succeeds
- `/install` HTML now renders the versioned URL; `/cli/agnes.whl` no longer appears in the rendered snippet
* fix(cli): remove /cli/agnes.whl alias entirely — it only confused users
The bareword alias was never actually usable:
- `uv tool install <server>/cli/agnes.whl` fails at filename validation
before any HTTP fetch, so neither the Content-Disposition header nor a
302 redirect rescued it.
- The 302-to-versioned-path fallback left a visibly "working" URL in
browser / curl -L contexts, which is exactly how the original bug got
reported in the first place ("the URL loads, why doesn't install work?").
Remove the endpoint and scrub all remaining references. The only CLI wheel
URL is now `/cli/wheel/{filename}` with the real PEP 427 filename, which
the setup-instructions template already generates server-side.
Existing tests that referenced /cli/agnes.whl become negative tests
("must not appear") so we don't regress.
* feat(cli): --version flag; sync --dry-run + progress indicator (#38)
* feat(cli): add --version / -V flag
Prints `da <version>` from package metadata (importlib.metadata). Falls
back to "unknown" when the package is not installed (e.g. running from a
source checkout without `uv pip install -e .`), instead of crashing.
Eager typer callback, so `da --version` exits before subcommand
resolution and does not require any auth/config.
* feat(cli): da sync --dry-run + X/N progress indicator
--dry-run reports what would be downloaded/uploaded without hitting the
API or writing local state. Supports the full flag set (--table, --json,
--upload-only); JSON shape is {"dry_run": true, "would_download": [...],
"summary": {...}}.
Progress bar now shows "[X/N] Downloading <table>..." with a Rich
BarColumn + TaskProgressColumn + TimeElapsedColumn instead of a bare
spinner — makes long syncs visible.
* feat(cli): durable sync + server gzip + auto-update check (#41)
* fix(sync): atomic writes + manifest hash verification + retry on transient errors
Three durability hooks around stream_download and the sync command:
1. Atomic writes. stream_download now streams into `<target>.tmp` and
calls os.replace() on success, so the real target file never exists
in a half-written state. On failure the tmp is unlinked — no cleanup
leftovers, no guard needed at read time.
2. Retry with backoff. Transient errors (ConnectError, ReadError,
WriteError, RemoteProtocolError, TimeoutException, 5xx) are retried
up to 3× with 0.3s / 1s / 3s backoff. 4xx (auth, 404) surfaces
immediately — retrying those is pointless.
3. Manifest-hash verification. After download, sync.py computes MD5 of
the target (same 8KiB chunking as app/api/sync.py:_file_hash) and
compares against `server_tables[tid]["hash"]`. Mismatch ⇒ unlink,
record error, skip state commit. The PAR1 structural check survives
as a fallback for legacy manifests without a hash.
Also makes _rebuild_duckdb_views tolerant: single broken parquet is
skipped with a stderr warning instead of killing the whole rebuild.
Supersedes #40 — this commit is a strict super-set (hash check + PAR1
fallback + atomic write + retry). #40 can be closed without merging.
* perf(server): enable GZipMiddleware for JSON / HTML responses
GZipMiddleware at minimum_size=1024 shaves bandwidth on manifest-style
JSON endpoints (/api/sync/manifest, /api/version, …) and the /install
HTML preview. Parquet file downloads are already columnar-compressed so
the middleware sees limited benefit there — but it doesn't hurt, httpx
on the client side decompresses transparently.
Placed after session middleware so gzip wraps the session-Set-Cookie
response too, and before CORSMiddleware so compression is applied to
both cross-origin and same-origin responses.
* feat(cli): auto-check for newer CLI version on startup
Server side
- GET /cli/latest returns {version, wheel_filename, download_url_path}
for whatever wheel is currently in AGNES_CLI_DIST_DIR. Public,
cacheable, no secrets — consumed by the CLI auto-update probe.
Client side
- New cli/update_check.py: reads /cli/latest with a 3s timeout, caches
the result in $DA_CONFIG_DIR/update_check.json for 24h. Cache is
invalidated when the installed version changes (e.g. after a fresh
`uv tool install`) so stale "you're behind" warnings don't linger.
- Root typer callback fires the probe before subcommand dispatch; any
failure is swallowed so a bad network never blocks a working command.
- Outdated → one-line stderr warning:
[update] da 2.0.0 is out of date — latest on this server is 2.1.0.
Upgrade: uv tool install --force <server>/cli/wheel/<…>.whl
- Disable with DA_NO_UPDATE_CHECK=1.
* fix(pr-review): None-guard the upgrade line + skip gzip on parquet paths
Two follow-ups from Devin review on #41.
1. format_outdated_notice(UpdateInfo(download_url=None)) emitted literal
"uv tool install --force None" — copy-pasting that fails. Drop the
upgrade snippet when the URL is absent and keep only the version line.
2. GZipMiddleware compressed everything over 1024 bytes, including the
parquet FileResponses served by /api/data/{tid}/download,
/cli/wheel/{name}, and /cli/download. Parquet is already columnar-
compressed — gzip there is pure CPU + latency with no size win, and
/api/data bodies can reach hundreds of MB. Wrap GZipMiddleware in a
small _SelectiveGZipMiddleware that skips those path prefixes and
delegates the rest to the stock middleware. JSON / HTML endpoints
(manifest, /install, /api/version, …) still get compressed.
* release: bump to 2.1.0 — unify AGNES_VERSION with pyproject.toml version (#42)
Before: two independent version systems. pyproject.toml carried semver
(2.0.0 → wheel filename → `da --version`) while release.yml injected
CalVer into AGNES_VERSION (e.g. 2026.04.155 → /api/version). Users saw
different strings in the CLI vs. the /install page, and the CLI auto-
update check couldn't tell "new deploy, same package version" apart
from "new package version".
Make pyproject.toml [project].version the single product-version source
of truth. release.yml extracts it and feeds AGNES_VERSION, so every
surface (/api/version, /api/health, /cli/latest, `da --version`) agrees
on one number. The CalVer tag keeps doing what CalVer is for: release
identity on the git tag and Docker image tag (versioned_tag).
Also wires AGNES_TAG through the build: release.yml → Dockerfile ARG →
env, so /api/version.image_tag finally reports the actual image tag
instead of the "unknown" fallback.
Bump to 2.1.0 to reflect the PRs shipped on ps/wheel-name-fix: durable
sync (atomic writes + manifest MD5 + retry), server GZip, CLI auto-
update probe, setup snippet PEP 427 URL.
* fix(pr-review): directional version compare in is_outdated()
UpdateInfo.is_outdated() used `self.latest != self.installed`, which
fires in both directions. If the server is rolled back or the user
connects to an older deployment, the CLI would warn "out of date"
and — worse — the formatted notice would prompt
uv tool install --force <older-version>.whl
i.e. an unintended downgrade.
Compare with packaging.version.Version (PEP 440 aware, handles pre-
release tags). Fall back to dotted-int tuple compare if packaging is
somehow missing, and return False on unparseable strings — better to
miss an upgrade hint than to silently suggest a downgrade.
Adds 4 test cases: installed older (True), installed newer (False),
10.0.0 vs 2.1.0 lexical-compare trap (correct), unparseable strings
(False).
Addresses Devin review on #43.
* fix(pr-review): read FastAPI app version from package metadata
app/main.py:80 hardcoded `version="2.0.0"` in the FastAPI constructor.
After #42 bumped pyproject.toml to 2.1.0, /api/version, /cli/latest,
and `da --version` all reported 2.1.0 while /openapi.json and the
/docs UI still advertised 2.0.0.
Read `agnes-the-ai-analyst` version via importlib.metadata (same
pattern cli/main.py:_cli_version already uses), with a `"dev"`
fallback when the package is not installed (source checkout). This
way pyproject.toml stays the single source of truth across every
version surface — /openapi.json now tracks the bump automatically.
Adds a dedicated test file to pin this behavior so a future
regression to a hardcoded literal fails at CI.
Addresses second Devin finding on #43.
* fix(pr-review): _fmt_bytes PiB label + negative cache in update_check
Two more follow-ups from Devin review on #43.
1. _fmt_bytes off-by-unit. The old loop exited at TiB but the fallback
labelled PiB, so 1 PiB rendered as "1024.0 PiB". Restructure: put
every unit inside the loop (KiB through EiB) so the division count
always matches the label. Covers up to 1 ZiB cleanly; anything
beyond renders as "<big>.0 EiB" rather than crashing.
2. Negative cache for failed /cli/latest probes. On a corporate
firewall / VPN that silently drops packets, the 3s HTTP timeout
fired on *every* `da` invocation. Writing a `latest=None` cache
entry with a 5-minute TTL caps that at one probe per 5min. Successful
probes still use the 24h TTL. Reading logic branches on whether the
cached `latest` is None.
Adds TestFmtBytes (2 cases: small/medium sizes and the PiB/EiB fallback
regression), plus two TestSync update-check cases covering negative-
cache reuse and TTL expiry.
Implements Task 4 — five Typer commands under `da metrics`:
- list/show use api_get() to query the server API
- import/export/validate access DuckDB directly via MetricRepository
and TableRegistryRepository (no server required)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
