agnes-the-ai-analyst/app/web/templates
ZdenekSrotyr aa5921da67
release: 0.47.0 — source-agnostic catalog metadata + cache discipline (#223)
## Summary

- Catalog enrichment for `query_mode='remote'` rows: `rows`, `size_bytes`, `partition_by`, `clustered_by` per table (BQ + Keboola providers).
- `/api/v2/schema/{id}` cache miss: 2 BQ jobs → 1 (-50%) via shared `fetch_bq_columns_full`.
- All four catalog/schema/sample/metadata caches flush on registry change; single-row re-warm scheduled.
- Automatic cache warmup at server startup (bounded concurrency, opt-out via `AGNES_SKIP_CACHE_WARMUP=1`).
- SSE-driven freshness toolbar on `/admin/tables` with progress bar, log, and per-row badge.
- New admin doc `docs/admin/query-modes.md` — single source of truth on `local` / `remote` / `materialized` choice.

Closes #155.
Closes #156.

## Test plan

- [x] 65+ targeted tests pass across 11 new test modules + 3 modified ones.
- [x] No DB migration; no wire-break; `MIN_COMPAT_CLI_VERSION` unchanged.
- [ ] Reviewer: register a remote BQ table via `/admin/tables`, observe the toolbar populates within ~2 s and the per-row badge transitions warming → fresh.
- [ ] Reviewer: trigger `Re-warm all`, verify SSE log scrolls and `cacheWarmupBar` progresses.
- [ ] Reviewer: edit a registered row's bucket, verify `agnes schema <id>` returns updated columns immediately (no 1-hour staleness).
- [ ] Reviewer: confirm `agnes admin register-table --query-mode remote` prints the new IAM-smoke-check hint.

## Notable design decisions

- BigQuery `INFORMATION_SCHEMA.TABLE_STORAGE` is the only valid scope for size+rows (verified live 2026-05-07; dataset-scoped doesn't exist). Region resolved from `instance.yaml.data_source.bigquery.location` → `bq.client().get_dataset(...)` → fall back to legacy `__TABLES__`.
- VIEW handling: TABLE_STORAGE returns no rows for views, fall through to `__TABLES__` (also empty) → `TableMetadata(rows=None, size_bytes=None, partition_by=..., clustered_by=...)`. Null size signals analyst Claude to apply existing CLAUDE.md guidance.
- `size_bytes` is `active_logical_bytes + long_term_logical_bytes` — full BQ scan reads both; reporting only active undercounts aged partitioned tables.
- Source-agnostic provider seam: per-source `connectors/<source>/metadata.py:fetch(MetadataRequest)`; dispatcher in `app/api/v2_catalog.py:_metadata_provider_for` lazily imports per source_type so a Keboola-only deployment doesn't pay the BQ-extension import cost.
- Warmup non-blocking: FastAPI `lifespan` schedules `asyncio.create_task(_warm_catalog_caches_bg)` before `yield`. Per-row failures isolated.

## Out of scope

- Profile / column histograms / dimension cardinality for remote tables (separate issue).
- Onboarding nudge ("you have 0 remote tables, consider registering some BQ ones") — separate UX call.
- Provider plug-in registration via entry-points (the dispatch table is a hardcoded if-tree today; one line per future source).

## Release

Bumps `pyproject.toml` 0.46.1 → 0.47.0 (main shipped 0.46.0 + 0.46.1 during this PR — see commit `d98976ec`). New CHANGELOG section under `## [0.47.0] — 2026-05-07`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/keboola/agnes-the-ai-analyst/pull/223" target="_blank">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
    <img src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1" alt="Open in Devin Review">
  </picture>
</a>
<!-- devin-review-badge-end -->
2026-05-07 18:33:55 +02:00
..
_app_header.html Merge remote-tracking branch 'origin/main' into pr180-review 2026-05-05 12:05:50 +02:00
_claude_setup_instructions.jinja feat(web): rename /install → /setup; nav label 'Setup local agent' 2026-05-03 16:12:13 +02:00
_message.html feat(auth): password reset & invite flows for web + admin (#34) (#37) 2026-04-22 17:43:57 +02:00
_theme.html feat: complete system — web UI, all API endpoints, governance, admin, CLI commands 2026-03-27 16:52:22 +01:00
_version_badge.html feat(ui): version badge as shared partial, injected into every full-page template 2026-04-21 20:51:55 +02:00
activity_center.html User management + PAT + CLI distribution + HTML auth redirect (#9 #10 #11 #12) (#28) 2026-04-22 14:24:28 +02:00
admin_access.html feat(admin-ui): /admin/tables per-connector tabs + Keboola materialized parity + form cleanup + Manage access deep link 2026-05-01 20:26:29 +02:00
admin_group_detail.html feat(admin): users/groups UI polish + SSO lock + v18 migration (#142) 2026-04-30 15:16:04 +02:00
admin_groups.html feat(admin): users/groups UI polish + SSO lock + v18 migration (#142) 2026-04-30 15:16:04 +02:00
admin_marketplaces.html feat(store): /store + /my-ai-stack — community marketplace + per-user composition 2026-05-05 02:53:49 +02:00
admin_scheduler_runs.html fix(anthropic): strict json_schema (additionalProperties=false) + add /admin/scheduler-runs UI 2026-05-05 08:00:57 +02:00
admin_server_config.html feat(config): expose materialize.lock_ttl_seconds in server-config 2026-05-04 18:52:54 +02:00
admin_tables.html release: 0.47.0 — source-agnostic catalog metadata + cache discipline (#223) 2026-05-07 18:33:55 +02:00
admin_tokens.html User management + PAT + CLI distribution + HTML auth redirect (#9 #10 #11 #12) (#28) 2026-04-22 14:24:28 +02:00
admin_user_detail.html fix(admin/users): explain empty group dropdown instead of silent placeholder 2026-05-07 09:09:45 +02:00
admin_users.html feat(admin): users/groups UI polish + SSO lock + v18 migration (#142) 2026-04-30 15:16:04 +02:00
admin_welcome.html fix(ui): tighten dashboard token row gap; lift editor/preview labels above panes 2026-05-03 19:51:34 +02:00
admin_workspace_prompt.html chore(cli-rename): replace stale da verbs in active code paths 2026-05-04 21:10:43 +02:00
base.html User management + PAT + CLI distribution + HTML auth redirect (#9 #10 #11 #12) (#28) 2026-04-22 14:24:28 +02:00
base_login.html feat(ui): version badge as shared partial, injected into every full-page template 2026-04-21 20:51:55 +02:00
catalog.html feat(rbac): drop dataset_permissions + users.role + is_public; v19 migration (#150) 2026-04-30 22:02:16 +02:00
corporate_memory.html fix(ui): admin pending-review banner on /corporate-memory (#176) 2026-05-05 00:01:22 +02:00
corporate_memory_admin.html feat(memory): admin Edit + MEMORY_DOMAIN RBAC + ai-section UI (#141) 2026-04-30 11:04:41 +02:00
dashboard.html fix(devin-review): address 4 findings on PR #167 2026-05-03 21:15:01 +02:00
desktop_link.html feat: complete system — web UI, all API endpoints, governance, admin, CLI commands 2026-03-27 16:52:22 +01:00
error.html feat(observability): request_id end-to-end + dev debug toolbar + centralized logging (#136) 2026-04-29 22:54:21 +02:00
install.html refactor(install.html): single tile, single PAT-mint body shape 2026-05-04 22:18:00 +02:00
login.html feat(auth): Google Workspace group prefix filter + system mapping (#131) 2026-04-29 14:08:04 +02:00
login_email.html feat(auth): password reset & invite flows for web + admin (#34) (#37) 2026-04-22 17:43:57 +02:00
login_magic_link.html feat: complete system — web UI, all API endpoints, governance, admin, CLI commands 2026-03-27 16:52:22 +01:00
login_magic_link_sent.html feat: complete system — web UI, all API endpoints, governance, admin, CLI commands 2026-03-27 16:52:22 +01:00
me_debug.html feat(auth): /me/debug self-only auth diagnostic page (#116) 2026-04-29 06:36:28 +02:00
my_ai_stack.html feat(store): /store + /my-ai-stack — community marketplace + per-user composition 2026-05-05 02:53:49 +02:00
my_tokens.html User management + PAT + CLI distribution + HTML auth redirect (#9 #10 #11 #12) (#28) 2026-04-22 14:24:28 +02:00
password_reset.html feat(auth): password reset & invite flows for web + admin (#34) (#37) 2026-04-22 17:43:57 +02:00
password_setup.html feat(auth): password reset & invite flows for web + admin (#34) (#37) 2026-04-22 17:43:57 +02:00
profile.html feat(admin): users/groups UI polish + SSO lock + v18 migration (#142) 2026-04-30 15:16:04 +02:00
profile_sessions.html feat(profile): per-session jsonl download from /profile/sessions 2026-05-05 09:15:12 +02:00
setup.html fix: address PR review findings — config write, CalVer, error handling 2026-04-10 13:16:40 +02:00
store_detail.html fix(store): security + correctness blockers found in PR review (F1, F2, F4, F5) 2026-05-05 08:18:02 +02:00
store_listing.html feat(store): /store + /my-ai-stack — community marketplace + per-user composition 2026-05-05 02:53:49 +02:00
store_upload.html feat(store): /store + /my-ai-stack — community marketplace + per-user composition 2026-05-05 02:53:49 +02:00