c3e82972c8
* feat(bq): decouple table_registry bucket from BQ dataset name (#343) Adds optional `bq_fqn` column (schema v51) carrying the fully-qualified BigQuery path (project.dataset.table) so the rebuild path no longer has to reconstruct it from the dual-purpose `bucket` field (which is also a UX/RBAC label). - Schema v51 migration + _SYSTEM_SCHEMA carry the nullable column; rows without it keep using the legacy bucket+source_table+ remote_attach.project path (backwards compat). - BQ extractor honors bq_fqn per row when present: dataset/table override on same-project rows; cross-project VIEW path works via bigquery_query(billing, ...); cross-project BASE TABLE skipped with a clear warning (multi-ATTACH per project deferred to follow-up). - Orchestrator pre-pass detects drift between extract.duckdb _remote_attach.url and overlay data_source.bigquery.project, calls rebuild_from_registry to regenerate when they differ. Closes the operational hazard where /admin/server-config edits silently left the on-disk extract pointing at the old project until the next manual sync. - Startup config check warns when project ≠ billing_project without location set (the on-disk symptom is "provider returned no data" silently in metadata cache), and when a warehouse-like data project has no billing_project override (silent 403 serviceusage path). - _resolve_bq_location warning now points at the location config key explicitly so operators see the actionable fix in the log. - POST /api/admin/register-table and PUT /api/admin/registry/{id} accept bq_fqn; malformed values rejected at the API boundary (422). - 25 tests covering parse_bq_fqn matrix, extractor override paths (same-project + cross-project VIEW + cross-project BASE TABLE skip), orchestrator drift sync, startup-validator heuristic, admin models. UI surface for bq_fqn input in /admin/tables intentionally omitted from this PR (3.5k-line template change) — admins can register through the REST API or `agnes admin` CLI in the meantime. Multi-project ATTACH support is the same scope deferral as the cross-project BASE TABLE skip; both ride a follow-up PR. * review fixes: abstract CHANGELOG, merge duplicate Changed, bump docs schema version - CHANGELOG.md: remove customer-specific hostname + incident date range from the orchestrator drift-sync entry (vendor-agnostic OSS rule), fold the entry into the existing [Unreleased] ### Changed section instead of opening a duplicate heading. - docs/architecture.md: bump 'Current schema version' from 19 to 51 to match SCHEMA_VERSION (per agnes-orchestrator skill rule #4). * review fixes: vendor-agnostic test fixture + Schema v51 internal bullet - tests/test_bq_fqn.py: replace customer GCP project ID with generic 'my-warehouse-project' placeholder (vendor-agnostic OSS rule). Test asserts on the warehouse-like heuristic, not the literal project name, so the rename is behavior-neutral. - CHANGELOG.md: add explicit '\*\*Schema v51\*\*' bullet under `### Internal` naming the new version + summarizing the additive nullable column (matches the convention from v47/v48 bullets). * fix(bq): cross-project _detect_table_type bills against extractor project Addresses Devin review on #346 — pre-fix _detect_table_type passed the data project as BOTH the FROM-clause target AND the bigquery_query() first arg (billing project). For cross-project bq_fqn rows where fqn_project != project_id, the data SA holds bigquery.dataViewer on fqn_project but the serviceusage.services.use permission only on project_id, so the call 403'd. init_extract's broad except Exception swallowed the error and silently skipped the row, meaning the cross-project VIEW path at extractor.py:~696 — the PR's primary cross-project use case — never executed. - Add optional billing_project kwarg to _detect_table_type; defaults to project for backwards compat (same-project callers unaffected). - Update the init_extract call site to pass billing_project=project_id explicitly. Same-project rows (fqn_project == project_id) are a no-op; cross-project rows now route billing to the project where the SA actually has services.use. - 2 new tests in TestDetectTableTypeBilling cover (a) explicit billing_project routing to bigquery_query 1st arg + data project staying in FROM, and (b) the backwards-compat default. Plus test_cross_project_detect_call_bills_against_extractor_project pins the call-site wiring — captures the (project, billing_project) pair the extractor passes for a cross-project bq_fqn row. * release: 0.54.29 — bq_fqn decoupling + marketplace refactor + setup-script UX Accumulated [Unreleased] content from #342 (flea marketplace refactor), #344 (setup script step-2 cwd check), and #346 (this PR — bq_fqn column + orchestrator drift sync + startup config check). Schema v51. |
||
|---|---|---|
| .. | ||
| admin | ||
| archive | ||
| examples | ||
| HOWTO | ||
| metrics | ||
| operator | ||
| setup | ||
| superpowers | ||
| testing | ||
| ADR-corporate-memory-v1.md | ||
| agent-setup-prompt.md | ||
| agent-workspace-prompt.md | ||
| architecture.md | ||
| auth-google-oauth.md | ||
| auth-groups.md | ||
| CONFIGURATION.md | ||
| corporate-memory-governance.md | ||
| curated-marketplace-format.md | ||
| DATA_SOURCES.md | ||
| DEPLOYMENT.md | ||
| development.md | ||
| HEADLESS_USAGE.md | ||
| initial-workspace-override.md | ||
| llm-routing.md | ||
| local-development.md | ||
| marketplace.md | ||
| observability.md | ||
| ONBOARDING.md | ||
| PLATFORM_SETUP.md | ||
| QUICKSTART.md | ||
| RBAC.md | ||
| README.md | ||
| RELEASE_CHECKLIST.md | ||
| RELEASING.md | ||
| sample-data.md | ||
| state-dir.md | ||
| STORE_GUARDRAILS.md | ||
| theme-reference.html | ||
Agnes documentation
Index of all documentation, organized by who needs it. New here? Start with the row that matches your role.
| You are… | Start with |
|---|---|
| Analyst — using Agnes to query data | QUICKSTART.md, then HOWTO/ |
| Operator — deploying & running an instance | PLATFORM_SETUP.md |
| Developer — working on Agnes itself | ../ARCHITECTURE.md + architecture.md |
For analysts
Using the platform to analyze data.
QUICKSTART.md— local setup + first syncHOWTO/— task-oriented cookbook (querying, snapshots, common workflows)DATA_SOURCES.md— data source connectors (Keboola, BigQuery, CSV) and how tables surfacemetrics/— canonical business-metric definitions (YAML)HEADLESS_USAGE.md— PAT auth for CI / headless clients
For operators
Deploying, configuring, and running an Agnes instance.
PLATFORM_SETUP.md— the consolidated operator playbook (bootstrap, TLS, marketplaces, scheduler, telemetry)ONBOARDING.md— end-to-end Terraform deployment into a new GCP projectDEPLOYMENT.md— picks between the Terraform and Docker Compose pathsCONFIGURATION.md—instance.yaml, env vars, per-instance optionsstate-dir.md— persistent data layout (data+statetiers, mount layouts, migration)RBAC.md— access control: groups, members, resource grantsauth-google-oauth.md— Google OAuth setup + operator gotchasauth-groups.md— Google Workspace group syncadmin/query-modes.md— table registration query modesagent-setup-prompt.md— customize the/setuppage banneragent-workspace-prompt.md— customize the generated analystCLAUDE.mdinitial-workspace-override.md— per-instance analyst-workspace skeleton overridecurated-marketplace-format.md— authoringmarketplace-metadata.jsonfor curated marketplacesobservability.md— PostHog integration (exceptions, tracing, session replay)operator/news-content-guide.md— editorial guidelines for in-app news content
For developers
Working on the Agnes codebase.
../ARCHITECTURE.md— high-level system overview (the summary)architecture.md— detailed architecture reference (module map, extract.duckdb contract, components)../CLAUDE.md— project instructions for AI agents working in this repodevelopment.md— logging, request correlation, debug toolbarlocal-development.md—LOCAL_DEV_MODEsetup (what's mocked vs. real)RELEASING.md— release process, deploy workflows, CI quirksRELEASE_CHECKLIST.md— pre-merge checks for bootstrap-path changestesting/— test plans (clean-analyst bootstrap, VM test)marketplace.md— Claude Code marketplace ingestion + re-serving internalsSTORE_GUARDRAILS.md— flea-market upload guardrails (static checks + LLM review)corporate-memory-governance.md— knowledge-distribution governance designADR-corporate-memory-v1.md— ADR: corporate-memory v1 decisionsllm-routing.md— design: provider-agnostic LLM routingsample-data.md— sample data generator (e-commerce schema, size presets)theme-reference.html— web UI theme/color reference../dev_docs/— server/developer-internal docs (not synced to analyst machines): server ops, disaster recovery, security audit, desktop app, design system, Telegram bot
Code-adjacent READMEs: ../connectors/jira/README.md,
../services/corporate_memory/README.md,
../scripts/README.md.
Agent skill files: ../cli/skills/.
Other
../CHANGELOG.md— full change history (Keep-a-Changelog format)archive/— historical planning artifacts and superseded docs; not maintained, seearchive/README.md