* feat(bq): decouple table_registry bucket from BQ dataset name (#343)
Adds optional `bq_fqn` column (schema v51) carrying the fully-qualified
BigQuery path (project.dataset.table) so the rebuild path no longer has
to reconstruct it from the dual-purpose `bucket` field (which is also a
UX/RBAC label).
- Schema v51 migration + _SYSTEM_SCHEMA carry the nullable column;
rows without it keep using the legacy bucket+source_table+
remote_attach.project path (backwards compat).
- BQ extractor honors bq_fqn per row when present: dataset/table
override on same-project rows; cross-project VIEW path works via
bigquery_query(billing, ...); cross-project BASE TABLE skipped with
a clear warning (multi-ATTACH per project deferred to follow-up).
- Orchestrator pre-pass detects drift between extract.duckdb
_remote_attach.url and overlay data_source.bigquery.project, calls
rebuild_from_registry to regenerate when they differ. Closes the
operational hazard where /admin/server-config edits silently left
the on-disk extract pointing at the old project until the next
manual sync.
- Startup config check warns when project ≠ billing_project without
location set (the on-disk symptom is "provider returned no data"
silently in metadata cache), and when a warehouse-like data project
has no billing_project override (silent 403 serviceusage path).
- _resolve_bq_location warning now points at the location config key
explicitly so operators see the actionable fix in the log.
- POST /api/admin/register-table and PUT /api/admin/registry/{id}
accept bq_fqn; malformed values rejected at the API boundary (422).
- 25 tests covering parse_bq_fqn matrix, extractor override paths
(same-project + cross-project VIEW + cross-project BASE TABLE skip),
orchestrator drift sync, startup-validator heuristic, admin models.
UI surface for bq_fqn input in /admin/tables intentionally omitted from
this PR (3.5k-line template change) — admins can register through the
REST API or `agnes admin` CLI in the meantime. Multi-project ATTACH
support is the same scope deferral as the cross-project BASE TABLE
skip; both ride a follow-up PR.
* review fixes: abstract CHANGELOG, merge duplicate Changed, bump docs schema version
- CHANGELOG.md: remove customer-specific hostname + incident date range
from the orchestrator drift-sync entry (vendor-agnostic OSS rule),
fold the entry into the existing [Unreleased] ### Changed section
instead of opening a duplicate heading.
- docs/architecture.md: bump 'Current schema version' from 19 to 51 to
match SCHEMA_VERSION (per agnes-orchestrator skill rule #4).
* review fixes: vendor-agnostic test fixture + Schema v51 internal bullet
- tests/test_bq_fqn.py: replace customer GCP project ID with generic
'my-warehouse-project' placeholder (vendor-agnostic OSS rule). Test
asserts on the warehouse-like heuristic, not the literal project
name, so the rename is behavior-neutral.
- CHANGELOG.md: add explicit '\*\*Schema v51\*\*' bullet under
`### Internal` naming the new version + summarizing the additive
nullable column (matches the convention from v47/v48 bullets).
* fix(bq): cross-project _detect_table_type bills against extractor project
Addresses Devin review on #346 — pre-fix _detect_table_type passed the
data project as BOTH the FROM-clause target AND the bigquery_query()
first arg (billing project). For cross-project bq_fqn rows where
fqn_project != project_id, the data SA holds bigquery.dataViewer on
fqn_project but the serviceusage.services.use permission only on
project_id, so the call 403'd. init_extract's broad except Exception
swallowed the error and silently skipped the row, meaning the
cross-project VIEW path at extractor.py:~696 — the PR's primary
cross-project use case — never executed.
- Add optional billing_project kwarg to _detect_table_type; defaults
to project for backwards compat (same-project callers unaffected).
- Update the init_extract call site to pass billing_project=project_id
explicitly. Same-project rows (fqn_project == project_id) are a
no-op; cross-project rows now route billing to the project where
the SA actually has services.use.
- 2 new tests in TestDetectTableTypeBilling cover (a) explicit
billing_project routing to bigquery_query 1st arg + data project
staying in FROM, and (b) the backwards-compat default. Plus
test_cross_project_detect_call_bills_against_extractor_project
pins the call-site wiring — captures the (project, billing_project)
pair the extractor passes for a cross-project bq_fqn row.
* release: 0.54.29 — bq_fqn decoupling + marketplace refactor + setup-script UX
Accumulated [Unreleased] content from #342 (flea marketplace refactor),
#344 (setup script step-2 cwd check), and #346 (this PR — bq_fqn column
+ orchestrator drift sync + startup config check). Schema v51.
c3e82972c8