agnes-the-ai-analyst/docs/archive/superpowers/plans/2026-04-29-issue-134-bq-access-unify-plan.md

Issue #134 — Unify BigQuery access behind BqAccess (implementation plan)

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Fix issue #134 (v2 BigQuery endpoints returning bare HTTP 500) by (a) translating Google API errors into structured 502/400 responses and (b) extracting the duplicated BQ-access pattern into a single BqAccess facade in connectors/bigquery/access.py.

Architecture: Two-phase rollout. Phase 1 = inline try/except in three v2 endpoint files emitting the final structured body shape — fixes Pavel's bug atomically. Phase 2 = extract BqAccess facade + migrate all four call sites (v2_scan, v2_sample, v2_schema two blocks, RemoteQueryEngine) to it; delete _bq_client_factory test injection point. Final PR has two reviewable commits.

Tech Stack: Python 3.11, FastAPI, DuckDB, google-cloud-bigquery, pytest. See docs/superpowers/specs/2026-04-29-issue-134-bq-access-unify-design.md for the full design.

---

File structure

Created (Phase 2):

• connectors/bigquery/access.py — BqProjects, BqAccessError, BqAccess, get_bq_access, translate_bq_error, default factories.
• tests/test_bq_access.py — unit tests for the new module.

Modified:

• app/api/v2_sample.py — Phase 1: billing_project + try/except; Phase 2: replace with BqAccess.
• app/api/v2_scan.py — Phase 1: try/except around BQ calls; Phase 2: replace with BqAccess.
• app/api/v2_schema.py — Phase 1: try/except around _fetch_bq_schema only; Phase 2: replace with BqAccess (both blocks, preserve _fetch_bq_table_options swallow-all).
• src/remote_query.py — Phase 2: lazy bq_access kwarg; drop _bq_client_factory + fallback chain + stale docstring.
• tests/test_v2_scan.py, tests/test_v2_scan_estimate.py, tests/test_v2_sample.py, tests/test_v2_schema.py — add structured-error tests; in Phase 2 update the _fetch_* lambda signatures.
• tests/test_remote_query.py — Phase 2: migrate _bq_client_factory= injection to bq_access=BqAccess(..., client_factory=...).
• tests/conftest.py — Phase 2: add bq_access fixture.
• CHANGELOG.md — Phase 1 (Fixed); Phase 2 (Changed BREAKING + Internal).

---

Phase 1 — Commit 1: Atomic bug fix (no BqAccess yet)

Goal: structured-body responses for cross-project Forbidden / BadRequest from BQ on all three v2 endpoints. Same body shape that Phase 2 will produce, so client-side parsers (CLI, UI) see one consistent contract throughout the rollout.

Body shape (used in all Phase 1 tasks):

# 502 case (Forbidden / upstream BadRequest)
{"error": "<kind>", "message": "<bq error text>", "details": {<contextual>}}
# 400 case (user-derived BadRequest in v2_scan only)
{"error": "bq_bad_request", "message": "<bq error text>", "details": {}}

<kind> for Forbidden = "cross_project_forbidden" if "serviceusage" in str(e).lower() else "bq_forbidden".

---

Task 1.1: v2_sample — add billing_project fallback + wrap _fetch_bq_sample

Files:

• Modify: app/api/v2_sample.py:21-48 (_fetch_bq_sample), :97-110 (sample endpoint).

• Test: tests/test_v2_sample.py (add new tests).

• Step 1: Read current state

Run: head -120 app/api/v2_sample.py to confirm baseline (today: reads only data_source.bigquery.project; _fetch_bq_sample has no try/except).

• Step 2: Write failing test for cross-project Forbidden → 502 + structured body

Add to tests/test_v2_sample.py:

def test_sample_returns_502_on_bq_forbidden_serviceusage(reload_db, monkeypatch):
    from app.api import v2_sample
    from google.api_core.exceptions import Forbidden

    def _raise_forbidden(*args, **kwargs):
        raise Forbidden("Permission denied: serviceusage.services.use on project foo")

    monkeypatch.setattr(v2_sample, "_fetch_bq_sample", _raise_forbidden)

    # ... seed registry with a bigquery table, then call the endpoint
    # via TestClient (match the pattern used by other tests in this file)
    # Assert response.status_code == 502
    # Assert response.json()["detail"]["error"] == "cross_project_forbidden"
    # Assert "billing_project" in response.json()["detail"]["details"]["hint"].lower()

NOTE: match the test-client construction style of existing tests in this file. If they use a TestClient fixture, reuse it; if they call helpers directly, follow that.

• Step 3: Write failing test for billing_project fallback being read

def test_sample_reads_billing_project_from_instance_yaml(reload_db, monkeypatch):
    """Regression guard for the original bug: the project passed to _fetch_bq_sample
    must come from billing_project when set, not from project."""
    captured = {}

    def _capture(project, dataset, table, n):
        captured["project"] = project
        return []

    monkeypatch.setattr("app.api.v2_sample._fetch_bq_sample", _capture)
    monkeypatch.setattr("app.instance_config.get_value", lambda *keys, **kw: {
        ("data_source", "bigquery", "project"): "data-proj",
        ("data_source", "bigquery", "billing_project"): "billing-proj",
    }.get(keys, kw.get("default", "")))

    # ... seed bigquery table, call /api/v2/sample/<id>, assert 200
    assert captured["project"] == "billing-proj"

• Step 4: Run tests — expect FAIL

Run: pytest tests/test_v2_sample.py::test_sample_returns_502_on_bq_forbidden_serviceusage tests/test_v2_sample.py::test_sample_reads_billing_project_from_instance_yaml -v Expected: both FAIL — sample endpoint today returns 500 on Forbidden and reads only project.

• Step 5: Implement billing_project fallback

Edit app/api/v2_sample.py:104 (in the sample endpoint):

project_id = (
    get_value("data_source", "bigquery", "billing_project", default="")
    or get_value("data_source", "bigquery", "project", default="")
    or ""
)

(Replace today's project_id = get_value("data_source", "bigquery", "project", default="") or "".)

• Step 6: Wrap _fetch_bq_sample with structured-error translation

Edit app/api/v2_sample.py:21-48. Replace the function body's interior with:

def _fetch_bq_sample(project: str, dataset: str, table: str, n: int) -> list[dict]:
    import duckdb
    from google.api_core import exceptions as gax
    from connectors.bigquery.auth import get_metadata_token
    from src.identifier_validation import validate_quoted_identifier
    from fastapi import HTTPException

    if not (validate_quoted_identifier(project, "BQ project")
            and validate_quoted_identifier(dataset, "BQ dataset")
            and validate_quoted_identifier(table, "BQ source_table")):
        raise ValueError("unsafe BQ identifier in registry — refusing to query")

    token = get_metadata_token()
    conn = duckdb.connect(":memory:")
    try:
        conn.execute("INSTALL bigquery FROM community; LOAD bigquery;")
        escaped = token.replace("'", "''")
        conn.execute(f"CREATE OR REPLACE SECRET bq_s (TYPE bigquery, ACCESS_TOKEN '{escaped}')")
        bq_sql = f"SELECT * FROM `{project}.{dataset}.{table}` LIMIT {int(n)}"
        try:
            df = conn.execute(
                "SELECT * FROM bigquery_query(?, ?)",
                [project, bq_sql],
            ).fetchdf()
        except gax.Forbidden as e:
            kind = "cross_project_forbidden" if "serviceusage" in str(e).lower() else "bq_forbidden"
            raise HTTPException(
                status_code=502,
                detail={
                    "error": kind,
                    "message": str(e),
                    "details": {
                        "billing_project": project,
                        "hint": (
                            "Set data_source.bigquery.billing_project in instance.yaml to a project "
                            "where the SA has serviceusage.services.use, or grant the SA that role "
                            "on the data project."
                        ) if kind == "cross_project_forbidden" else "",
                    },
                },
            )
        except gax.BadRequest as e:
            # /sample SQL is server-constructed (validated identifiers + LIMIT n);
            # a BadRequest here means registry corruption → upstream error, not user fault.
            raise HTTPException(
                status_code=502,
                detail={"error": "bq_upstream_error", "message": str(e), "details": {}},
            )
        except gax.GoogleAPICallError as e:
            raise HTTPException(
                status_code=502,
                detail={"error": "bq_upstream_error", "message": str(e), "details": {}},
            )
        return df.to_dict(orient="records")
    finally:
        conn.close()

• Step 7: Run the two new tests — expect PASS

Run: pytest tests/test_v2_sample.py::test_sample_returns_502_on_bq_forbidden_serviceusage tests/test_v2_sample.py::test_sample_reads_billing_project_from_instance_yaml -v Expected: both PASS.

• Step 8: Run full v2_sample test file — expect no regressions

Run: pytest tests/test_v2_sample.py -v Expected: all green (existing tests unaffected — the change is additive).

• Step 9: Commit (intermediate, will be squashed before PR)

cd <worktree-root>
git add app/api/v2_sample.py tests/test_v2_sample.py
git commit -m "fix(v2_sample): #134 add billing_project fallback + structured 502 on BQ Forbidden"

---

Task 1.2: v2_scan — wrap _bq_dry_run_bytes

Files:

• Modify: app/api/v2_scan.py:43-55 (_bq_dry_run_bytes).

• Test: tests/test_v2_scan_estimate.py.

• Step 1: Read current state

Run: sed -n '43,55p' app/api/v2_scan.py to confirm: today no try/except.

• Step 2: Write failing test for /scan/estimate 502 on Forbidden

Add to tests/test_v2_scan_estimate.py (match the file's existing test style):

def test_scan_estimate_returns_502_on_bq_forbidden_serviceusage(reload_db, monkeypatch):
    from app.api import v2_scan
    from google.api_core.exceptions import Forbidden

    def _raise_forbidden(project, sql):
        raise Forbidden("Permission denied: serviceusage.services.use on project foo")

    monkeypatch.setattr(v2_scan, "_bq_dry_run_bytes", _raise_forbidden)

    # ... seed bigquery table, call POST /api/v2/scan/estimate
    # Assert response.status_code == 502
    # Assert response.json()["detail"]["error"] == "cross_project_forbidden"
    # Assert "hint" in response.json()["detail"]["details"]

def test_scan_estimate_returns_400_on_bq_bad_request(reload_db, monkeypatch):
    from app.api import v2_scan
    from google.api_core.exceptions import BadRequest

    def _raise_bad_request(project, sql):
        raise BadRequest("Syntax error: unexpected token at line 1, column 5")

    monkeypatch.setattr(v2_scan, "_bq_dry_run_bytes", _raise_bad_request)

    # ... call POST /api/v2/scan/estimate
    # Assert response.status_code == 400
    # Assert response.json()["detail"]["error"] == "bq_bad_request"
    # Assert "Syntax error" in response.json()["detail"]["message"]

• Step 3: Run tests — expect FAIL

Run: pytest tests/test_v2_scan_estimate.py::test_scan_estimate_returns_502_on_bq_forbidden_serviceusage tests/test_v2_scan_estimate.py::test_scan_estimate_returns_400_on_bq_bad_request -v Expected: both FAIL with bare 500.

• Step 4: Wrap _bq_dry_run_bytes

Replace app/api/v2_scan.py:43-55 with:

def _bq_dry_run_bytes(project: str, sql: str) -> int:
    """Run a BQ dry-run via the google-cloud-bigquery client and return totalBytesProcessed.

    Errors translated to HTTPException with the structured-body shape used across
    all v2 endpoints. SQL here is user-derived (built from req.select/where/order_by),
    so BadRequest → 400.
    """
    from google.cloud import bigquery
    from google.api_core import exceptions as gax
    from google.api_core.client_options import ClientOptions
    from fastapi import HTTPException

    client = bigquery.Client(
        project=project,
        client_options=ClientOptions(quota_project_id=project),
    )
    try:
        job = client.query(
            sql,
            job_config=bigquery.QueryJobConfig(dry_run=True, use_query_cache=False),
        )
        return int(job.total_bytes_processed or 0)
    except gax.Forbidden as e:
        kind = "cross_project_forbidden" if "serviceusage" in str(e).lower() else "bq_forbidden"
        raise HTTPException(
            status_code=502,
            detail={
                "error": kind,
                "message": str(e),
                "details": {
                    "billing_project": project,
                    "hint": (
                        "Set data_source.bigquery.billing_project in instance.yaml to a project "
                        "where the SA has serviceusage.services.use, or grant the SA that role "
                        "on the data project."
                    ) if kind == "cross_project_forbidden" else "",
                },
            },
        )
    except gax.BadRequest as e:
        raise HTTPException(
            status_code=400,
            detail={"error": "bq_bad_request", "message": str(e), "details": {}},
        )
    except gax.GoogleAPICallError as e:
        raise HTTPException(
            status_code=502,
            detail={"error": "bq_upstream_error", "message": str(e), "details": {}},
        )

• Step 5: Run tests — expect PASS

Run: pytest tests/test_v2_scan_estimate.py -v Expected: all green.

• Step 6: Commit

git add app/api/v2_scan.py tests/test_v2_scan_estimate.py
git commit -m "fix(v2_scan): #134 structured 502/400 on BQ errors in dry-run path"

---

Task 1.3: v2_scan — wrap _run_bq_scan

Files:

• Modify: app/api/v2_scan.py:266-282 (_run_bq_scan).

• Test: tests/test_v2_scan.py.

• Step 1: Write failing tests (mirror Task 1.2)

Add to tests/test_v2_scan.py:

def test_scan_returns_502_on_bq_forbidden_serviceusage(reload_db, monkeypatch):
    from app.api import v2_scan
    from google.api_core.exceptions import Forbidden

    def _raise(project, sql):
        raise Forbidden("Permission denied: serviceusage.services.use on project foo")

    monkeypatch.setattr(v2_scan, "_run_bq_scan", _raise)

    # ... seed bigquery table, call POST /api/v2/scan
    # Assert response.status_code == 502
    # Assert response.json()["detail"]["error"] == "cross_project_forbidden"


def test_scan_returns_400_on_bq_bad_request(reload_db, monkeypatch):
    from app.api import v2_scan
    from google.api_core.exceptions import BadRequest

    def _raise(project, sql):
        raise BadRequest("Syntax error")

    monkeypatch.setattr(v2_scan, "_run_bq_scan", _raise)

    # ... call POST /api/v2/scan
    # Assert response.status_code == 400
    # Assert response.json()["detail"]["error"] == "bq_bad_request"

• Step 2: Run — expect FAIL

Run: pytest tests/test_v2_scan.py::test_scan_returns_502_on_bq_forbidden_serviceusage tests/test_v2_scan.py::test_scan_returns_400_on_bq_bad_request -v Expected: FAIL with bare 500.

• Step 3: Wrap _run_bq_scan

Replace app/api/v2_scan.py:266-282 with:

def _run_bq_scan(project: str, sql: str):
    """Run a BQ query via DuckDB BQ extension. Returns Arrow table.

    Errors translated to HTTPException with the structured-body shape used across
    all v2 endpoints. SQL here is user-derived → BadRequest → 400.
    """
    import duckdb
    from google.api_core import exceptions as gax
    from connectors.bigquery.auth import get_metadata_token
    from fastapi import HTTPException

    token = get_metadata_token()
    conn = duckdb.connect(":memory:")
    try:
        conn.execute("INSTALL bigquery FROM community; LOAD bigquery;")
        escaped = token.replace("'", "''")
        conn.execute(f"CREATE OR REPLACE SECRET bq_s (TYPE bigquery, ACCESS_TOKEN '{escaped}')")
        try:
            return conn.execute(
                "SELECT * FROM bigquery_query(?, ?)",
                [project, sql],
            ).arrow()
        except gax.Forbidden as e:
            kind = "cross_project_forbidden" if "serviceusage" in str(e).lower() else "bq_forbidden"
            raise HTTPException(
                status_code=502,
                detail={
                    "error": kind,
                    "message": str(e),
                    "details": {
                        "billing_project": project,
                        "hint": (
                            "Set data_source.bigquery.billing_project in instance.yaml to a project "
                            "where the SA has serviceusage.services.use, or grant the SA that role "
                            "on the data project."
                        ) if kind == "cross_project_forbidden" else "",
                    },
                },
            )
        except gax.BadRequest as e:
            raise HTTPException(
                status_code=400,
                detail={"error": "bq_bad_request", "message": str(e), "details": {}},
            )
        except gax.GoogleAPICallError as e:
            raise HTTPException(
                status_code=502,
                detail={"error": "bq_upstream_error", "message": str(e), "details": {}},
            )
    finally:
        conn.close()

• Step 4: Run — expect PASS

Run: pytest tests/test_v2_scan.py -v Expected: all green.

• Step 5: Commit

git add app/api/v2_scan.py tests/test_v2_scan.py
git commit -m "fix(v2_scan): #134 structured 502/400 on BQ errors in scan path"

---

Task 1.4: v2_schema — wrap _fetch_bq_schema (strict block only)

Files:

• Modify: app/api/v2_schema.py:36-73 (_fetch_bq_schema function — exact line range may vary; the strict block is the FIRST one in the file, NOT _fetch_bq_table_options).
• Test: tests/test_v2_schema.py.

IMPORTANT — preserve _fetch_bq_table_options exactly as-is in this task. That function (lines 90-129) wraps everything in try/except Exception → return {} and is the best-effort partition-info path. Phase 2 will migrate it to use BqAccess while preserving the swallow-all. Do not touch it in Phase 1.

• Step 1: Write failing test for /schema 502 on _fetch_bq_schema Forbidden

Add to tests/test_v2_schema.py:

def test_schema_returns_502_on_bq_forbidden_serviceusage(reload_db, monkeypatch):
    from app.api import v2_schema
    from google.api_core.exceptions import Forbidden

    def _raise(project, dataset, table):
        raise Forbidden("Permission denied: serviceusage.services.use on project foo")

    monkeypatch.setattr(v2_schema, "_fetch_bq_schema", _raise)

    # ... seed bigquery table, call GET /api/v2/schema/<id>
    # Assert response.status_code == 502
    # Assert response.json()["detail"]["error"] == "cross_project_forbidden"

• Step 2: Write regression-guard test for /schema 200 with empty partition info on _fetch_bq_table_options failure

def test_schema_returns_200_with_empty_partition_on_table_options_failure(reload_db, monkeypatch):
    """Regression guard: _fetch_bq_table_options is best-effort. /schema must keep
    returning successfully even if partition-info fetch fails."""
    from app.api import v2_schema
    from google.api_core.exceptions import Forbidden

    def _ok_schema(project, dataset, table):
        return [{"name": "event_date", "type": "DATE"}]

    def _fail_options(project, dataset, table):
        raise Forbidden("denied")

    monkeypatch.setattr(v2_schema, "_fetch_bq_schema", _ok_schema)
    monkeypatch.setattr(v2_schema, "_fetch_bq_table_options", _fail_options)

    # ... seed bigquery table, call GET /api/v2/schema/<id>
    # Assert response.status_code == 200
    # Assert response.json()["columns"] == [{"name": "event_date", "type": "DATE"}]
    # Assert "partition_by" not in response.json() OR response.json().get("partition_by") is None

(NOTE: today _fetch_bq_table_options already has the swallow-all try/except → return {}, so this test should pass even without changes — it's a regression guard for Phase 2.)

• Step 3: Run — expect first test FAIL, second PASS

Run: pytest tests/test_v2_schema.py::test_schema_returns_502_on_bq_forbidden_serviceusage tests/test_v2_schema.py::test_schema_returns_200_with_empty_partition_on_table_options_failure -v Expected: 502 test FAILS (today returns 500); regression-guard PASSES.

• Step 4: Wrap _fetch_bq_schema with the same try/except shape

Edit app/api/v2_schema.py:36-73 (the FIRST INSTALL/LOAD/SECRET block, in _fetch_bq_schema). Wrap the conn.execute("SELECT * FROM bigquery_query(?, ?, ?)", ...) line in:

from google.api_core import exceptions as gax
from fastapi import HTTPException

try:
    rows = conn.execute(
        "SELECT * FROM bigquery_query(?, ?, ?)",
        [project, bq_sql, table],
    ).fetchall()
except gax.Forbidden as e:
    kind = "cross_project_forbidden" if "serviceusage" in str(e).lower() else "bq_forbidden"
    raise HTTPException(
        status_code=502,
        detail={
            "error": kind,
            "message": str(e),
            "details": {
                "billing_project": project,
                "hint": (
                    "Set data_source.bigquery.billing_project in instance.yaml to a project "
                    "where the SA has serviceusage.services.use, or grant the SA that role "
                    "on the data project."
                ) if kind == "cross_project_forbidden" else "",
            },
        },
    )
except gax.GoogleAPICallError as e:
    raise HTTPException(
        status_code=502,
        detail={"error": "bq_upstream_error", "message": str(e), "details": {}},
    )

(/schema SQL hits INFORMATION_SCHEMA, fully server-constructed → BadRequest folded into upstream_error 502; no separate 400 case.)

Do not modify _fetch_bq_table_options.

• Step 5: Run — expect PASS

Run: pytest tests/test_v2_schema.py -v Expected: all green.

• Step 6: Commit

git add app/api/v2_schema.py tests/test_v2_schema.py
git commit -m "fix(v2_schema): #134 structured 502 on BQ errors in strict schema path"

---

Task 1.5: CHANGELOG entry for Phase 1; finalize commit 1

Files: CHANGELOG.md.

• Step 1: Read current state

Run: head -20 CHANGELOG.md to confirm there is an ## [Unreleased] section. If not, create one above the topmost release version.

• Step 2: Add Fixed entries under ## [Unreleased] / ### Fixed

### Fixed
- v2 `/sample` endpoint: BigQuery cross-project queries now respect `data_source.bigquery.billing_project` from `instance.yaml` (mirrors v2 `/scan` fix from `33a9964`). Closes #134 for `/sample`.
- v2 `/scan`, `/scan/estimate`, `/sample`, `/schema`: BigQuery upstream errors no longer return bare HTTP 500 with empty body. `Forbidden` from BQ now returns HTTP 502 with structured JSON body (`{"error": "cross_project_forbidden", "message": "...", "details": {"hint": "..."}}`); user-derived `BadRequest` on `/scan*` returns HTTP 400 with `kind=bq_bad_request`. Closes #134.
- v2 `/schema`: best-effort partition-info path (`_fetch_bq_table_options`) preserves its swallow-all behavior unchanged; `/schema` still returns 200 with empty partition info when BQ partition queries fail.

• Step 3: Stage CHANGELOG and verify Phase 1 working tree is clean

Run: git add CHANGELOG.md && git status Expected: only CHANGELOG.md staged; nothing else dirty.

• Step 4: Run full test suite to confirm no regressions before sealing Phase 1

Run: pytest tests/ -v -x --ignore=tests/test_telegram --ignore=tests/test_ws_gateway 2>&1 | tail -60 Expected: all green (or only pre-existing failures unrelated to BQ/v2).

If any new failures appear, fix them before committing — Phase 1 must ship green.

• Step 5: Squash Phase 1 commits (intermediate WIP) into a single Phase-1 commit

# Identify the first commit in this phase (it has "fix(v2_sample): #134 add billing_project")
git log --oneline | grep "#134" | tail -1
# Note the commit BEFORE that one — call it $BASE
# Soft-reset to $BASE, then make one clean commit
git reset --soft $BASE
git commit -m "fix(v2): #134 structured 502/400 on BQ errors across /scan, /scan/estimate, /sample, /schema

Wraps the BigQuery call sites in v2_scan, v2_sample, and v2_schema (strict
block only) with try/except for google.api_core exceptions, translating to
HTTPException with a structured body shape: {error, message, details}.

Fixes Pavel's report (#134) where these endpoints returned bare HTTP 500
with no body when the SA on <your-dev-instance> hit cross-project Forbidden
on serviceusage.services.use.

Also fixes /sample's missing billing_project fallback (the bug 33a9964
fixed for /scan never landed here).

Body shape matches what the upcoming BqAccess refactor (next commit) will
produce, so client-side parsers see one consistent contract throughout
the staged rollout. _fetch_bq_table_options preserved exactly as-is —
its swallow-all-and-return-empty contract is intentional and survives
into the refactor."

This is the final Phase 1 commit (the one that goes into the PR).

---

Phase 2 — Commit 2: BqAccess facade extraction + migration

Goal: extract the BqAccess facade and migrate all four call sites to use it. Delete the inline try/except blocks from Phase 1; route through translate_bq_error instead. Drop _bq_client_factory from RemoteQueryEngine.

---

Task 2.1: Create connectors/bigquery/access.py skeleton with BqProjects + BqAccessError

Files:

• Create: connectors/bigquery/access.py.

• Create: tests/test_bq_access.py.

• Step 1: Write failing tests for BqProjects and BqAccessError

Create tests/test_bq_access.py:

"""Tests for connectors/bigquery/access.py — the BqAccess facade."""
import pytest


class TestBqProjects:
    def test_bq_projects_is_frozen_dataclass(self):
        from connectors.bigquery.access import BqProjects
        p = BqProjects(billing="b", data="d")
        assert p.billing == "b"
        assert p.data == "d"
        with pytest.raises(Exception):  # FrozenInstanceError or AttributeError
            p.billing = "other"


class TestBqAccessError:
    def test_carries_kind_message_details(self):
        from connectors.bigquery.access import BqAccessError
        e = BqAccessError("my_kind", "boom", {"foo": "bar"})
        assert e.kind == "my_kind"
        assert e.message == "boom"
        assert e.details == {"foo": "bar"}
        assert str(e) == "boom"

    def test_default_details_is_empty_dict(self):
        from connectors.bigquery.access import BqAccessError
        e = BqAccessError("k", "m")
        assert e.details == {}

    def test_http_status_map_covers_all_kinds(self):
        from connectors.bigquery.access import BqAccessError
        expected = {
            "not_configured": 500,
            "bq_lib_missing": 500,
            "auth_failed": 502,
            "cross_project_forbidden": 502,
            "bq_forbidden": 502,
            "bq_bad_request": 400,
            "bq_upstream_error": 502,
        }
        assert BqAccessError.HTTP_STATUS == expected

• Step 2: Run — expect FAIL (module doesn't exist)

Run: pytest tests/test_bq_access.py -v Expected: collection error / ModuleNotFoundError on connectors.bigquery.access.

• Step 3: Create the module skeleton

Create connectors/bigquery/access.py:

"""Single entry point for BigQuery access — config resolution, client construction,
DuckDB-extension session, and Google-API error translation.

See docs/superpowers/specs/2026-04-29-issue-134-bq-access-unify-design.md for the
full design rationale.
"""
from __future__ import annotations

import functools
import logging
from contextlib import contextmanager
from dataclasses import dataclass
from typing import Callable, Iterator, Literal

logger = logging.getLogger(__name__)


@dataclass(frozen=True)
class BqProjects:
    """Pair of GCP project IDs used by Agnes.

    `billing` is the project the BQ client bills jobs to (also used as quota_project_id).
    `data` is the default data project for FROM-clause construction. Today equal to
    instance.yaml `data_source.bigquery.project`; locked to a single project per instance
    until table_registry grows a per-table source_project column. See spec "Non-goals".
    """
    billing: str
    data: str


class BqAccessError(Exception):
    """Typed error for BQ access failures.

    `kind` is one of HTTP_STATUS keys; endpoint translation maps it to status codes.
    """

    HTTP_STATUS = {
        "not_configured":          500,  # admin/config bug — page on-call
        "bq_lib_missing":          500,  # deployment bug
        "auth_failed":             502,  # GCP metadata server unreachable
        "cross_project_forbidden": 502,  # SA lacks serviceusage.services.use on billing project
        "bq_forbidden":            502,  # other Forbidden from BQ
        "bq_bad_request":          400,  # 400 from BQ when caller flagged it as client-derived
        "bq_upstream_error":       502,  # all other upstream BQ failures
    }

    def __init__(self, kind: str, message: str, details: dict | None = None):
        self.kind = kind
        self.message = message
        self.details = details or {}
        super().__init__(message)

• Step 4: Run — expect PASS

Run: pytest tests/test_bq_access.py::TestBqProjects tests/test_bq_access.py::TestBqAccessError -v Expected: all green.

• Step 5: Commit

git add connectors/bigquery/access.py tests/test_bq_access.py
git commit -m "feat(bq_access): skeleton — BqProjects + BqAccessError"

---

Task 2.2: Add translate_bq_error to connectors/bigquery/access.py

Files:

• Modify: connectors/bigquery/access.py.

• Modify: tests/test_bq_access.py.

• Step 1: Add failing tests for translate_bq_error

Append to tests/test_bq_access.py:

class TestTranslateBqError:
    def setup_method(self):
        from connectors.bigquery.access import BqProjects
        self.projects = BqProjects(billing="bill", data="data")

    def test_passes_through_BqAccessError(self):
        """CRITICAL: bq.client() / bq.duckdb_session() raise BqAccessError directly
        for bq_lib_missing / auth_failed. translate_bq_error must pass them through,
        not reclassify as 'unknown' and re-raise."""
        from connectors.bigquery.access import BqAccessError, translate_bq_error
        original = BqAccessError("bq_lib_missing", "no google lib")
        result = translate_bq_error(original, self.projects, bad_request_status="client_error")
        assert result is original

    def test_forbidden_serviceusage_to_cross_project(self):
        from google.api_core.exceptions import Forbidden
        from connectors.bigquery.access import translate_bq_error
        e = Forbidden("Permission denied: serviceusage.services.use on project foo")
        result = translate_bq_error(e, self.projects, bad_request_status="client_error")
        assert result.kind == "cross_project_forbidden"
        assert "billing_project" in result.details
        assert "hint" in result.details

    def test_forbidden_no_serviceusage_to_bq_forbidden(self):
        from google.api_core.exceptions import Forbidden
        from connectors.bigquery.access import translate_bq_error
        e = Forbidden("Permission denied on table-level ACL")
        result = translate_bq_error(e, self.projects, bad_request_status="client_error")
        assert result.kind == "bq_forbidden"

    def test_forbidden_diff_projects_no_serviceusage_still_bq_forbidden(self):
        """billing != data is the NORMAL cross-project setup, not a signal of failure.
        Heuristic must rely on 'serviceusage' substring only."""
        from google.api_core.exceptions import Forbidden
        from connectors.bigquery.access import translate_bq_error, BqProjects
        e = Forbidden("Permission denied on table-level ACL")
        result = translate_bq_error(e, BqProjects(billing="b", data="d"),
                                     bad_request_status="client_error")
        assert result.kind == "bq_forbidden"  # NOT cross_project_forbidden

    def test_bad_request_client_error_to_bq_bad_request_400(self):
        from google.api_core.exceptions import BadRequest
        from connectors.bigquery.access import translate_bq_error, BqAccessError
        e = BadRequest("Syntax error at line 1")
        result = translate_bq_error(e, self.projects, bad_request_status="client_error")
        assert result.kind == "bq_bad_request"
        assert BqAccessError.HTTP_STATUS[result.kind] == 400

    def test_bad_request_upstream_error_to_bq_upstream_error_502(self):
        from google.api_core.exceptions import BadRequest
        from connectors.bigquery.access import translate_bq_error, BqAccessError
        e = BadRequest("malformed identifier")
        result = translate_bq_error(e, self.projects, bad_request_status="upstream_error")
        assert result.kind == "bq_upstream_error"
        assert BqAccessError.HTTP_STATUS[result.kind] == 502

    def test_other_google_api_error_to_bq_upstream_error(self):
        from google.api_core.exceptions import InternalServerError
        from connectors.bigquery.access import translate_bq_error
        e = InternalServerError("BQ borked")
        result = translate_bq_error(e, self.projects, bad_request_status="client_error")
        assert result.kind == "bq_upstream_error"

    def test_unknown_exception_reraises(self):
        from connectors.bigquery.access import translate_bq_error
        with pytest.raises(RuntimeError, match="oops"):
            translate_bq_error(RuntimeError("oops"), self.projects,
                               bad_request_status="client_error")

• Step 2: Run — expect FAIL (function doesn't exist)

Run: pytest tests/test_bq_access.py::TestTranslateBqError -v Expected: ImportError on translate_bq_error.

• Step 3: Implement translate_bq_error

Append to connectors/bigquery/access.py:

def translate_bq_error(
    e: Exception,
    projects: BqProjects,
    *,
    bad_request_status: Literal["client_error", "upstream_error"],
) -> BqAccessError:
    """Convert Google API exceptions into a typed BqAccessError.

    Mapping (FIRST match wins):
      1. BqAccessError                    -> pass through unchanged (CRITICAL: bq.client()
                                             and bq.duckdb_session() can raise BqAccessError
                                             directly for bq_lib_missing / auth_failed; those
                                             must round-trip without reclassification)
      2. Forbidden + 'serviceusage' in str(e).lower()
                                          -> cross_project_forbidden (with hint)
      3. Forbidden                        -> bq_forbidden
      4. BadRequest, bad_request_status='client_error'
                                          -> bq_bad_request (HTTP 400)
      5. BadRequest, bad_request_status='upstream_error'
                                          -> bq_upstream_error (HTTP 502)
      6. GoogleAPICallError (other)       -> bq_upstream_error
      7. Anything else                    -> RE-RAISED unchanged (don't swallow programmer errors)
    """
    if isinstance(e, BqAccessError):
        return e

    try:
        from google.api_core import exceptions as gax  # type: ignore
    except ImportError:
        # No google lib installed → can't classify Google errors. Re-raise.
        raise e

    msg = str(e)

    if isinstance(e, gax.Forbidden):
        if "serviceusage" in msg.lower():
            return BqAccessError(
                "cross_project_forbidden",
                msg,
                details={
                    "billing_project": projects.billing,
                    "data_project": projects.data,
                    "hint": (
                        "Set data_source.bigquery.billing_project in instance.yaml to a project "
                        "where the SA has serviceusage.services.use, or grant the SA that role "
                        "on the data project."
                    ),
                },
            )
        return BqAccessError(
            "bq_forbidden",
            msg,
            details={"billing_project": projects.billing, "data_project": projects.data},
        )

    if isinstance(e, gax.BadRequest):
        if bad_request_status == "client_error":
            return BqAccessError("bq_bad_request", msg)
        return BqAccessError("bq_upstream_error", msg)

    if isinstance(e, gax.GoogleAPICallError):
        return BqAccessError("bq_upstream_error", msg)

    # Don't swallow programmer errors / unknown exceptions
    raise e

• Step 4: Run — expect PASS

Run: pytest tests/test_bq_access.py::TestTranslateBqError -v Expected: all green.

• Step 5: Commit

git add connectors/bigquery/access.py tests/test_bq_access.py
git commit -m "feat(bq_access): translate_bq_error — typed BqAccessError mapping"

---

Task 2.3: Add _default_client_factory and _default_duckdb_session_factory

Files:

• Modify: connectors/bigquery/access.py.

• Modify: tests/test_bq_access.py.

• Step 1: Write failing tests

Append to tests/test_bq_access.py:

class TestDefaultClientFactory:
    def test_constructs_client_with_billing_project_as_quota(self, monkeypatch):
        """quota_project_id must be projects.billing, NOT projects.data."""
        from connectors.bigquery.access import _default_client_factory, BqProjects

        captured = {}

        class FakeClientOptions:
            def __init__(self, **kwargs):
                captured["client_options_kwargs"] = kwargs

        class FakeClient:
            def __init__(self, project, client_options):
                captured["project"] = project
                captured["client_options"] = client_options

        import google.cloud.bigquery as bq_mod
        import google.api_core.client_options as co_mod
        monkeypatch.setattr(bq_mod, "Client", FakeClient)
        monkeypatch.setattr(co_mod, "ClientOptions", FakeClientOptions)

        _default_client_factory(BqProjects(billing="bill", data="data"))

        assert captured["project"] == "bill"
        assert captured["client_options_kwargs"]["quota_project_id"] == "bill"

    def test_raises_bq_lib_missing_on_importerror(self, monkeypatch):
        """If google-cloud-bigquery is not installed, raise BqAccessError, not ImportError."""
        from connectors.bigquery.access import _default_client_factory, BqProjects, BqAccessError
        import builtins
        real_import = builtins.__import__

        def fake_import(name, *args, **kwargs):
            if name == "google.cloud" or name.startswith("google.cloud.bigquery"):
                raise ImportError("no google-cloud-bigquery")
            return real_import(name, *args, **kwargs)

        monkeypatch.setattr(builtins, "__import__", fake_import)
        with pytest.raises(BqAccessError) as exc_info:
            _default_client_factory(BqProjects(billing="b", data="d"))
        assert exc_info.value.kind == "bq_lib_missing"


class TestDefaultDuckdbSessionFactory:
    def test_yields_duckdb_conn_with_secret_then_closes(self, monkeypatch):
        from connectors.bigquery.access import _default_duckdb_session_factory, BqProjects

        executed_sql = []

        class FakeConn:
            def __init__(self):
                self.closed = False
            def execute(self, sql, params=None):
                executed_sql.append((sql, params))
                return self
            def close(self):
                self.closed = True

        fake_conn = FakeConn()
        monkeypatch.setattr("duckdb.connect", lambda _: fake_conn)
        monkeypatch.setattr("connectors.bigquery.auth.get_metadata_token", lambda: "tok123")

        with _default_duckdb_session_factory(BqProjects(billing="b", data="d")) as conn:
            assert conn is fake_conn
        assert fake_conn.closed is True

        # Verify INSTALL/LOAD/SECRET sequence ran
        assert any("INSTALL bigquery" in sql for sql, _ in executed_sql)
        assert any("LOAD bigquery" in sql for sql, _ in executed_sql)
        assert any("CREATE OR REPLACE SECRET" in sql and "tok123" in sql for sql, _ in executed_sql)

    def test_closes_on_exception_inside_with_block(self, monkeypatch):
        from connectors.bigquery.access import _default_duckdb_session_factory, BqProjects

        class FakeConn:
            closed = False
            def execute(self, *a, **kw): return self
            def close(self): self.closed = True

        fake_conn = FakeConn()
        monkeypatch.setattr("duckdb.connect", lambda _: fake_conn)
        monkeypatch.setattr("connectors.bigquery.auth.get_metadata_token", lambda: "t")

        with pytest.raises(RuntimeError, match="boom"):
            with _default_duckdb_session_factory(BqProjects(billing="b", data="d")) as conn:
                raise RuntimeError("boom")
        assert fake_conn.closed is True

    def test_translates_metadata_auth_error_to_auth_failed(self, monkeypatch):
        from connectors.bigquery.access import _default_duckdb_session_factory, BqProjects, BqAccessError
        from connectors.bigquery.auth import BQMetadataAuthError

        def fail():
            raise BQMetadataAuthError("metadata server unreachable")

        monkeypatch.setattr("connectors.bigquery.auth.get_metadata_token", fail)

        with pytest.raises(BqAccessError) as exc_info:
            with _default_duckdb_session_factory(BqProjects(billing="b", data="d")):
                pass
        assert exc_info.value.kind == "auth_failed"

• Step 2: Run — expect FAIL

Run: pytest tests/test_bq_access.py::TestDefaultClientFactory tests/test_bq_access.py::TestDefaultDuckdbSessionFactory -v Expected: FAIL — factories don't exist.

• Step 3: Implement the two default factories

Append to connectors/bigquery/access.py:

def _default_client_factory(projects: BqProjects):
    """Real BigQuery client construction. Raises BqAccessError on import / config issues."""
    try:
        from google.cloud import bigquery  # type: ignore
        from google.api_core.client_options import ClientOptions  # type: ignore
    except ImportError as e:
        raise BqAccessError(
            "bq_lib_missing",
            "google-cloud-bigquery is not installed",
            details={"original": str(e)},
        )

    return bigquery.Client(
        project=projects.billing,
        client_options=ClientOptions(quota_project_id=projects.billing),
    )


@contextmanager
def _default_duckdb_session_factory(projects: BqProjects):
    """Yield an in-memory DuckDB conn with bigquery extension loaded + SECRET set
    from get_metadata_token(). Auto-cleanup. Translates auth/install failures
    to BqAccessError(kind='auth_failed' or 'bq_lib_missing').

    Note: `projects.billing` is not used by this factory directly — bigquery_query()
    callers pass it themselves as the first positional arg to identify the billing
    project. The factory keeps the parameter for symmetry with _default_client_factory.
    """
    import duckdb  # type: ignore
    from connectors.bigquery.auth import get_metadata_token, BQMetadataAuthError

    try:
        token = get_metadata_token()
    except BQMetadataAuthError as e:
        raise BqAccessError(
            "auth_failed",
            f"could not fetch GCP metadata token: {e}",
            details={"original": str(e)},
        )

    conn = duckdb.connect(":memory:")
    try:
        try:
            conn.execute("INSTALL bigquery FROM community; LOAD bigquery;")
            escaped = token.replace("'", "''")
            conn.execute(
                f"CREATE OR REPLACE SECRET bq_s (TYPE bigquery, ACCESS_TOKEN '{escaped}')"
            )
        except Exception as e:
            raise BqAccessError(
                "bq_lib_missing",
                f"failed to install/load BigQuery DuckDB extension: {e}",
                details={"original": str(e)},
            )
        yield conn
    finally:
        conn.close()

• Step 4: Run — expect PASS

Run: pytest tests/test_bq_access.py::TestDefaultClientFactory tests/test_bq_access.py::TestDefaultDuckdbSessionFactory -v Expected: all green.

• Step 5: Commit

git add connectors/bigquery/access.py tests/test_bq_access.py
git commit -m "feat(bq_access): default factories for client + duckdb_session"

---

Task 2.4: Add BqAccess class

Files:

• Modify: connectors/bigquery/access.py.

• Modify: tests/test_bq_access.py.

• Step 1: Write failing tests

Append to tests/test_bq_access.py:

class TestBqAccess:
    def test_uses_default_factories_when_none_passed(self, monkeypatch):
        from connectors.bigquery.access import BqAccess, BqProjects

        captured = []
        monkeypatch.setattr(
            "connectors.bigquery.access._default_client_factory",
            lambda projects: captured.append(("client", projects)) or "FAKE_CLIENT",
        )
        bq = BqAccess(BqProjects(billing="b", data="d"))
        assert bq.client() == "FAKE_CLIENT"
        assert captured == [("client", BqProjects(billing="b", data="d"))]

    def test_injected_client_factory_overrides_default(self):
        from connectors.bigquery.access import BqAccess, BqProjects
        bq = BqAccess(
            BqProjects(billing="b", data="d"),
            client_factory=lambda projects: "MOCK_CLIENT",
        )
        assert bq.client() == "MOCK_CLIENT"

    def test_injected_duckdb_session_factory_overrides_default(self):
        from connectors.bigquery.access import BqAccess, BqProjects
        from contextlib import contextmanager

        @contextmanager
        def fake_session(projects):
            yield "FAKE_CONN"

        bq = BqAccess(
            BqProjects(billing="b", data="d"),
            duckdb_session_factory=fake_session,
        )
        with bq.duckdb_session() as conn:
            assert conn == "FAKE_CONN"

    def test_projects_property(self):
        from connectors.bigquery.access import BqAccess, BqProjects
        p = BqProjects(billing="b", data="d")
        bq = BqAccess(p)
        assert bq.projects is p

• Step 2: Run — expect FAIL (BqAccess class doesn't exist)

Run: pytest tests/test_bq_access.py::TestBqAccess -v Expected: AttributeError on BqAccess.

• Step 3: Implement BqAccess

Append to connectors/bigquery/access.py:

class BqAccess:
    """Single entry point for BigQuery access. Stateless after construction.

    Factories are injectable for tests:
        bq = BqAccess(
            BqProjects(billing="test-billing", data="test-data"),
            client_factory=lambda projects: mock_client,
        )
    """

    def __init__(
        self,
        projects: BqProjects,
        *,
        client_factory: Callable[[BqProjects], object] | None = None,
        duckdb_session_factory: Callable[[BqProjects], object] | None = None,
    ):
        self._projects = projects
        self._client_factory = client_factory or _default_client_factory
        self._duckdb_session_factory = duckdb_session_factory or _default_duckdb_session_factory

    @property
    def projects(self) -> BqProjects:
        return self._projects

    def client(self):
        """Construct (or retrieve from injected factory) a BigQuery client."""
        return self._client_factory(self._projects)

    @contextmanager
    def duckdb_session(self) -> Iterator[object]:
        """Yield in-memory DuckDB conn with bigquery extension loaded + SECRET set."""
        with self._duckdb_session_factory(self._projects) as conn:
            yield conn

• Step 4: Run — expect PASS

Run: pytest tests/test_bq_access.py::TestBqAccess -v Expected: all green.

• Step 5: Commit

git add connectors/bigquery/access.py tests/test_bq_access.py
git commit -m "feat(bq_access): BqAccess facade with injectable factories"

---

Task 2.5: Add get_bq_access (module-level cached entry point)

Files:

• Modify: connectors/bigquery/access.py.

• Modify: tests/test_bq_access.py.

• Step 1: Write failing tests

Append to tests/test_bq_access.py:

class TestGetBqAccess:
    def setup_method(self):
        # Clear the cache between tests
        from connectors.bigquery.access import get_bq_access
        get_bq_access.cache_clear()

    def test_env_var_wins(self, monkeypatch):
        from connectors.bigquery.access import get_bq_access
        monkeypatch.setenv("BIGQUERY_PROJECT", "env-proj")
        bq = get_bq_access()
        assert bq.projects.billing == "env-proj"
        assert bq.projects.data == "env-proj"

    def test_billing_project_from_yaml_when_no_env(self, monkeypatch):
        from connectors.bigquery.access import get_bq_access
        monkeypatch.delenv("BIGQUERY_PROJECT", raising=False)

        def fake_get_value(*keys, default=""):
            return {
                ("data_source", "bigquery", "billing_project"): "yaml-bill",
                ("data_source", "bigquery", "project"): "yaml-data",
            }.get(keys, default)

        monkeypatch.setattr("app.instance_config.get_value", fake_get_value)
        bq = get_bq_access()
        assert bq.projects.billing == "yaml-bill"
        assert bq.projects.data == "yaml-data"

    def test_billing_falls_back_to_project_when_no_billing(self, monkeypatch):
        from connectors.bigquery.access import get_bq_access
        monkeypatch.delenv("BIGQUERY_PROJECT", raising=False)

        def fake_get_value(*keys, default=""):
            return {
                ("data_source", "bigquery", "project"): "yaml-data",
            }.get(keys, default)

        monkeypatch.setattr("app.instance_config.get_value", fake_get_value)
        bq = get_bq_access()
        assert bq.projects.billing == "yaml-data"
        assert bq.projects.data == "yaml-data"

    def test_raises_not_configured_when_neither_set(self, monkeypatch):
        from connectors.bigquery.access import get_bq_access, BqAccessError
        monkeypatch.delenv("BIGQUERY_PROJECT", raising=False)
        monkeypatch.setattr("app.instance_config.get_value", lambda *k, default="": default)

        with pytest.raises(BqAccessError) as exc_info:
            get_bq_access()
        assert exc_info.value.kind == "not_configured"
        assert "billing_project" in exc_info.value.details["hint"].lower() or \
               "project" in exc_info.value.details["hint"].lower()

    def test_is_cached(self, monkeypatch):
        from connectors.bigquery.access import get_bq_access
        monkeypatch.setenv("BIGQUERY_PROJECT", "p")
        a = get_bq_access()
        b = get_bq_access()
        assert a is b

    def test_does_not_cache_exceptions(self, monkeypatch):
        """functools.cache does not cache exceptions — config can be fixed and retried."""
        from connectors.bigquery.access import get_bq_access, BqAccessError
        monkeypatch.delenv("BIGQUERY_PROJECT", raising=False)
        monkeypatch.setattr("app.instance_config.get_value", lambda *k, default="": default)

        with pytest.raises(BqAccessError):
            get_bq_access()

        # Now "fix" the config
        monkeypatch.setenv("BIGQUERY_PROJECT", "p")
        bq = get_bq_access()
        assert bq.projects.billing == "p"

• Step 2: Run — expect FAIL (function doesn't exist)

Run: pytest tests/test_bq_access.py::TestGetBqAccess -v Expected: ImportError on get_bq_access.

• Step 3: Implement get_bq_access

Append to connectors/bigquery/access.py:

@functools.cache
def get_bq_access() -> BqAccess:
    """Module-level FastAPI Depends target. Resolves projects from config and returns
    a BqAccess instance with default factories.

    Resolution order:
      1. BIGQUERY_PROJECT env var → both billing + data (legacy override)
      2. instance.yaml data_source.bigquery.billing_project → billing
      3. instance.yaml data_source.bigquery.project → data, and billing if (2) is unset

    Process-cached. Hot-reload of instance.yaml is out of scope; restart the container
    on config change. functools.cache does NOT cache exceptions, so a failed call is
    retried on the next invocation.

    Tests inject via `app.dependency_overrides[get_bq_access] = lambda: bq` for
    endpoints, or construct `BqAccess(...)` directly for non-endpoint code.

    Module-level (not a classmethod) to avoid the @classmethod + @functools.cache
    stacking footgun and to give FastAPI's dependency introspection a clean signature.
    """
    import os

    env_project = os.environ.get("BIGQUERY_PROJECT", "").strip()
    if env_project:
        return BqAccess(BqProjects(billing=env_project, data=env_project))

    from app.instance_config import get_value
    billing = (get_value("data_source", "bigquery", "billing_project", default="") or "").strip()
    data = (get_value("data_source", "bigquery", "project", default="") or "").strip()

    if not data:
        raise BqAccessError(
            "not_configured",
            "BigQuery project not configured",
            details={
                "hint": (
                    "Set data_source.bigquery.project in instance.yaml "
                    "(and optionally data_source.bigquery.billing_project for cross-project "
                    "deployments). BIGQUERY_PROJECT env var also accepted as legacy override."
                ),
            },
        )

    if not billing:
        billing = data

    return BqAccess(BqProjects(billing=billing, data=data))

• Step 4: Run — expect PASS

Run: pytest tests/test_bq_access.py::TestGetBqAccess -v Expected: all green.

• Step 5: Run the WHOLE bq_access test file as a sanity check

Run: pytest tests/test_bq_access.py -v Expected: every test green.

• Step 6: Commit

git add connectors/bigquery/access.py tests/test_bq_access.py
git commit -m "feat(bq_access): get_bq_access — cached module-level entry point"

---

Task 2.6: Add bq_access fixture to tests/conftest.py

Files:

• Modify: tests/conftest.py.

• Step 1: Add fixture without test (it's infrastructure for later tasks)

Append to tests/conftest.py:

import contextlib as _contextlib


@pytest.fixture
def bq_access():
    """Build a BqAccess with pluggable factories and override the FastAPI Depends.

    Usage:
        def test_x(bq_access):
            mock_client = MagicMock()
            bq = bq_access(client=mock_client)
            # endpoint test code

    Override is auto-cleared on fixture teardown.

    NOTE: `contextlib.nullcontext(duckdb_conn)` does NOT close the conn on exit.
    The production path closes via _default_duckdb_session_factory. Tests that
    care about close behavior should use that factory directly (see
    tests/test_bq_access.py::TestDefaultDuckdbSessionFactory).
    """
    from connectors.bigquery.access import BqAccess, BqProjects, get_bq_access
    from app.main import app

    def _build(*, client=None, duckdb_conn=None,
               billing="test-billing", data="test-data"):
        bq = BqAccess(
            BqProjects(billing=billing, data=data),
            client_factory=(lambda projects: client) if client is not None else None,
            duckdb_session_factory=(
                lambda projects: _contextlib.nullcontext(duckdb_conn)
            ) if duckdb_conn is not None else None,
        )
        app.dependency_overrides[get_bq_access] = lambda: bq
        return bq

    yield _build
    from app.main import app as _app
    _app.dependency_overrides.pop(get_bq_access, None)

(Note: import path for app may need adjustment depending on how tests import it; match the existing pattern.)

• Step 2: Verify fixture loads (no test, just collection)

Run: pytest --collect-only tests/conftest.py 2>&1 | tail -5 Expected: no collection errors.

• Step 3: Commit

git add tests/conftest.py
git commit -m "test: add bq_access fixture for FastAPI dep override"

---

Task 2.7: Migrate app/api/v2_scan.py to BqAccess

Files:

• Modify: app/api/v2_scan.py.

• Modify: tests/test_v2_scan.py, tests/test_v2_scan_estimate.py (update lambda signatures + use new fixture).

• Step 1: Read current state to confirm Phase-1 try/except blocks are still in place

Run: sed -n '43,90p' app/api/v2_scan.py Expected: _bq_dry_run_bytes has the inline try/except added in Task 1.2.

• Step 2: Update _bq_dry_run_bytes signature and body

Replace app/api/v2_scan.py:43-95 (the entire _bq_dry_run_bytes from Task 1.2):

def _bq_dry_run_bytes(bq, sql: str) -> int:
    """Run a BQ dry-run via the google-cloud-bigquery client and return totalBytesProcessed."""
    from google.cloud import bigquery
    from connectors.bigquery.access import translate_bq_error

    client = bq.client()  # raises BqAccessError(bq_lib_missing/auth_failed) — propagates as-is
    try:
        job = client.query(
            sql, job_config=bigquery.QueryJobConfig(dry_run=True, use_query_cache=False),
        )
        return int(job.total_bytes_processed or 0)
    except Exception as e:
        raise translate_bq_error(e, bq.projects, bad_request_status="client_error")

• Step 3: Update _run_bq_scan similarly

Replace app/api/v2_scan.py:266-... (the entire _run_bq_scan from Task 1.3):

def _run_bq_scan(bq, sql: str):
    """Run a BQ query via DuckDB BQ extension. Returns Arrow table."""
    from connectors.bigquery.access import translate_bq_error

    with bq.duckdb_session() as conn:
        try:
            return conn.execute(
                "SELECT * FROM bigquery_query(?, ?)",
                [bq.projects.billing, sql],
            ).arrow()
        except Exception as e:
            raise translate_bq_error(e, bq.projects, bad_request_status="client_error")

• Step 4: Update estimate function signature and call site

In app/api/v2_scan.py:135-... (def estimate(...)):

• Remove project_id: str, billing_project: str | None = None parameters.

• Add bq parameter.

• Inside: replace _bq_dry_run_bytes(billing_project or project_id, bq_sql) with _bq_dry_run_bytes(bq, bq_sql).

• Replace _build_bq_sql(row, project_id, req) with _build_bq_sql(row, bq.projects.data, req).

• Replace _resolve_schema(conn, user, req.table_id, project_id) with _resolve_schema(conn, user, req.table_id, bq.projects.data).

• Step 5: Update run_scan function similarly

Same pattern: drop project_id, billing_project params; add bq. Replace internal references.

• Step 6: Update scan_endpoint and scan_estimate_endpoint

Replace lines app/api/v2_scan.py:217-... (scan_estimate_endpoint) and :378-... (scan_endpoint):

@router.post("/scan/estimate")
async def scan_estimate_endpoint(
    raw: dict,
    user: dict = Depends(get_current_user),
    conn: duckdb.DuckDBPyConnection = Depends(_get_db),
    bq: BqAccess = Depends(get_bq_access),
):
    try:
        return estimate(conn, user, raw, bq=bq)
    except WhereValidationError as e:
        raise HTTPException(status_code=400, detail={"error": "validator_rejected", "kind": e.kind, "details": e.detail or {}})
    except PermissionError:
        raise HTTPException(status_code=403, detail="not authorized for this table")
    except FileNotFoundError as e:
        raise HTTPException(status_code=404, detail=f"table {e!s} not found")
    except ValueError as e:
        raise HTTPException(status_code=400, detail=str(e))
    except BqAccessError as e:
        raise HTTPException(
            status_code=BqAccessError.HTTP_STATUS.get(e.kind, 500),
            detail={"error": e.kind, "message": e.message, "details": e.details},
        )


@router.post("/scan")
async def scan_endpoint(
    raw: dict,
    user: dict = Depends(get_current_user),
    conn: duckdb.DuckDBPyConnection = Depends(_get_db),
    bq: BqAccess = Depends(get_bq_access),
):
    quota = _build_quota_tracker()
    try:
        ipc = run_scan(conn, user, raw, bq=bq, quota=quota)
        return Response(content=ipc, media_type=CONTENT_TYPE)
    except WhereValidationError as e:
        raise HTTPException(status_code=400, detail={"error": "validator_rejected", "kind": e.kind, "details": e.detail or {}})
    except QuotaExceededError as e:
        raise HTTPException(status_code=429, detail={"error": "quota_exceeded", **(e.detail or {})})
    except PermissionError:
        raise HTTPException(status_code=403, detail="not authorized for this table")
    except FileNotFoundError as e:
        raise HTTPException(status_code=404, detail=f"table {e!s} not found")
    except ValueError as e:
        raise HTTPException(status_code=400, detail=str(e))
    except BqAccessError as e:
        raise HTTPException(
            status_code=BqAccessError.HTTP_STATUS.get(e.kind, 500),
            detail={"error": e.kind, "message": e.message, "details": e.details},
        )

Add at top of app/api/v2_scan.py:

from connectors.bigquery.access import BqAccess, BqAccessError, get_bq_access

• Step 7: Update existing tests in tests/test_v2_scan.py and tests/test_v2_scan_estimate.py for the new signature

Tests that previously did monkeypatch.setattr(v2_scan, "_bq_dry_run_bytes", lambda project, sql: ...) now need lambda bq, sql: .... Same for _run_bq_scan. Trace through all such monkeypatches and update.

The Forbidden / BadRequest tests from Task 1.2 / 1.3 also need updating: instead of patching _bq_dry_run_bytes to raise, you can either (a) keep patching that function (now with new signature), or (b) use the new bq_access fixture to inject a client that raises. Pattern (b) is cleaner long-term:

def test_scan_estimate_returns_502_on_bq_forbidden_serviceusage(reload_db, bq_access):
    from google.api_core.exceptions import Forbidden
    from unittest.mock import MagicMock

    mock_client = MagicMock()
    mock_client.query.side_effect = Forbidden("Permission denied: serviceusage.services.use on project foo")
    bq_access(client=mock_client)

    # ... call POST /api/v2/scan/estimate
    # Assert response.status_code == 502
    # Assert response.json()["detail"]["error"] == "cross_project_forbidden"

• Step 8: Run v2_scan tests — expect PASS

Run: pytest tests/test_v2_scan.py tests/test_v2_scan_estimate.py -v Expected: all green.

• Step 9: Commit

git add app/api/v2_scan.py tests/test_v2_scan.py tests/test_v2_scan_estimate.py
git commit -m "refactor(v2_scan): #134 migrate to BqAccess facade"

---

Task 2.8: Migrate app/api/v2_sample.py to BqAccess

Files:

• Modify: app/api/v2_sample.py.

• Modify: tests/test_v2_sample.py.

• Step 1: Update _fetch_bq_sample signature and body

Replace the entire function (added in Task 1.1):

def _fetch_bq_sample(bq, dataset: str, table: str, n: int) -> list[dict]:
    from connectors.bigquery.access import translate_bq_error
    from src.identifier_validation import validate_quoted_identifier

    if not (validate_quoted_identifier(bq.projects.data, "BQ project")
            and validate_quoted_identifier(dataset, "BQ dataset")
            and validate_quoted_identifier(table, "BQ source_table")):
        raise ValueError("unsafe BQ identifier in registry — refusing to query")

    bq_sql = f"SELECT * FROM `{bq.projects.data}.{dataset}.{table}` LIMIT {int(n)}"
    with bq.duckdb_session() as conn:
        try:
            df = conn.execute(
                "SELECT * FROM bigquery_query(?, ?)",
                [bq.projects.billing, bq_sql],
            ).fetchdf()
            return df.to_dict(orient="records")
        except Exception as e:
            raise translate_bq_error(e, bq.projects, bad_request_status="upstream_error")

• Step 2: Update build_sample signature

Replace def build_sample(conn, user, table_id, *, n, project_id): with:

def build_sample(conn, user, table_id, *, n, bq):

Inside, replace _fetch_bq_sample(project_id, ...) with _fetch_bq_sample(bq, ...). Local-source (parquet) path unchanged.

• Step 3: Update sample endpoint

Replace app/api/v2_sample.py:97-110:

@router.get("/sample/{table_id}")
async def sample(
    table_id: str,
    n: int = Query(default=5, ge=1, le=_MAX_N),
    user: dict = Depends(get_current_user),
    conn: duckdb.DuckDBPyConnection = Depends(_get_db),
    bq: BqAccess = Depends(get_bq_access),
):
    try:
        return build_sample(conn, user, table_id, n=n, bq=bq)
    except FileNotFoundError:
        raise HTTPException(status_code=404, detail=f"table {table_id!r} not found")
    except PermissionError:
        raise HTTPException(status_code=403, detail="not authorized for this table")
    except ValueError as e:
        raise HTTPException(
            status_code=400,
            detail={"error": "unsafe_identifier", "message": str(e), "details": {}},
        )
    except BqAccessError as e:
        raise HTTPException(
            status_code=BqAccessError.HTTP_STATUS.get(e.kind, 500),
            detail={"error": e.kind, "message": e.message, "details": e.details},
        )

Add at top:

from connectors.bigquery.access import BqAccess, BqAccessError, get_bq_access

• Step 4: Update tests in tests/test_v2_sample.py

The Phase-1 tests patched _fetch_bq_sample with a 4-arg lambda (project, dataset, table, n). Update all such patches to (bq, dataset, table, n). Convert the Forbidden test to use bq_access fixture (matches pattern from Task 2.7 step 7).

• Step 5: Run — expect PASS

Run: pytest tests/test_v2_sample.py -v Expected: all green.

• Step 6: Commit

git add app/api/v2_sample.py tests/test_v2_sample.py
git commit -m "refactor(v2_sample): #134 migrate to BqAccess facade"

---

Task 2.9: Migrate app/api/v2_schema.py (both blocks, preserve best-effort semantics)

Files:

• Modify: app/api/v2_schema.py.

• Modify: tests/test_v2_schema.py.

• Step 1: Update _fetch_bq_schema (strict block) — replace Phase-1 inline try/except with translate_bq_error

Replace app/api/v2_schema.py:36-... (the _fetch_bq_schema function). Body becomes:

def _fetch_bq_schema(bq, dataset: str, table: str) -> list[dict]:
    from connectors.bigquery.access import translate_bq_error
    from src.identifier_validation import validate_quoted_identifier

    if not (validate_quoted_identifier(bq.projects.data, "BQ project")
            and validate_quoted_identifier(dataset, "BQ dataset")
            and validate_quoted_identifier(table, "BQ source_table")):
        raise ValueError("unsafe BQ identifier in registry")

    bq_sql = (
        f"SELECT column_name, data_type FROM `{bq.projects.data}.{dataset}.INFORMATION_SCHEMA.COLUMNS` "
        f"WHERE table_name = ? "
        f"ORDER BY ordinal_position"
    )
    with bq.duckdb_session() as conn:
        try:
            rows = conn.execute(
                "SELECT * FROM bigquery_query(?, ?, ?)",
                [bq.projects.billing, bq_sql, table],
            ).fetchall()
        except Exception as e:
            raise translate_bq_error(e, bq.projects, bad_request_status="upstream_error")
    return [{"name": r[0], "type": r[1]} for r in rows]

(Adjust the SELECT shape to match what the function returns today; this sketch preserves the {name, type} shape of the existing function.)

• Step 2: Update _fetch_bq_table_options (best-effort block) — preserve swallow-all

Replace app/api/v2_schema.py:90-129:

def _fetch_bq_table_options(bq, dataset: str, table: str) -> dict:
    """Best-effort partition/cluster info. Returns {} on ANY failure (preserved
    from pre-refactor; /schema endpoint must keep returning successfully even
    when partition queries fail)."""
    from src.identifier_validation import validate_quoted_identifier

    if not (validate_quoted_identifier(bq.projects.data, "BQ project")
            and validate_quoted_identifier(dataset, "BQ dataset")
            and validate_quoted_identifier(table, "BQ source_table")):
        return {}

    try:
        with bq.duckdb_session() as conn:
            bq_sql = (
                f"SELECT column_name, is_partitioning_column, clustering_ordinal_position "
                f"FROM `{bq.projects.data}.{dataset}.INFORMATION_SCHEMA.COLUMNS` "
                f"WHERE table_name = ? "
                f"ORDER BY clustering_ordinal_position NULLS LAST"
            )
            rows = conn.execute(
                "SELECT * FROM bigquery_query(?, ?, ?)",
                [bq.projects.billing, bq_sql, table],
            ).fetchall()
        if not rows:
            return {}
        partition_by = next(
            (r[0] for r in rows if (r[1] or "").upper() == "YES"),
            None,
        )
        clustered_by = [r[0] for r in rows if r[2] is not None]
        return {"partition_by": partition_by, "clustered_by": clustered_by}
    except Exception as e:
        logger.warning(
            "BQ table options fetch failed for %s.%s.%s: %s",
            bq.projects.data, dataset, table, e,
        )
        return {}

Note: the outer try/except Exception → return {} is the load-bearing contract preserved from today.

• Step 3: Update build_schema and schema endpoint signatures

build_schema(conn, user, table_id, *, project_id) → build_schema(conn, user, table_id, *, bq). Inside: _fetch_bq_schema(project_id, dataset, table) → _fetch_bq_schema(bq, dataset, table). Same for _fetch_bq_table_options.

schema endpoint (app/api/v2_schema.py:...):

@router.get("/schema/{table_id}")
async def schema(
    table_id: str,
    user: dict = Depends(get_current_user),
    conn: duckdb.DuckDBPyConnection = Depends(_get_db),
    bq: BqAccess = Depends(get_bq_access),
):
    try:
        return build_schema(conn, user, table_id, bq=bq)
    except FileNotFoundError:
        raise HTTPException(status_code=404, detail=f"table {table_id!r} not found")
    except PermissionError:
        raise HTTPException(status_code=403, detail="not authorized for this table")
    except ValueError as e:
        raise HTTPException(
            status_code=400,
            detail={"error": "unsafe_identifier", "message": str(e), "details": {}},
        )
    except BqAccessError as e:
        raise HTTPException(
            status_code=BqAccessError.HTTP_STATUS.get(e.kind, 500),
            detail={"error": e.kind, "message": e.message, "details": e.details},
        )

Add at top:

from connectors.bigquery.access import BqAccess, BqAccessError, get_bq_access

• Step 4: Update tests in tests/test_v2_schema.py

Update _fetch_bq_schema and _fetch_bq_table_options monkeypatches to new signatures (bq, dataset, table). The Phase-1 regression-guard test for partition-info failure should still pass (the contract is preserved).

• Step 5: Run — expect PASS

Run: pytest tests/test_v2_schema.py -v Expected: all green, including the regression guard.

• Step 6: Commit

git add app/api/v2_schema.py tests/test_v2_schema.py
git commit -m "refactor(v2_schema): #134 migrate to BqAccess (strict + best-effort blocks)"

---

Task 2.10: Migrate src/remote_query.py — lazy bq_access, drop _bq_client_factory

Files:

• Modify: src/remote_query.py.

• Step 1: Read current state of __init__ and _get_bq_client

Run: sed -n '195,230p' src/remote_query.py && echo '---' && sed -n '407,455p' src/remote_query.py Expected: see today's _bq_client_factory parameter and the fallback chain in _get_bq_client.

• Step 2: Update __init__ signature

In src/remote_query.py RemoteQueryEngine.__init__:

• Remove _bq_client_factory=None parameter.

• Add bq_access: "BqAccess | None" = None parameter.

• Add self._bq = bq_access line in body.

• Remove self._bq_client_factory = _bq_client_factory line.

• Update the docstring to remove the old factory reference (the stale line 204 about scripts.duckdb_manager._create_bq_client).

• Step 3: Replace _get_bq_client

Delete the existing _get_bq_client (lines ~407-450) and replace with:

def _get_bq_client(self):
    """Lazy-resolve BqAccess on first use. Many tests construct RemoteQueryEngine
    for DuckDB-only paths and never touch BQ — those must not fail with not_configured."""
    if self._bq is None:
        from connectors.bigquery.access import get_bq_access
        self._bq = get_bq_access()  # may raise BqAccessError; that's fine
    return self._bq.client()

• Step 4: Run existing remote_query tests — expect MOSTLY PASS, with failures in tests that injected _bq_client_factory

Run: pytest tests/test_remote_query.py -v 2>&1 | tail -40 Expected: ~10 failures from tests that pass _bq_client_factory=... (now an unknown kwarg) or that mocked the old factory factory chain.

• Step 5: DO NOT commit yet — Task 2.11 fixes the tests. Both must land together.

---

Task 2.11: Migrate tests/test_remote_query.py from _bq_client_factory to bq_access injection

Files:

• Modify: tests/test_remote_query.py.

• Step 1: Find all _bq_client_factory usages

Run: grep -n "_bq_client_factory" tests/test_remote_query.py Note: the spec mentions ~12+ call sites; expect roughly that count.

• Step 2: For each occurrence, migrate to bq_access=BqAccess(..., client_factory=...)

Pattern:

# Before
engine = RemoteQueryEngine(conn, _bq_client_factory=lambda project: mock_client)

# After
from connectors.bigquery.access import BqAccess, BqProjects
bq = BqAccess(
    BqProjects(billing="test-billing", data="test-data"),
    client_factory=lambda projects: mock_client,
)
engine = RemoteQueryEngine(conn, bq_access=bq)

For tests where the factory was lambda project: ... and used project argument: now it's lambda projects: ... (a BqProjects instance). If the test asserted on the project argument, change to assert call.projects.billing == "expected-project".

• Step 3: DuckDB-only tests need NO change

Tests like RemoteQueryEngine(analytics_conn) (no factory) at lines 106, 148, 188, 196, 417, 520, 529, 538: these now have bq_access=None by default and don't trigger get_bq_access() because they never call _get_bq_client. Confirm these pass without modification.

• Step 4: Run — expect PASS

Run: pytest tests/test_remote_query.py -v Expected: all green.

• Step 5: Commit Tasks 2.10 + 2.11 together

git add src/remote_query.py tests/test_remote_query.py
git commit -m "refactor(remote_query): #134 lazy BqAccess injection, drop _bq_client_factory

RemoteQueryEngine.__init__ no longer accepts _bq_client_factory. Tests
migrate to bq_access=BqAccess(projects, client_factory=...) injection.
DuckDB-only tests that never touch BQ continue to work without changes
because bq_access defaults to None and get_bq_access() is only invoked
on first BQ call (lazy resolution).

Drops the stale docstring at the previous src/remote_query.py:204
that referenced scripts.duckdb_manager._create_bq_client as the default
factory — RemoteQueryEngine never actually used that function.

CLI (cli/commands/query.py) is unaffected: it never injected the factory,
and the new bq_access kwarg's None default routes through get_bq_access()
on first BQ call — matching today's eager behavior for the CLI path."

---

Task 2.12: CHANGELOG entries for Phase 2

Files: CHANGELOG.md.

• Step 1: Add ### Changed BREAKING entry under ## [Unreleased]

Add immediately below the ### Fixed block from Phase 1:

### Changed
- **BREAKING for deployments using `BIGQUERY_PROJECT` env var alongside `data_source.bigquery.project` in `instance.yaml`.** The env var now sets BOTH billing and data project, overriding `data_source.bigquery.project` for FROM-clause construction in `v2_scan` / `v2_sample` / `v2_schema`. Migrate by clearing `BIGQUERY_PROJECT` and using `data_source.bigquery.billing_project` + `data_source.bigquery.project` in `instance.yaml`. (Previously `BIGQUERY_PROJECT` only affected `RemoteQueryEngine` billing.)

• Step 2: Add ### Internal entries

### Internal
- New shared module `connectors/bigquery/access.py` — `BqAccess` facade unifies BQ project resolution, client construction, DuckDB-extension session, and Google-API error translation across `v2_scan`, `v2_sample`, `v2_schema`, and `RemoteQueryEngine`. Replaces four duplicate code paths with one.
- **Internal API change:** `RemoteQueryEngine.__init__` no longer accepts `_bq_client_factory`. Callers that injected it migrate to `RemoteQueryEngine(..., bq_access=BqAccess(projects, client_factory=...))`. The CLI (`cli/commands/query.py`) is unaffected — it never injected the factory and the new `bq_access` kwarg defaults to `None` (lazy `get_bq_access()` on first BQ call).
- Removed stale docstring in `src/remote_query.py` referencing `scripts.duckdb_manager._create_bq_client` as the default BQ client factory (the engine never actually used that function).
- Two known-duplicate BQ-access sites (`connectors/bigquery/extractor.py`, `scripts/duckdb_manager.register_bq_table`) explicitly out of scope for this PR; tracked as follow-up.

• Step 3: Commit

git add CHANGELOG.md
git commit -m "docs(changelog): #134 Phase 2 BqAccess refactor — Changed BREAKING + Internal"

---

Phase 3 — Verification

Task 3.1: Run full test suite

• Step 1: Run all tests

Run:

pytest tests/ -v --tb=short 2>&1 | tee /tmp/test-output-issue-134.log

Expected: every test passes. If failures appear that aren't in BQ/v2 paths, investigate — they may be flaky tests unrelated to this PR; document any pre-existing failures.

• Step 2: Run linters / type checks if the repo enforces them

Run: ruff check connectors/bigquery/access.py app/api/v2_*.py src/remote_query.py tests/test_bq_access.py 2>&1 | head -30 Expected: no new lint errors introduced.

Run: mypy connectors/bigquery/access.py 2>&1 | head -30 Expected: no new type errors. (CI may run with continue-on-error per repo convention — match that.)

• Step 3: If any failures, fix them before proceeding

Do NOT proceed to E2E with red CI.

---

Task 3.2: Squash dev commits into final PR shape (two commits)

The intermediate commits during development can be squashed into the two PR-shape commits that the spec calls for.

• Step 1: Identify the boundary

Phase 1 ended with the squash commit fix(v2): #134 structured 502/400 on BQ errors across /scan, /scan/estimate, /sample, /schema. Find its SHA:

git log --oneline | grep "structured 502/400 on BQ errors across"

Note this SHA — call it $PHASE1.

• Step 2: Soft-reset everything after $PHASE1 to staging

git reset --soft $PHASE1
git status   # expect a large pile of staged changes representing Phase 2

• Step 3: Make the single Phase 2 commit

git commit -m "refactor(bq): #134 BqAccess facade — unify v2_scan, v2_sample, v2_schema, RemoteQueryEngine

Extracts the duplicated BigQuery-access pattern (project resolution +
client construction + DuckDB-extension session + Google-API error
translation) into connectors/bigquery/access.py. Migrates four
call sites to use it:

- app/api/v2_scan.py — _bq_dry_run_bytes, _run_bq_scan
- app/api/v2_sample.py — _fetch_bq_sample
- app/api/v2_schema.py — _fetch_bq_schema (strict translation),
  _fetch_bq_table_options (preserves swallow-all best-effort contract)
- src/remote_query.py — RemoteQueryEngine, lazy bq_access kwarg

Removes _bq_client_factory parameter from RemoteQueryEngine.__init__
and the stale docstring referencing scripts.duckdb_manager._create_bq_client.
Tests migrate from _bq_client_factory injection to
bq_access=BqAccess(projects, client_factory=...) injection. DuckDB-only
RemoteQueryEngine tests need no changes (lazy resolution skips
get_bq_access() when bq_access is None and BQ is never touched).

BREAKING for deployments combining BIGQUERY_PROJECT env var with
data_source.bigquery.project in instance.yaml — the env var now
overrides data project too. See CHANGELOG.

Two known-duplicate BQ-access sites (connectors/bigquery/extractor.py,
scripts/duckdb_manager.register_bq_table) explicitly out of scope;
tracked as follow-up."

• Step 4: Verify final shape

Run: git log --oneline | head -10 Expected: top two commits are exactly the two intended for the PR (one fix(v2), one refactor(bq)).

---

Task 3.3: E2E verification on <your-dev-instance>

This task happens AFTER the PR is merged and deployed to <your-dev-instance>. Per the spec, this is the success criterion for closing #134 — without it, "fixed" is unverifiable.

Prerequisite: <your-dev-instance> is running the PR's image; reproduce the PAT used in Pavel's report.

• Step 1: Pre-config baseline (BEFORE setting billing_project)

PAT=...

curl -k -i -H "Authorization: Bearer $PAT" \
  https://<your-agnes-host>/api/v2/sample/<bq_table_id>?n=2

curl -k -i -X POST -H "Authorization: Bearer $PAT" -H "Content-Type: application/json" \
  https://<your-agnes-host>/api/v2/scan/estimate \
  -d '{"table_id":"<bq_table_id>","select":["event_date"],"where":"event_date = DATE \"2026-04-21\""}'

curl -k -i -X POST -H "Authorization: Bearer $PAT" -H "Content-Type: application/json" \
  https://<your-agnes-host>/api/v2/scan \
  -d '{"table_id":"<bq_table_id>","select":["event_date"],"where":"event_date = DATE \"2026-04-21\"","limit":50}'

curl -k -i -H "Authorization: Bearer $PAT" \
  https://<your-agnes-host>/api/v2/schema/<bq_table_id>

Expected (per spec § E2E manual verification):

• /sample, /scan, /scan/estimate: HTTP 502 with JSON body containing error=cross_project_forbidden + details.hint mentioning billing_project.

• /schema: HTTP 200 (INFORMATION_SCHEMA queries don't trip serviceusage; only fails if something else breaks the strict block).

• Step 2: Operator action — set billing_project and restart

In instance.yaml on the VM:

data_source:
  bigquery:
    project: <your-data-project>        # data project, unchanged
    billing_project: <project-where-SA-can-bill>   # NEW

Restart the container.

• Step 3: Post-config verification

Repeat the four curls from Step 1.

Expected: HTTP 200 on all four, with valid response bodies.

• Step 4: Document and close #134

Add a comment to issue #134 with:

• The four curl invocations + their pre/post status codes.
• Confirmation that the structured body shape is what was specced.
• Note any unexpected behavior.

Close the issue.

---

Self-review (against the spec)

Spec section	Plan task
Bug A — v2_scan missing try/except	Tasks 1.2, 1.3 + 2.7
Bug B — v2_sample missing billing_project + try/except	Tasks 1.1 + 2.8
Bug C — v2_schema (two blocks, two semantics)	Tasks 1.4 + 2.9
BqProjects + BqAccessError	Task 2.1
translate_bq_error (incl. BqAccessError pass-through)	Task 2.2
_default_client_factory, _default_duckdb_session_factory	Task 2.3
BqAccess class with injectable factories	Task 2.4
get_bq_access module-level cached	Task 2.5
bq_access fixture in conftest	Task 2.6
RemoteQueryEngine lazy bq_access; drop _bq_client_factory	Tasks 2.10 + 2.11
CHANGELOG (Fixed + Changed BREAKING + Internal)	Tasks 1.5 + 2.12
Two-commit PR shape	Task 3.2
E2E verification protocol	Task 3.3
Cross-project Forbidden heuristic ('serviceusage' only)	Task 2.2 (test_forbidden_diff_projects_no_serviceusage_still_bq_forbidden)
Status code split (400/502/500)	Task 2.1 (test_http_status_map_covers_all_kinds)
_fetch_bq_table_options swallow-all preserved	Task 2.9 + Task 1.4 regression guard

All spec sections have at least one task implementing them. No placeholders in the plan body.

Final notes

• Each task's tests come BEFORE the implementation per TDD. Run the failing test first, then implement, then verify pass.
• Intermediate commits during dev are fine — Task 3.2 squashes them into the two PR-shape commits.
• The plan stays in the worktree (<worktree-root>, branch fix/134-bq-access-unify). All commits land there. Push to GitHub when ready for PR.
• If a task's TDD step finds an issue with the spec's design, STOP and update the spec before proceeding. Don't paper over design holes in implementation.