agnes-the-ai-analyst/config
ZdenekSrotyr 83209f32b0 perf(bq): pool DuckDB BQ extension sessions to amortize INSTALL/LOAD/ATTACH cost
Each BqAccess.duckdb_session() acquire previously created a fresh
in-memory DuckDB conn and ran INSTALL bigquery; LOAD bigquery;
CREATE SECRET; ATTACH on it -- costing ~0.5 s per request even before
any BQ work. Add a process-local pool (deque + lock) of pre-warmed
sessions; acquire reuses a warm entry when available, refreshing the
auth SECRET so a long-lived pool entry doesn't keep a stale GCE
metadata token past its TTL. Liveness probe (cheap SELECT 1) drops
broken entries before handing them to callers.

On exception inside the with-block the conn is closed instead of
returned to pool (session may carry dirty state). Pool size is
data_source.bigquery.session_pool_size (default 4; sentinel 0
disables pooling). Process-cached, not fork-safe (single uvicorn
worker is the supported deployment shape per CLAUDE.md).

All call sites get faster automatically: /api/query, /api/v2/{scan,
sample,schema}, materialize, the orchestrator's remote-attach, and
the BQ dry-run cap-guard.
2026-05-06 13:06:25 +02:00
..
.env.template feat(observability): request_id end-to-end + dev debug toolbar + centralized logging (#136) 2026-04-29 22:54:21 +02:00
__init__.py Initial commit: OSS data distribution platform 2026-03-08 23:31:28 +01:00
agnes_workspace_template.txt feat(cli): agnes init orchestrator + AGNES_WORKSPACE.md template 2026-05-04 18:15:08 +02:00
claude_md_template.txt feat: clean CLI errors + init progress + skip-materialize + claude.md catalog pointer 2026-05-05 18:11:59 +02:00
deploy.staging.yml feat: add dataset permissions, script execution, Kamal config, CI/CD 2026-03-27 15:40:11 +01:00
deploy.yml chore(deploy): trust proxy headers + document HTTPS env vars (#48) 2026-04-24 08:52:53 +02:00
instance.yaml.example perf(bq): pool DuckDB BQ extension sessions to amortize INSTALL/LOAD/ATTACH cost 2026-05-06 13:06:25 +02:00
loader.py feat(ci+tests): deploy safety audit — linting, rollback, smoke tests, 50+ new tests (#120) 2026-04-29 09:18:55 +02:00