agnes-the-ai-analyst/connectors
ZdenekSrotyr 83209f32b0 perf(bq): pool DuckDB BQ extension sessions to amortize INSTALL/LOAD/ATTACH cost
Each BqAccess.duckdb_session() acquire previously created a fresh
in-memory DuckDB conn and ran INSTALL bigquery; LOAD bigquery;
CREATE SECRET; ATTACH on it -- costing ~0.5 s per request even before
any BQ work. Add a process-local pool (deque + lock) of pre-warmed
sessions; acquire reuses a warm entry when available, refreshing the
auth SECRET so a long-lived pool entry doesn't keep a stale GCE
metadata token past its TTL. Liveness probe (cheap SELECT 1) drops
broken entries before handing them to callers.

On exception inside the with-block the conn is closed instead of
returned to pool (session may carry dirty state). Pool size is
data_source.bigquery.session_pool_size (default 4; sentinel 0
disables pooling). Process-cached, not fork-safe (single uvicorn
worker is the supported deployment shape per CLAUDE.md).

All call sites get faster automatically: /api/query, /api/v2/{scan,
sample,schema}, materialize, the orchestrator's remote-attach, and
the BQ dry-run cap-guard.
2026-05-06 13:06:25 +02:00
..
bigquery perf(bq): pool DuckDB BQ extension sessions to amortize INSTALL/LOAD/ATTACH cost 2026-05-06 13:06:25 +02:00
jira chore(docs): replace stale da verbs and vendor-specific install paths 2026-05-04 21:22:19 +02:00
keboola fix(keboola): per-table fallback to legacy Storage-API client (#183) 2026-05-05 15:47:44 +02:00
llm fix(anthropic): strict json_schema (additionalProperties=false) + add /admin/scheduler-runs UI 2026-05-05 08:00:57 +02:00
openmetadata feat(scheduler): re-wire sync_schedule + script.schedule; tune via env; OpenMetadata TLS (#135) 2026-04-29 22:06:30 +02:00
__init__.py Extract Jira into connectors/jira module 2026-03-09 11:17:50 +01:00