agnes-the-ai-analyst/connectors/bigquery
Petr 8bb46a9e0a Add per-partition streaming sync and hybrid query architecture
Partitioned sync: iterates day-by-day instead of loading full dataset.
Each partition: query BQ -> stream to disk -> free RAM. Peak ~50 MB.
New helpers: _sync_single_partition, _cleanup_old_partitions, _generate_partition_dates.

Config: added partition_column_type (DATE/TIMESTAMP/DATETIME), query_mode (local/remote/hybrid).
DuckDB manager: hybrid architecture support (local Parquet + remote BQ tables).
Data sync: skips remote tables, filters by query_mode.

Tests: 113 passing (adapter, client, config, data_sync, duckdb_manager).
2026-03-12 13:20:41 +01:00
..
__init__.py Add BigQuery data source adapter 2026-03-11 13:56:12 +01:00
adapter.py Add per-partition streaming sync and hybrid query architecture 2026-03-12 13:20:41 +01:00
client.py Pass explicit bqstorage_client to to_arrow_iterable() for Storage API 2026-03-12 10:51:44 +01:00