758910463b
BigQuery connector that syncs BQ tables to local Parquet files via PyArrow (no CSV intermediate step). Supports full refresh, timestamp-based incremental (via incremental_column), and partition-based sync strategies. - connectors/bigquery/client.py: BQ API wrapper with ADC auth, parameterized queries, metadata cache, cross-project support (job project != data project) - connectors/bigquery/adapter.py: DataSource implementation with merge/dedup - src/config.py: Add incremental_column field to TableConfig - 72 unit tests (mocked, no GCP SDK required)
60 lines
1.9 KiB
Text
60 lines
1.9 KiB
Text
# Data source adapters (install only what you need)
|
|
kbcstorage>=0.9.0 # For Keboola adapter
|
|
google-cloud-bigquery>=3.0.0 # For BigQuery adapter
|
|
google-cloud-bigquery-storage>=2.0.0 # For BigQuery adapter (fast Arrow transfer)
|
|
|
|
# Data processing
|
|
# pandas - core tabular data processing library
|
|
# pyarrow - Parquet format support and fast operations
|
|
# pytz - timezone support required by DuckDB for reading timezone-aware Parquet columns
|
|
pandas>=2.0.0
|
|
pyarrow>=12.0.0
|
|
pytz>=2024.1
|
|
|
|
# Analytical database
|
|
# DuckDB - in-process SQL OLAP database for analytical queries
|
|
duckdb>=0.9.0
|
|
|
|
# Configuration
|
|
# python-dotenv - loading environment variables from .env files
|
|
# pyyaml - parsing YAML configuration from data_description.md
|
|
python-dotenv>=1.0.0
|
|
pyyaml>=6.0
|
|
|
|
# Progress tracking and logging
|
|
# tqdm - progress bars for long-running operations (download, sync)
|
|
tqdm>=4.65.0
|
|
|
|
# Web application (Google SSO portal)
|
|
# flask - web framework for self-service portal
|
|
# authlib - OAuth 2.0 / OpenID Connect library for Google SSO
|
|
# gunicorn - WSGI server for production deployment
|
|
flask>=3.0.0
|
|
authlib>=1.3.0
|
|
gunicorn>=21.0.0
|
|
|
|
# Telegram notification bot
|
|
# httpx - async HTTP client for Telegram API and unix socket communication
|
|
# aiohttp - async HTTP server for bot's internal send API
|
|
httpx>=0.27.0
|
|
aiohttp>=3.9.0
|
|
matplotlib>=3.8.0
|
|
numpy>=1.24.0
|
|
|
|
# Desktop app authentication
|
|
# PyJWT - JWT token creation and validation for desktop app auth
|
|
PyJWT>=2.8.0
|
|
|
|
# Password authentication for external users
|
|
# argon2-cffi - modern password hashing algorithm (Argon2id)
|
|
# sendgrid - email service for setup/reset links
|
|
argon2-cffi>=23.1.0
|
|
sendgrid>=6.11.0
|
|
|
|
# Corporate Memory knowledge extraction
|
|
# anthropic - Claude API client for HAIKU-based knowledge extraction
|
|
anthropic>=0.39.0
|
|
|
|
# Sample data generation (development/testing)
|
|
# faker - realistic synthetic data for demo datasets
|
|
faker>=24.0.0
|