Table Management

Cache freshness

Re-warm all

Loading…

Show log

Live log from Re-warm all. Each row reports [hh:mm:ss] ok/error <table_id> (duration) as the BQ metadata refresh streams in via SSE. Only query_mode='remote' BigQuery tables are warmed (e.g. product_inventory_remote); materialized + internal tables don't need it. Historical runs aren't persisted; check /admin/activity for the audit trail.

{# Phase D: tab-split scaffold. Per-connector tabs (BigQuery / Keboola / Jira) replace the single mixed form. Each tab has its own Register button + listing div + (later) form modals. The initial active tab matches data_source.type from instance.yaml; the operator can still switch tabs to manage a secondary source. Phase E moves the BQ form into #tab-content-bigquery; Phase F builds the Keboola form inside #tab-content-keboola. For now the existing Jinja-branched panels below stay in place. #} {% set initial_tab = data_source_type if data_source_type in ['bigquery', 'keboola', 'jira'] else 'keboola' %}

Register BigQuery table

Register BigQuery Table

{# Two orthogonal questions: (1) live vs synced, (2) when synced, whole table vs custom SQL. Visibility classes: bq-access-live — only when accessMode='live' bq-access-synced — only when accessMode='synced' bq-source-table — only when accessMode='live' OR (accessMode='synced' AND syncMode='whole') bq-source-custom — only when accessMode='synced' AND syncMode='custom' Backend payload: live → query_mode='remote'; synced/whole → query_mode='materialized' with auto-built SELECT *; synced/custom → query_mode='materialized' with admin SQL. Server auto-detects BASE TABLE vs VIEW at register time, so the UI doesn't ask. #}

How should analysts access this data?

Live from BigQuery

Each analyst query goes straight to BQ. Always current. Latency ≈ seconds; 0 disk on the analyst machine; cost = bytes scanned per query. Best for huge tables or when freshness matters.

Synced locally

Agnes runs a SELECT on a schedule and ships a parquet to analysts. Analyst-side latency <100 ms; disk = snapshot size. Best when analysts hit the same data often and speed beats freshness.

What to sync?

Whole table

Agnes runs SELECT * automatically. No SQL required. Disk + sync cost = full table size.

Custom query

You write the SELECT — filter, project, or aggregate before the sync. Cuts disk + cost; cap via max_bytes_per_materialize guardrail.

Dataset Discover

BigQuery dataset name (no project prefix — read from instance.yaml). Click Discover to populate the autocomplete from the BQ project's dataset list.

Source Table / View List tables

Table or view name within the dataset. Click List tables after filling Dataset to populate autocomplete.
Live access: BASE TABLEs query via bq."dataset"."table" (Storage Read API; predicate pushdown). VIEWs and MATERIALIZED_VIEWs query via the BQ jobs API (full-scan estimate; cost-guarded by bq_max_scan_bytes). agnes query --remote works for both.
Synced access: handles both table and view transparently — the scheduler runs SELECT * through the jobs API and writes a parquet.

SQL Use table as base

SELECT statement, no trailing semicolon. Native BQ identifiers (`project.dataset.table`) recommended — DuckDB three-part names like bq."ds"."t" work for the COPY but disable the cost guardrail's BQ dry-run.

View Name

Name analysts use to query the data (e.g. SELECT * FROM orders_90d). Required for Custom query; defaults to the source table for the other modes.

Description (optional)

Folder (optional)

Logical grouping for catalog organization

Sync Schedule (optional, default every 1h)

How often Agnes refreshes the local copy. Examples: every 15m, every 6h, daily 03:00, daily 07:00,13:00,18:00 (UTC).

Source check

Cancel Register Table

Edit BigQuery Table

Table ID

Slugified id, immutable. Source type: bigquery

How should analysts access this data? ?

Live from BigQuery

Each query goes to BQ. No local copy.

Synced locally

Scheduled SELECT → parquet, queried locally.

What to sync?

Whole table

SELECT * on a schedule.

Custom query

Filter / aggregate before sync.

Dataset Discover

Source Table / View List tables

Table or view name within the dataset.
Live access: BASE TABLEs query via bq."dataset"."table" (Storage Read API; predicate pushdown). VIEWs and MATERIALIZED_VIEWs query via the BQ jobs API (full-scan estimate; cost-guarded by bq_max_scan_bytes). agnes query --remote works for both.
Synced access: handles both transparently — the scheduler runs SELECT * through the jobs API and writes a parquet.

SQL Use table as base

SELECT statement, no trailing semicolon. Native BQ identifiers recommended for the cost guardrail to engage.

Sync Schedule (optional)

How often Agnes refreshes the local copy. every 15m, every 6h, daily 03:00 (UTC).

Description (optional)

Folder (optional)

Logical grouping for catalog organization (does not affect storage).

Cancel Save Changes

Register Keboola table

Register Keboola Table

{# Three sync-mode radios: - whole / custom → query_mode='materialized' (DuckDB Keboola extension; whole synthesizes SELECT *, custom uses admin SQL) - direct → query_mode='local' (Storage API SDK, supports v26 sync strategies: incremental/partitioned + where_filters) #}

What to sync?

Whole table (extension)

DuckDB Keboola extension pulls the full table on each tick. Fastest path; full overwrite each run.

Direct extract (Storage API)

Storage API SDK. Supports incremental sync (changedSince + PK merge), partitioned files, and server-side where_filters.

Custom SQL

Pre-aggregate or filter with your own SELECT (e.g. last 30 days only, per-day rollup).

View name (analyst-visible)

{# Discover/List tables backend currently routes by instance's data_source.type ignoring the `source` query param. Hiding the buttons on non-Keboola instances prevents wrong-shape responses; inputs stay for manual entry. Future fix: make /api/admin/discover-tables accept ?source=keboola and remove this guard. #}

Bucket {% if data_source_type == 'keboola' %} Discover {% endif %}

Source Table {% if data_source_type == 'keboola' %} List tables {% endif %}

SQL {% if data_source_type == 'keboola' %} Use table as base {% endif %}

SELECT against kbc."bucket"."table". Result is materialized to parquet and distributed via agnes pull.

Sync Schedule (optional, default every 1h)

How often Agnes refreshes the local copy. Examples: every 15m, every 6h, daily 03:00, daily 07:00,13:00,18:00 (UTC).

Description (optional)

Folder (optional)

Advanced (optional)

Primary Key

Comma-separated list. Required for Direct extract → Incremental (used as the dedup key on delta merge). Auto-filled from the Keboola source when available.

Direct extract — sync strategy

Sync strategy Full refresh — pull entire table each tick Incremental — pull rows changed since last sync Partitioned — per-partition files, per-month/day/year

Incremental window (days) (default 7)

Backtrack window applied to last_sync timestamp on each tick. Higher = more reliable on late-arriving rows; lower = less data per tick.

Max history (days) (first sync only, default unbounded)

Cap on how far back the first-ever sync goes. Multi-year tables without this can OOM at write — set 90/180/365 for safety.

Partition by column (required)

Date / timestamp column whose value drives the partition key. Rows with NULL or unparseable values are dropped (logged warning).

Granularity Month — YYYY_MM.parquet (default) Day — YYYY_MM_DD.parquet Year — YYYY.parquet

Initial-load chunk size (days) (default 30)

First-sync chunked load step. Smaller = more API calls, less memory per chunk. Larger = fewer calls, more memory.

Where filters (JSON array, optional)

Server-side row filter. Operators: eq, ne, gt, ge, lt, le. Date placeholders resolved at sync time: {{ '{{today}}' }}, {{ '{{last_week}}' }}, {{ '{{last_month}}' }}, {{ '{{last_2_months}}' }}, {{ '{{last_3_months}}' }}, {{ '{{last_6_months}}' }}, {{ '{{last_year}}' }}, {{ '{{last_2_years}}' }}, {{ '{{start_of_3_months_ago}}' }}. Not compatible with Incremental strategy.

Cancel Register

Edit Keboola Table

Table ID

Slugified id, immutable.

{# Three sync-mode radios (mirror of Register). #}

What to sync?

Whole table (extension)

DuckDB Keboola extension; full overwrite each tick.

Direct extract (Storage API)

Storage API SDK. Supports incremental, partitioned, where_filters.

Custom SQL

Pre-aggregate or filter with your own SELECT.

{# Discover/List tables backend currently routes by instance's data_source.type ignoring the `source` query param. Hiding the buttons on non-Keboola instances prevents wrong-shape responses; inputs stay for manual entry. Future fix: make /api/admin/discover-tables accept ?source=keboola and remove this guard. #}

Bucket {% if data_source_type == 'keboola' %} Discover {% endif %}

Source Table {% if data_source_type == 'keboola' %} List tables {% endif %}

SQL {% if data_source_type == 'keboola' %} Use table as base {% endif %}

Sync Schedule (optional)

Description (optional)

Folder (optional)

Advanced (optional)

Primary Key

Comma-separated list. Required for Direct extract → Incremental.

Direct extract — sync strategy

Sync strategy Full refresh — pull entire table each tick Incremental — pull rows changed since last sync Partitioned — per-partition files, per-month/day/year

Incremental window (days) (default 7)

Max history (days) (first sync only)

Partition by column (required)

Granularity Month (default) Day Year

Initial-load chunk size (days) (default 30)

Where filters (JSON array, optional)

Operators: eq, ne, gt, ge, lt, le. Date placeholders ({{ '{{today}}' }}, {{ '{{last_3_months}}' }}, etc.) resolved at sync time. Not compatible with Incremental.

Cancel Save Changes

Jira tables are populated by webhooks. To register a new Jira webhook integration, see docs/connectors/jira.md.

{# Fourth tab: Agnes internal data source. Auto-seeded read-only rows backed by system.duckdb (agnes_sessions / agnes_telemetry / agnes_audit). No Register button — these are seeded on every app boot from connectors/internal/registry.py. The tab-content visibility matches the BQ/Keboola/Jira pattern above so switchTab() handles it without code changes. #}

System tables — sessions, telemetry, audit. Also available locally for analysis. Read-only here; Edit / Delete are hidden because the rows are seeded on every app boot.

{# Legacy out-of-tab panels (BQ Register card, Keboola Discovery card, shared Registered Tables wrapper) removed — each tab now owns its own header (with Register button) and listing div. The Refresh action is implicit: registration / edit / delete flows already call loadRegistry(), which re-renders all three per-tab listings. #}