Table Management

Discover, register, and manage data tables from your source

{# Phase D: tab-split scaffold. Per-connector tabs (BigQuery / Keboola / Jira) replace the single mixed form. Each tab has its own Register button + listing div + (later) form modals. The initial active tab matches data_source.type from instance.yaml; the operator can still switch tabs to manage a secondary source. Phase E moves the BQ form into #tab-content-bigquery; Phase F builds the Keboola form inside #tab-content-keboola. For now the existing Jinja-branched panels below stay in place. #} {% set initial_tab = data_source_type if data_source_type in ['bigquery', 'keboola', 'jira'] else 'keboola' %}

{# Two orthogonal questions: (1) live vs synced, (2) when synced, whole table vs custom SQL. Visibility classes: bq-access-live — only when accessMode='live' bq-access-synced — only when accessMode='synced' bq-source-table — only when accessMode='live' OR (accessMode='synced' AND syncMode='whole') bq-source-custom — only when accessMode='synced' AND syncMode='custom' Backend payload: live → query_mode='remote'; synced/whole → query_mode='materialized' with auto-built SELECT *; synced/custom → query_mode='materialized' with admin SQL. Server auto-detects BASE TABLE vs VIEW at register time, so the UI doesn't ask. #}

How should analysts access this data?

Live from BigQuery

Each analyst query goes straight to BQ. Always current. Latency ≈ seconds; 0 disk on the analyst machine; cost = bytes scanned per query. Best for huge tables or when freshness matters.

Synced locally

Agnes runs a SELECT on a schedule and ships a parquet to analysts. Analyst-side latency <100 ms; disk = snapshot size. Best when analysts hit the same data often and speed beats freshness.

Dataset

BigQuery dataset name (no project prefix — read from instance.yaml). Click Discover to populate the autocomplete from the BQ project's dataset list.

Source Table / View

Table or view name within the dataset. Click List tables after filling Dataset to populate autocomplete.
Live access: BASE TABLEs query via bq."dataset"."table" (Storage Read API; predicate pushdown). VIEWs and MATERIALIZED_VIEWs query via the BQ jobs API (full-scan estimate; cost-guarded by bq_max_scan_bytes). da query --remote works for both.
Synced access: handles both table and view transparently — the scheduler runs SELECT * through the jobs API and writes a parquet.

View Name

Name analysts use to query the data (e.g. SELECT * FROM orders_90d). Required for Custom query; defaults to the source table for the other modes.

Description (optional)

Folder (optional)

Logical grouping for catalog organization

Source check

Table ID

Slugified id, immutable. Source type: bigquery

How should analysts access this data?

Live from BigQuery

Each query goes to BQ. No local copy.

Synced locally

Scheduled SELECT → parquet, queried locally.

Description (optional)

Folder (optional)

Logical grouping for catalog organization (does not affect storage).

{# Q2 radio — Sync mode. (Q1 is implicitly 'synced'; Keboola has no Live mode.) Whole and Custom both map to query_mode='materialized'. #}

What to sync?

Whole table

Pull everything in the bucket/table on each schedule tick. Disk + sync cost = full table size.

Custom SQL

Pre-aggregate or filter with your own SELECT (e.g. last 30 days only, per-day rollup).

View name (analyst-visible)

{# Discover/List tables backend currently routes by instance's data_source.type ignoring the `source` query param. Hiding the buttons on non-Keboola instances prevents wrong-shape responses; inputs stay for manual entry. Future fix: make /api/admin/discover-tables accept ?source=keboola and remove this guard. #}

Bucket {% if data_source_type == 'keboola' %} {% endif %}

Source Table {% if data_source_type == 'keboola' %} {% endif %}

Sync Schedule (optional, default every 1h)

How often Agnes refreshes the local copy. Examples: every 15m, every 6h, daily 03:00, daily 07:00,13:00,18:00 (UTC).

Description (optional)

Folder (optional)

Advanced (optional)

Primary Key

Comma-separated list. Catalog metadata only — Agnes always does full-overwrite sync; no upsert/dedup. Auto-filled from the Keboola source when available.

Table ID

Slugified id, immutable.

{# Q2 radio — Sync mode (mirror of Register). #}

What to sync?

Whole table

Pull everything in the bucket/table on each schedule tick.

Custom SQL

Pre-aggregate or filter with your own SELECT.

Bucket {% if data_source_type == 'keboola' %} {% endif %}

Source Table {% if data_source_type == 'keboola' %} {% endif %}

Sync Schedule (optional)

Description (optional)

Folder (optional)

Advanced (optional)

Primary Key

Comma-separated list. Catalog metadata only — Agnes always does full-overwrite sync.

Jira tables are populated by webhooks. To register a new Jira webhook integration, see docs/connectors/jira.md.

{# Legacy out-of-tab panels (BQ Register card, Keboola Discovery card, shared Registered Tables wrapper) removed — each tab now owns its own header (with Register button) and listing div. The Refresh action is implicit: registration / edit / delete flows already call loadRegistry(), which re-renders all three per-tab listings. #}

Table Management

Register BigQuery Table

Edit BigQuery Table

Register Keboola Table

Edit Keboola Table

Edit Table