-
- {% else %}
-
-
-
-
-
-
-
-
-
-
-
- Register BigQuery Table
- Manually register a BQ table or view as a remote DuckDB view
-
- BigQuery dataset/table discovery lands in Milestone 2 of issue #108. For now, enter the dataset + table by hand.
-
-
-
- {% endif %}
+ {# Phase D: tab-split scaffold. Per-connector tabs (BigQuery /
+ Keboola / Jira) replace the single mixed form. Each tab has its
+ own Register button + listing div + (later) form modals. The
+ initial active tab matches data_source.type from instance.yaml;
+ the operator can still switch tabs to manage a secondary source.
-
-
-
-
-
-
-
-
-
-
-
- Discover Tables
- Scan your data source for available tables
-
-
-
- Click "Discover tables from source" to scan for available tables
-
-
-
-
-
-
+ Phase E moves the BQ form into #tab-content-bigquery; Phase F
+ builds the Keboola form inside #tab-content-keboola. For now
+ the existing Jinja-branched panels below stay in place. #}
+ {% set initial_tab = data_source_type if data_source_type in ['bigquery', 'keboola', 'jira'] else 'keboola' %}
+
+
+
+
+
+
+
+
+
+
+
+
+
-
+
- Register BigQuery Table
+
-
-
Registered Tables
- Tables currently configured for sync
+
+ {# Two orthogonal questions: (1) live vs synced, (2) when synced,
+ whole table vs custom SQL. Visibility classes:
+ bq-access-live — only when accessMode='live'
+ bq-access-synced — only when accessMode='synced'
+ bq-source-table — only when accessMode='live' OR
+ (accessMode='synced' AND syncMode='whole')
+ bq-source-custom — only when accessMode='synced' AND syncMode='custom'
+ Backend payload: live → query_mode='remote'; synced/whole →
+ query_mode='materialized' with auto-built SELECT *; synced/custom
+ → query_mode='materialized' with admin SQL. Server auto-detects
+ BASE TABLE vs VIEW at register time, so the UI doesn't ask. #}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ BigQuery dataset name (no project prefix — read from instance.yaml).
+ Click Discover to populate the autocomplete from the BQ project's dataset list.
+
+
+
+
+
+ Table or view name within the dataset. Click
+ List tables after filling Dataset to populate autocomplete.
+
Live access: BASE TABLEs are queryable directly via +
Synced access: handles both table and view transparently + — the scheduler runs
+ Live access: BASE TABLEs are queryable directly via +
da query --remote; VIEWs are registered but analysts must run
+ da fetch to materialize a local snapshot (or the admin can flip
+ data_source.bigquery.legacy_wrap_views=true to wrap views via the
+ BQ jobs API).
+ Synced access: handles both table and view transparently + — the scheduler runs
SELECT * through the jobs API and writes a
+ parquet.
+
+
+
+
+ SELECT statement, no trailing semicolon. Native BQ identifiers
+ (
+ `project.dataset.table`) recommended — DuckDB three-part
+ names like bq."ds"."t" work for the COPY but disable the
+ cost guardrail's BQ dry-run.
+
+
+
+
+ Name analysts use to query the data (e.g.
+
+ SELECT * FROM orders_90d). Required for Custom query; defaults
+ to the source table for the other modes.
+
+
+
+
+
+
+
+ Logical grouping for catalog organization
+
+
+
+
+
+ How often Agnes refreshes the local copy. Examples:
+
+ every 15m, every 6h,
+ daily 03:00, daily 07:00,13:00,18:00 (UTC).
+
+
+ Source check
+
+
+
+
-
+
+
+
+
+
+ {# Legacy out-of-tab panels (BQ Register card, Keboola Discovery card,
+ shared Registered Tables wrapper) removed — each tab now owns its
+ own header (with Register button) and listing div. The Refresh
+ action is implicit: registration / edit / delete flows already
+ call loadRegistry(), which re-renders all three per-tab listings. #}
-
-
-
-
- Loading registry...
+
+
+
+
+
+
+
-
+
+
+ Edit BigQuery Table
+ +
+
+
+
+
+
+
+ Slugified id, immutable. Source type:
+ bigquery
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Table or view name within the dataset.
+
Live access: BASE TABLEs are queryable directly via +
Synced access: handles both transparently — the + scheduler runs
+ Live access: BASE TABLEs are queryable directly via +
da query --remote; VIEWs are registered but analysts must run
+ da fetch to materialize a local snapshot (or admin can flip
+ data_source.bigquery.legacy_wrap_views=true).
+ Synced access: handles both transparently — the + scheduler runs
SELECT * through the jobs API and writes a
+ parquet.
+
+
+
+ SELECT statement, no trailing semicolon. Native BQ
+ identifiers recommended for the cost guardrail to engage.
+
+
+
+
+
+ How often Agnes refreshes the local copy.
+
+ every 15m, every 6h,
+ daily 03:00 (UTC).
+
+
+
+
+
+
+
+ Logical grouping for catalog organization (does not affect storage).
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Register Keboola Table
+ +
+ {# Q2 radio — Sync mode. (Q1 is implicitly 'synced'; Keboola
+ has no Live mode.) Whole and Custom both map to
+ query_mode='materialized'. #}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ {# Discover/List tables backend currently routes by instance's data_source.type
+ ignoring the `source` query param. Hiding the buttons on non-Keboola instances
+ prevents wrong-shape responses; inputs stay for manual entry. Future fix: make
+ /api/admin/discover-tables accept ?source=keboola and remove this guard. #}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ SELECT against
+ kbc."bucket"."table".
+ Result is materialized to parquet and distributed via
+ da sync.
+
+
+
+
+
+ How often Agnes refreshes the local copy. Examples:
+
+ every 15m, every 6h,
+ daily 03:00, daily 07:00,13:00,18:00 (UTC).
+
+
+
+
+
+
+
+
+
+
+
+ Advanced (optional)
+
+
+
+
+ Comma-separated list. Catalog
+ metadata only — Agnes always does full-overwrite
+ sync; no upsert/dedup. Auto-filled from the Keboola source
+ when available.
+
+
+
+
+
+
+
+
+
+
+ Edit Keboola Table
+ +
+
+
+
+
+
+
+ {# Q2 radio — Sync mode (mirror of Register). #}
+ Slugified id, immutable.
+
+
+
+
+ {# Discover/List tables backend currently routes by instance's data_source.type
+ ignoring the `source` query param. Hiding the buttons on non-Keboola instances
+ prevents wrong-shape responses; inputs stay for manual entry. Future fix: make
+ /api/admin/discover-tables accept ?source=keboola and remove this guard. #}
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ Advanced (optional)
+
+
+
+
+ Comma-separated list. Catalog
+ metadata only — Agnes always does full-overwrite sync.
+
+
+
+
+ Jira tables are populated by webhooks.
+ To register a new Jira webhook integration, see
+ docs/connectors/jira.md.
-
+ {# C3: legacy #registerModal removed. The Phase E #registerBqModal
+ (inside #tab-content-bigquery) and Phase F #registerKeboolaModal
+ (inside #tab-content-keboola) own the Register flows now. The
+ data-source-type marker moved to so DATA_SOURCE_TYPE still
+ has somewhere to read from. #}
-
+
-
-
-
- Register Table
- -
- {% if data_source_type == 'bigquery' %}
- {# BigQuery path — registers a remote view that queries BQ directly. #}
-
-
-
-
-
- BigQuery dataset name (no project prefix — that's read from instance.yaml)
-
-
-
-
- BigQuery table or view name within the dataset
-
-
-
-
- DuckDB view name analysts will use; defaults to the source table
-
-
-
-
-
-
-
-
- Logical grouping for catalog organization
-
-
-
-
- Cron expression. Note: scheduler does not yet evaluate this — see #79; addressed in M3 of #108.
-
-
- {% else %}
- {# Keboola (default) path — fields populated by Discover-tables prefill. #}
- Source check
-
-
-
-
-
-
-
-
-
- Derived from the source table identifier
-
-
-
-
- {# Hidden: the Keboola storage table identifier (the part after the
- bucket prefix in `t.id`, e.g. `company` for `in.c-sfdc.company`).
- `regTableName` is the human-friendly display name and is NOT
- safe to send as `source_table` — see review IMPORTANT 5 in
- PR #119. #}
-
- Source bucket (Keboola: in.c-foo)
-
-
-
-
- How data should be synchronized from the source
-
-
-
-
- Comma-separated list of primary key columns
-
-
-
-
-
-
-
-
- {% endif %}
- Logical grouping for catalog organization
-
-
-
-
-
@@ -965,8 +1408,15 @@
+
- Slugified id, immutable. Source type:
+ —
+
+
+
-
+
+
-
-
+
+
+
Logical grouping for catalog organization (does not affect storage).
@@ -1011,8 +1463,31 @@
Admin Tables - JavaScript
═══════════════════════════════════════════════════════════════ */
+ // ── Tab nav (Phase D) ───────────────────────────────────────
+ function switchTab(tab) {
+ document.querySelectorAll('.tab').forEach(function(b) {
+ b.setAttribute('aria-selected', b.dataset.tab === tab ? 'true' : 'false');
+ });
+ document.querySelectorAll('.tab-content').forEach(function(c) {
+ c.style.display = (c.id === ('tab-content-' + tab)) ? 'block' : 'none';
+ });
+ history.replaceState(null, '', '#' + tab);
+ }
+
+ function getActiveTabFromHash() {
+ var hash = window.location.hash.replace(/^#/, '');
+ if (hash === 'bigquery' || hash === 'keboola' || hash === 'jira') {
+ return hash;
+ }
+ return null;
+ }
+
+ (function initTabFromHash() {
+ var t = getActiveTabFromHash();
+ if (t) switchTab(t);
+ })();
+
// State
- let discoveryData = null;
let registryData = null;
let registryVersion = null;
let currentEditTableId = null;
@@ -1068,283 +1543,634 @@
return div.innerHTML;
}
- // ── Discovery ───────────────────────────────────────────────
-
- function discoverTables() {
- var btn = document.getElementById('discoverBtn');
- var resultsEl = document.getElementById('discoveryResults');
-
- // Loading state
- btn.disabled = true;
- btn.innerHTML = ' Discovering...';
- resultsEl.innerHTML = '
Scanning data source for tables...
';
-
- fetch('/api/admin/discover-tables')
- .then(function(r) {
- if (!r.ok) return r.json().then(function(d) { throw new Error(d.error || 'Discovery failed'); });
- return r.json();
- })
- .then(function(data) {
- discoveryData = data;
- // renderDiscoveryResults handles the per-source shape and
- // surfaces its own success toast (count derived from the
- // actual flat tables list, not whatever `data.total` says).
- renderDiscoveryResults(data);
- })
- .catch(function(err) {
- resultsEl.innerHTML = 'Discovery failed: ' + escapeHtml(err.message) + '
';
- showToast('Discovery failed: ' + err.message, 'error');
- })
- .finally(function() {
- btn.disabled = false;
- btn.innerHTML = ' Discover tables from source';
- });
- }
-
- function renderDiscoveryResults(data) {
- var el = document.getElementById('discoveryResults');
-
- // The discovery API returns a flat {tables: [...]} list; group on
- // the client by `bucket_id` so the rendered accordion still shows
- // bucket → tables. Fall back to the legacy `data.buckets` shape
- // if a future endpoint emits it pre-grouped.
- var buckets;
- if (data.buckets && data.buckets.length) {
- buckets = data.buckets;
- } else if (data.tables && data.tables.length) {
- var byBucket = {};
- data.tables.forEach(function(t) {
- var bid = t.bucket_id || '(ungrouped)';
- if (!byBucket[bid]) {
- byBucket[bid] = {
- bucket_id: bid,
- bucket_name: t.bucket_name || bid,
- tables: [],
- };
- }
- byBucket[bid].tables.push(t);
- });
- buckets = Object.keys(byBucket).sort().map(function(k) { return byBucket[k]; });
- } else {
- buckets = [];
- }
-
- if (!buckets.length) {
- el.innerHTML = 'No tables found in data source
';
- return;
- }
-
- var html = '';
- var totalTables = 0;
- buckets.forEach(function(bucket) {
- var registeredCount = bucket.tables.filter(function(t) { return t.is_registered; }).length;
- totalTables += bucket.tables.length;
-
- html += '';
- html += '';
- html += '
';
- });
-
- el.innerHTML = html;
- // Surface the discovery summary in the toast (count is whatever
- // discovery returned, not whatever the API claimed in `total`).
- showToast('Found ' + totalTables + ' tables in ' + buckets.length + ' buckets', 'success');
- }
-
- function toggleBucket(trigger) {
- var content = trigger.nextElementSibling;
- var isExpanded = trigger.classList.contains('expanded');
-
- if (isExpanded) {
- trigger.classList.remove('expanded');
- content.classList.remove('expanded');
- } else {
- trigger.classList.add('expanded');
- content.classList.add('expanded');
- }
- }
+ // C3: removed dead Discovery panel JS. The global Discovery card +
+ // its #discoverBtn / #discoveryResults DOM hooks were removed when
+ // the per-tab UI landed; per-tab Discover/List datalist helpers live
+ // in the per-source shims further down. The legacy "Register" button
+ // rendered per discovery row also went away — operators register
+ // through the per-tab Register modals.
// ── Registration Modal ──────────────────────────────────────
- // Server-rendered marker so the JS knows whether to drive the Keboola
- // form or the BigQuery form. Single source of truth — stays in sync
- // with the Jinja branch above.
- var DATA_SOURCE_TYPE = document.getElementById('registerModal').dataset.sourceType || 'keboola';
+ // Server-rendered marker so the JS knows the instance's data source
+ // type. Lives on after C3 (previously on the now-removed
+ // #registerModal).
+ var DATA_SOURCE_TYPE = document.body.dataset.sourceType || 'keboola';
- function openRegisterModal(table) {
- if (DATA_SOURCE_TYPE === 'bigquery') {
- // BQ uses a manual-entry form (no discovery panel for BQ in M1).
- // `table` may be partially populated by a future M2 prefill —
- // tolerate either an empty call or a {bucket, source_table, ...}
- // shape from a hypothetical future prefill.
- table = table || {};
- document.getElementById('bqDataset').value = table.bucket || '';
- document.getElementById('bqSourceTable').value = table.source_table || table.name || '';
- document.getElementById('bqViewName').value = table.name || '';
- document.getElementById('bqDescription').value = '';
- document.getElementById('bqFolder').value = '';
- document.getElementById('bqSyncSchedule').value = '';
- var summary = document.getElementById('bqPrecheckSummary');
- if (summary) summary.style.display = 'none';
- } else {
- // Keboola path — fields populated from the discovery click.
- // `table.id` from KeboolaClient.discover_all_tables() is the
- // full storage identifier (e.g. `in.c-sfdc.company`); strip the
- // bucket prefix to get the bare source table name (`company`)
- // that the extractor expects in `kbc."{bucket}"."{source_table}"`.
- // `table.name` is the human-friendly display name — NOT safe to
- // use as the storage identifier. See review IMPORTANT 5 in #119.
- var bucketId = table.bucket_id || (table.bucket && table.bucket.id) || '';
- var fullId = table.id || '';
- var bareSourceTable = '';
- if (fullId && bucketId && fullId.indexOf(bucketId + '.') === 0) {
- bareSourceTable = fullId.substring(bucketId.length + 1);
- } else if (fullId.indexOf('.') >= 0) {
- bareSourceTable = fullId.substring(fullId.lastIndexOf('.') + 1);
- } else {
- bareSourceTable = fullId || table.name || '';
- }
- document.getElementById('regTableId').value = fullId;
- document.getElementById('regTableName').value = table.name || '';
- document.getElementById('regBucket').value = bucketId;
- document.getElementById('regSourceTable').value = bareSourceTable;
- document.getElementById('regStrategy').value = 'full_refresh';
- document.getElementById('regPrimaryKey').value = (table.primary_key || []).join(', ');
- document.getElementById('regDescription').value = '';
- document.getElementById('regFolder').value = '';
+ function openRegisterModal(arg) {
+ // Phase E + F: dispatch by string argument.
+ // 'bigquery' → BQ tab Register button → #registerBqModal.
+ // 'keboola' → Keboola tab Register button → #registerKeboolaModal.
+ if (arg === 'bigquery') {
+ return _openBqRegisterModal({});
}
- document.getElementById('registerSubmitBtn').disabled = false;
- document.getElementById('registerSubmitBtn').textContent = 'Register Table';
- // Reset BQ button handler back to the precheck step in case the
- // operator left it on "Register" from a previous open.
- document.getElementById('registerSubmitBtn').onclick = registerTable;
- document.getElementById('registerModal').classList.add('active');
+ if (arg === 'keboola') {
+ return _openKeboolaTabRegisterModal();
+ }
+ // Fallback when called with no argument — pick by instance type.
+ if (DATA_SOURCE_TYPE === 'bigquery') {
+ return _openBqRegisterModal({});
+ }
+ return _openKeboolaTabRegisterModal();
}
- function closeRegisterModal() {
- document.getElementById('registerModal').classList.remove('active');
+ function _openBqRegisterModal(table) {
+ // BQ uses a manual-entry form (no discovery panel for BQ in M1).
+ // `table` may be partially populated by a future M2 prefill —
+ // tolerate either an empty call or a {bucket, source_table, ...}
+ // shape from a hypothetical future prefill.
+ table = table || {};
+ document.getElementById('bqDataset').value = table.bucket || '';
+ document.getElementById('bqSourceTable').value = table.source_table || table.name || '';
+ document.getElementById('bqViewName').value = table.name || '';
+ document.getElementById('bqDescription').value = '';
+ document.getElementById('bqFolder').value = '';
+ document.getElementById('bqSyncSchedule').value = '';
+ var summary = document.getElementById('bqPrecheckSummary');
+ if (summary) summary.style.display = 'none';
+ var btn = document.getElementById('registerBqSubmitBtn');
+ btn.disabled = false;
+ btn.textContent = 'Register Table';
+ btn.onclick = registerBqTable;
+ document.getElementById('registerBqModal').classList.add('active');
+ }
+
+ // C3: removed dead helpers _openKeboolaRegisterModal /
+ // closeRegisterModal that drove the now-deleted #registerModal.
+ // The Phase F #registerKeboolaModal owns the Keboola flow now.
+
+ function closeRegisterBqModal() {
+ document.getElementById('registerBqModal').classList.remove('active');
+ }
+
+ // ── Keboola tab register modal (Phase F1) ──────────────────────
+
+ function _openKeboolaTabRegisterModal() {
+ // Reset form to defaults each open. Whole mode is the default
+ // (Q2='whole'); the kb-source-table fields are visible.
+ var modal = document.getElementById('registerKeboolaModal');
+ if (!modal) return;
+ var radio = modal.querySelector('input[name="kbSyncMode"][value="whole"]');
+ if (radio) radio.checked = true;
+ ['kbViewName', 'kbBucket', 'kbSourceTable', 'kbSourceQuery',
+ 'kbSyncSchedule', 'kbDescription', 'kbFolder', 'kbPrimaryKey'].forEach(function(id) {
+ var el = document.getElementById(id);
+ if (el) el.value = '';
+ });
+ onKbSyncModeChange(); // apply default visibility
+ var btn = document.getElementById('registerKeboolaSubmitBtn');
+ btn.disabled = false;
+ btn.textContent = 'Register';
+ modal.classList.add('active');
+ }
+
+ function closeRegisterKeboolaModal() {
+ document.getElementById('registerKeboolaModal').classList.remove('active');
+ }
+
+ function _getKbSyncMode() {
+ var el = document.querySelector('input[name="kbSyncMode"]:checked');
+ return el ? el.value : 'whole';
+ }
+
+ function onKbSyncModeChange() {
+ var mode = _getKbSyncMode();
+ document.querySelectorAll('.kb-source-table').forEach(function(el) {
+ el.style.display = (mode === 'whole') ? '' : 'none';
+ });
+ document.querySelectorAll('.kb-source-custom').forEach(function(el) {
+ el.style.display = (mode === 'custom') ? '' : 'none';
+ });
}
function _buildKeboolaPayload() {
- var pk = document.getElementById('regPrimaryKey').value.trim();
- var primaryKey = pk ? pk.split(',').map(function(s) { return s.trim(); }).filter(Boolean) : [];
- // `regSourceTable` carries the bare Keboola storage identifier
- // (e.g. `company`), populated from the discovery row's `t.id`
- // by openRegisterModal. `regTableName` is the display name and
- // can include spaces or non-ASCII chars — never send it as
- // source_table. See review IMPORTANT 5 in #119.
- var sourceTable = (document.getElementById('regSourceTable').value || '').trim();
- if (!sourceTable) {
- // Manual-entry fallback (no discovery row clicked) — fall
- // back to the display name. Validators on the server will
- // reject it if it's not a safe identifier.
- sourceTable = document.getElementById('regTableName').value;
- }
- return {
- // RegisterTableRequest contract: name + source_type + bucket +
- // source_table; no `id`, no `version`, no `dataset` (the field
- // is `bucket`). Pre-fix the modal posted those phantom fields
- // and the API silently dropped them.
- name: document.getElementById('regTableName').value,
+ // Phase F: canonical Keboola payload builder for the Keboola-tab
+ // Register modal. Whole mode synthesizes SELECT * FROM kbc."b"."t";
+ // Custom mode posts the admin-supplied SELECT verbatim. Both map
+ // to query_mode='materialized'.
+ var mode = _getKbSyncMode();
+ var viewName = (document.getElementById('kbViewName').value || '').trim();
+ var bucket = (document.getElementById('kbBucket').value || '').trim();
+ var sourceTable = (document.getElementById('kbSourceTable').value || '').trim();
+ var pk = (document.getElementById('kbPrimaryKey').value || '').trim();
+ var primaryKey = pk
+ ? pk.split(',').map(function(s) { return s.trim(); }).filter(Boolean)
+ : [];
+
+ var common = {
+ name: viewName || sourceTable,
source_type: 'keboola',
- bucket: document.getElementById('regBucket').value,
- source_table: sourceTable,
- query_mode: 'local',
- sync_strategy: document.getElementById('regStrategy').value,
+ query_mode: 'materialized',
primary_key: primaryKey,
- description: document.getElementById('regDescription').value.trim() || null,
- folder: document.getElementById('regFolder').value.trim() || null,
+ sync_schedule: (document.getElementById('kbSyncSchedule').value || '').trim() || null,
+ description: (document.getElementById('kbDescription').value || '').trim() || null,
+ folder: (document.getElementById('kbFolder').value || '').trim() || null,
};
- }
- function _buildBigQueryPayload() {
- var dataset = document.getElementById('bqDataset').value.trim();
- var sourceTable = document.getElementById('bqSourceTable').value.trim();
- var viewName = document.getElementById('bqViewName').value.trim() || sourceTable;
- return {
- name: viewName,
- source_type: 'bigquery',
- bucket: dataset,
- source_table: sourceTable,
- // The server forces these for BQ rows, but we set them explicitly
- // so the network log + audit-log entry reflect the operator's
- // intent rather than a server-side mutation.
- query_mode: 'remote',
- profile_after_sync: false,
- description: document.getElementById('bqDescription').value.trim() || null,
- folder: document.getElementById('bqFolder').value.trim() || null,
- sync_schedule: document.getElementById('bqSyncSchedule').value.trim() || null,
- };
- }
-
- function registerTable() {
- var btn = document.getElementById('registerSubmitBtn');
- btn.disabled = true;
-
- if (DATA_SOURCE_TYPE === 'bigquery') {
- _registerBigQueryTable(btn);
- } else {
- _registerKeboolaTable(btn);
+ if (mode === 'custom') {
+ return Object.assign({}, common, {
+ source_query: (document.getElementById('kbSourceQuery').value || '').trim(),
+ });
}
+ // Whole — synthesize SELECT * FROM kbc."bucket"."table".
+ return Object.assign({}, common, {
+ bucket: bucket,
+ source_table: sourceTable,
+ source_query: 'SELECT * FROM kbc."' + bucket + '"."' + sourceTable + '"',
+ });
}
- function _registerKeboolaTable(btn) {
+ function registerKeboolaTable() {
+ var btn = document.getElementById('registerKeboolaSubmitBtn');
+ btn.disabled = true;
btn.textContent = 'Registering...';
var payload = _buildKeboolaPayload();
fetch('/api/admin/register-table', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
- body: JSON.stringify(payload)
+ body: JSON.stringify(payload),
})
- .then(function(r) {
- if (!r.ok) return r.json().then(function(d) { throw new Error(d.detail || d.error || 'Registration failed'); });
- return r.json();
- })
- .then(function(data) {
- closeRegisterModal();
- showToast('Table registered successfully', 'success');
- loadRegistry();
- if (discoveryData) discoverTables();
- })
- .catch(function(err) {
- showToast('Registration failed: ' + err.message, 'error');
- })
- .finally(function() {
- btn.disabled = false;
- btn.textContent = 'Register Table';
+ .then(function(r) {
+ if (!r.ok) {
+ return r.json().then(function(d) {
+ throw new Error(d.detail || d.error || 'Registration failed');
+ });
+ }
+ return r.json();
+ })
+ .then(function() {
+ closeRegisterKeboolaModal();
+ showToast('Table registered', 'success');
+ loadRegistry();
+ })
+ .catch(function(err) {
+ showToast('' + err.message, 'error');
+ })
+ .finally(function() {
+ btn.disabled = false;
+ btn.textContent = 'Register';
+ });
+ }
+
+ // Discovery shims — the existing /api/admin/discover-tables endpoint
+ // already routes by the instance's data_source.type (returning Keboola
+ // tables when the instance is Keboola-typed); the source=keboola query
+ // param is informational. Hidden behind `data_source_type == 'keboola'`
+ // because on a BQ-typed instance the endpoint would return BQ-shaped
+ // data (wrong shape, confusing); operators fall back to manual entry
+ // for cross-source registration. Future work: extend the endpoint to
+ // accept an explicit ?source= override so secondary-source registration
+ // works in both directions and we can remove this guard.
+ {% if data_source_type == 'keboola' %}
+ function discoverKeboolaBuckets(datalistId) {
+ fetch('/api/admin/discover-tables?source=keboola')
+ .then(function(r) {
+ if (!r.ok) return r.json().then(function(d) {
+ throw new Error(d.detail || d.error || 'Keboola discovery failed');
+ });
+ return r.json();
+ })
+ .then(function(data) {
+ var dl = document.getElementById(datalistId);
+ if (!dl) return;
+ dl.innerHTML = '';
+ // Endpoint may return either {buckets:[...]}, {datasets:[...]}
+ // or {tables:[...]} depending on routing; project to a flat
+ // bucket-id list. Keboola path returns tables → derive uniq
+ // bucket_ids.
+ var buckets = data.buckets || data.datasets;
+ if (!buckets && Array.isArray(data.tables)) {
+ var seen = {};
+ buckets = [];
+ data.tables.forEach(function(t) {
+ var b = t.bucket_id || (t.bucket && t.bucket.id);
+ if (b && !seen[b]) { seen[b] = 1; buckets.push(b); }
+ });
+ }
+ (buckets || []).forEach(function(b) {
+ var o = document.createElement('option');
+ o.value = (typeof b === 'string') ? b : (b.id || b.bucket_id || '');
+ dl.appendChild(o);
+ });
+ showToast('Loaded ' + (dl.children.length) + ' buckets', 'success');
+ })
+ .catch(function(err) {
+ showToast('' + err.message, 'error');
+ });
+ }
+
+ function discoverKeboolaTables(bucketInputId, tablesDatalistId) {
+ var bucketEl = document.getElementById(bucketInputId);
+ var bucket = bucketEl ? (bucketEl.value || '').trim() : '';
+ if (!bucket) {
+ showToast('Fill bucket first', 'error');
+ return;
+ }
+ fetch('/api/admin/discover-tables?source=keboola&bucket=' + encodeURIComponent(bucket))
+ .then(function(r) {
+ if (!r.ok) return r.json().then(function(d) {
+ throw new Error(d.detail || d.error || 'Keboola table discovery failed');
+ });
+ return r.json();
+ })
+ .then(function(data) {
+ var dl = document.getElementById(tablesDatalistId);
+ if (!dl) return;
+ dl.innerHTML = '';
+ var tables = data.tables || [];
+ // Filter to the selected bucket if endpoint didn't.
+ tables.filter(function(t) {
+ var b = t.bucket_id || (t.bucket && t.bucket.id);
+ return !bucket || !b || b === bucket;
+ }).forEach(function(t) {
+ var o = document.createElement('option');
+ var name = (typeof t === 'string') ? t : (t.name || t.id || '');
+ // Strip bucket prefix if present.
+ if (name.indexOf(bucket + '.') === 0) name = name.substring(bucket.length + 1);
+ o.value = name;
+ dl.appendChild(o);
+ });
+ showToast('Loaded ' + dl.children.length + ' tables in ' + bucket, 'success');
+ })
+ .catch(function(err) {
+ showToast('' + err.message, 'error');
+ });
+ }
+
+ // ── Keboola tab edit modal (Phase F2) ──────────────────────────
+
+ function _getEditKbSyncMode() {
+ var el = document.querySelector('input[name="editKbSyncMode"]:checked');
+ return el ? el.value : 'whole';
+ }
+
+ function onEditKbSyncModeChange() {
+ var mode = _getEditKbSyncMode();
+ document.querySelectorAll('.editkb-source-table').forEach(function(el) {
+ el.style.display = (mode === 'whole') ? '' : 'none';
});
+ document.querySelectorAll('.editkb-source-custom').forEach(function(el) {
+ el.style.display = (mode === 'custom') ? '' : 'none';
+ });
+ }
+
+ function _setEditKbRadio(value) {
+ var el = document.querySelector('input[name="editKbSyncMode"][value="' + value + '"]');
+ if (el) el.checked = true;
+ }
+
+ function openEditKeboolaModal(table) {
+ // Populate fields from a registry row. The classic Keboola row may
+ // have query_mode='local' (legacy) or 'materialized' with
+ // source_query. Auto-detect mode: a SELECT * FROM kbc."b"."t"
+ // synthetic SQL → Whole; anything else → Custom.
+ table = table || {};
+ document.getElementById('editKbTableId').value = table.id || '';
+ var bucket = table.bucket || '';
+ var sourceTable = table.source_table || '';
+ var sourceQuery = table.source_query || '';
+ var isAutoSelectStar = false;
+ if (sourceQuery && bucket && sourceTable) {
+ var auto = 'SELECT * FROM kbc."' + bucket + '"."' + sourceTable + '"';
+ isAutoSelectStar = sourceQuery.replace(/\s+/g, ' ').trim() === auto;
+ }
+ var mode = (sourceQuery && !isAutoSelectStar) ? 'custom' : 'whole';
+ _setEditKbRadio(mode);
+ document.getElementById('editKbBucket').value = bucket;
+ document.getElementById('editKbSourceTable').value = sourceTable;
+ document.getElementById('editKbSourceQuery').value = sourceQuery;
+ document.getElementById('editKbSyncSchedule').value = table.sync_schedule || '';
+ document.getElementById('editKbDescription').value = table.description || '';
+ document.getElementById('editKbFolder').value = table.folder || '';
+ document.getElementById('editKbPrimaryKey').value = (table.primary_key || []).join(', ');
+ onEditKbSyncModeChange();
+ var btn = document.getElementById('editKeboolaSubmitBtn');
+ btn.disabled = false;
+ btn.textContent = 'Save Changes';
+ document.getElementById('editKeboolaModal').classList.add('active');
+ }
+
+ function closeEditKeboolaModal() {
+ document.getElementById('editKeboolaModal').classList.remove('active');
+ }
+
+ function _buildKeboolaEditPayload() {
+ var mode = _getEditKbSyncMode();
+ var bucket = (document.getElementById('editKbBucket').value || '').trim();
+ var sourceTable = (document.getElementById('editKbSourceTable').value || '').trim();
+ var pk = (document.getElementById('editKbPrimaryKey').value || '').trim();
+ var primaryKey = pk
+ ? pk.split(',').map(function(s) { return s.trim(); }).filter(Boolean)
+ : [];
+ var common = {
+ query_mode: 'materialized',
+ primary_key: primaryKey,
+ sync_schedule: (document.getElementById('editKbSyncSchedule').value || '').trim() || null,
+ description: (document.getElementById('editKbDescription').value || '').trim() || null,
+ folder: (document.getElementById('editKbFolder').value || '').trim() || null,
+ };
+ if (mode === 'custom') {
+ return Object.assign({}, common, {
+ source_query: (document.getElementById('editKbSourceQuery').value || '').trim(),
+ });
+ }
+ return Object.assign({}, common, {
+ bucket: bucket,
+ source_table: sourceTable,
+ source_query: 'SELECT * FROM kbc."' + bucket + '"."' + sourceTable + '"',
+ });
+ }
+
+ function saveKeboolaTabEdit() {
+ var btn = document.getElementById('editKeboolaSubmitBtn');
+ var tableId = document.getElementById('editKbTableId').value;
+ if (!tableId) {
+ showToast('Missing table id', 'error');
+ return;
+ }
+ btn.disabled = true;
+ btn.textContent = 'Saving...';
+ var payload = _buildKeboolaEditPayload();
+ fetch('/api/admin/registry/' + encodeURIComponent(tableId), {
+ method: 'PUT',
+ headers: { 'Content-Type': 'application/json' },
+ body: JSON.stringify(payload),
+ })
+ .then(function(r) {
+ if (!r.ok) return r.json().then(function(d) {
+ throw new Error(d.detail || d.error || 'Update failed');
+ });
+ return r.json();
+ })
+ .then(function() {
+ closeEditKeboolaModal();
+ showToast('Table updated', 'success');
+ loadRegistry();
+ })
+ .catch(function(err) {
+ showToast('' + err.message, 'error');
+ })
+ .finally(function() {
+ btn.disabled = false;
+ btn.textContent = 'Save Changes';
+ });
+ }
+
+ function prefillFromKeboolaTable(textareaId) {
+ // Edit-modal callers may pass an editKbBucket / editKbSourceTable
+ // pair instead of the Register modal's kbBucket / kbSourceTable.
+ // Detect the modal via the textarea id prefix.
+ var prefix = textareaId.indexOf('editKb') === 0 ? 'editKb' : 'kb';
+ var bucket = (document.getElementById(prefix + 'Bucket').value || '').trim();
+ var sourceTable = (document.getElementById(prefix + 'SourceTable').value || '').trim();
+ if (!bucket || !sourceTable) {
+ showToast('Fill bucket + source table first', 'error');
+ return;
+ }
+ var ta = document.getElementById(textareaId);
+ if (ta.value.trim()) {
+ if (!confirm('Replace existing SQL?')) return;
+ }
+ ta.value = 'SELECT *\nFROM kbc."' + bucket + '"."' + sourceTable + '"\nWHERE -- your filter here';
+ }
+ {% endif %}{# data_source_type == 'keboola' — discover/prefill JS #}
+
+ // C3: removed dead helper _buildKeboolaLegacyPayload — the Phase F
+ // _buildKeboolaPayload (above) replaced it.
+
+ function _getBqAccessMode() {
+ // Q1 radio. Default 'live' if nothing's checked yet (model-validator
+ // safety net for the initial render).
+ var el = document.querySelector('input[name="bqAccessMode"]:checked');
+ return el ? el.value : 'live';
+ }
+
+ function _getBqSyncMode() {
+ // Q2 radio (only meaningful when access mode is 'synced').
+ var el = document.querySelector('input[name="bqSyncMode"]:checked');
+ return el ? el.value : 'whole';
+ }
+
+ function _buildBigQueryPayload() {
+ // Two-question form maps to backend `query_mode`:
+ // live → query_mode='remote' (server auto-detects
+ // BASE TABLE vs VIEW)
+ // synced/whole → query_mode='materialized' (auto SELECT *)
+ // synced/custom → query_mode='materialized' (admin SQL)
+ // The UI never asks "Table vs View" — that's a server-side detail.
+ var accessMode = _getBqAccessMode();
+ var syncMode = _getBqSyncMode();
+ var viewName = document.getElementById('bqViewName').value.trim();
+ var description = document.getElementById('bqDescription').value.trim() || null;
+ var folder = document.getElementById('bqFolder').value.trim() || null;
+ var syncSchedule = document.getElementById('bqSyncSchedule').value.trim() || null;
+
+ if (accessMode === 'synced' && syncMode === 'custom') {
+ return {
+ name: viewName,
+ source_type: 'bigquery',
+ query_mode: 'materialized',
+ source_query: document.getElementById('bqSourceQuery').value.trim(),
+ profile_after_sync: false,
+ description: description,
+ folder: folder,
+ sync_schedule: syncSchedule,
+ };
+ }
+
+ var dataset = document.getElementById('bqDataset').value.trim();
+ var sourceTable = document.getElementById('bqSourceTable').value.trim();
+
+ if (accessMode === 'synced' && syncMode === 'whole') {
+ // Whole-table sync. We don't ship the project to the browser, so
+ // the SQL uses DuckDB three-part syntax against the materialize
+ // session's ATTACH alias. Native BQ dry-run can't parse this form
+ // (DuckDB identifier quoting), so the cost guardrail falls
+ // fail-open with a warning — operator who needs the cap to engage
+ // picks Custom query and writes backtick-quoted native BQ
+ // identifiers.
+ return {
+ name: viewName || sourceTable,
+ source_type: 'bigquery',
+ query_mode: 'materialized',
+ source_query: 'SELECT * FROM bq."' + dataset + '"."' + sourceTable + '"',
+ profile_after_sync: false,
+ description: description,
+ folder: folder,
+ sync_schedule: syncSchedule,
+ };
+ }
+
+ // Live access — server auto-detects BASE TABLE vs VIEW at register
+ // time, so the UI doesn't make the operator pick.
+ return {
+ name: viewName || sourceTable,
+ source_type: 'bigquery',
+ bucket: dataset,
+ source_table: sourceTable,
+ query_mode: 'remote',
+ profile_after_sync: false,
+ description: description,
+ folder: folder,
+ sync_schedule: syncSchedule,
+ };
+ }
+
+ function onBqAccessModeChange() {
+ // Q1: toggle live ↔ synced. Q2 (sync mode) is meaningful only when
+ // synced; default it to 'whole' on first reveal so the form stays
+ // consistent without forcing the operator to click twice.
+ var accessMode = _getBqAccessMode();
+ var liveFields = document.querySelectorAll('.bq-access-live');
+ var syncedFields = document.querySelectorAll('.bq-access-synced');
+ liveFields.forEach(function(el) {
+ el.style.display = (accessMode === 'live') ? '' : 'none';
+ });
+ syncedFields.forEach(function(el) {
+ el.style.display = (accessMode === 'synced') ? '' : 'none';
+ });
+ // Q2 is fresh on first reveal; trigger its handler to apply the
+ // source-table vs source-custom visibility rules.
+ if (accessMode === 'synced') {
+ onBqSyncModeChange();
+ } else {
+ // Live mode: show source-table fields, hide custom-SQL textarea.
+ document.querySelectorAll('.bq-source-table').forEach(function(el) {
+ el.style.display = '';
+ });
+ document.querySelectorAll('.bq-source-custom').forEach(function(el) {
+ el.style.display = 'none';
+ });
+ }
+ }
+
+ function onBqSyncModeChange() {
+ // Q2: toggle whole-table ↔ custom-SQL. Whole reuses the
+ // dataset/source-table inputs (server-side SELECT *), Custom shows
+ // the SQL textarea. Only fires when access mode is already 'synced'.
+ var syncMode = _getBqSyncMode();
+ var tableFields = document.querySelectorAll('.bq-source-table');
+ var customFields = document.querySelectorAll('.bq-source-custom');
+ tableFields.forEach(function(el) {
+ el.style.display = (syncMode === 'whole') ? '' : 'none';
+ });
+ customFields.forEach(function(el) {
+ el.style.display = (syncMode === 'custom') ? '' : 'none';
+ });
+ }
+
+ // The discover / list-tables / prefill helpers are shared between the
+ // Register and Edit modals. The default datalist + input ids match the
+ // Register modal; Edit-modal callers pass their own ids so the
+ // autocomplete populates the right datalist and reads from the right
+ // dataset input.
+
+ function discoverBqDatasets(datalistId) {
+ // GET /api/admin/discover-tables (no `dataset`) → list datasets in
+ // the configured BQ project. Populate the named ';
-
- bucket.tables.forEach(function(table) {
- html += '
';
- html += '';
- html += '
';
- });
-
- html += '';
- html += '
';
- html += '' + escapeHtml(table.name) + '
';
- html += '';
- if (table.columns != null) html += '' + table.columns + ' columns';
- if (table.row_count != null) html += '' + formatNumber(table.row_count) + ' rows';
- if (table.size_bytes != null) html += '' + formatSize(table.size_bytes) + '';
- html += '
';
- html += '';
-
- if (table.is_registered) {
- html += 'Registered';
- } else {
- html += 'Available';
- html += '';
- }
-
- html += '
';
- html += '