506a378c3a
## Summary
Brings the Keboola connector to feature parity with the legacy internal data-analyst's per-table sync strategies. Closes the four documented gaps from the spec branch (`zs/keboola-connector-specs`):
- **Typed parquet** in the legacy SDK extraction path — column types from Keboola Storage metadata (provider cascade `user > ai-metadata-enrichment > keboola.snowflake-transformation`) survive the CSV → parquet roundtrip; invalid date strings (`'0000-00-00'`) and invalid numeric strings (`'Non-Manager'`) become NULL while keeping the column's typed schema. Pre-fix everything was VARCHAR.
- **Incremental sync** via Storage API `changedSince` — opt-in per table; pulls only delta rows, merges into the existing parquet by `primary_key` (drop_duplicates with keep='last'). Cuts daily extraction from O(full table) to O(delta).
- **Partitioned sync** — flat per-partition layout `data/<table>/<key>.parquet` (e.g. `2026_05.parquet`), per-affected-partition merge for daily updates, chunked initial load with 1-day overlap and 2-empty-chunk stop heuristic.
- **`where_filters`** — server-side row filter with date placeholders (`{{today}}`, `{{last_3_months}}`, `{{start_of_3_months_ago}}`, etc.) resolved at sync time. Force the SDK path; reject `incremental + where_filters` combination at API layer (changedSince already filters temporally).
## Architecture
- **Schema migration v25 → v26**: 7 new columns on `table_registry`. Existing `sync_strategy` column reused (pre-v26 it was inert catalog metadata; post-v26 the extractor dispatches off it).
- **Per-table dispatcher** in `extractor.run()` routes to one of `_extract_via_extension` (full_refresh + extension), `_extract_via_legacy` (full_refresh + filters or extension fallback), `extract_incremental`, or `extract_partitioned`.
- **API conflict policy**: `incremental + where_filters` → 422; `partitioned + query_mode='remote'` → 422; `partitioned ⇒ partition_by required`.
- **Admin UI**: third "Direct extract (Storage API)" radio in the Keboola Register / Edit modals, alongside existing "Whole table (extension)" and "Custom SQL". When selected, exposes a v26 sync-strategy panel with conditional fields per strategy.
## Test plan
- [x] **Unit + module** — 134 v26 tests covering migration, repo, parquet_io, where_filters, incremental (compute_changed_since + merge_parquet + extract_incremental E2E), partitioned (key derivation + merge_partition + chunked windows + extract_partitioned E2E), extractor dispatcher, admin API validators, PUT field clearing, registry-shape → dispatcher bridge
- [x] **HTML form structure** — all v26 inputs + visibility classes + JS payload fields verified in rendered template
- [x] **Real Keboola roundtrip** — registered a small test table as `sync_strategy='incremental'` against a test Storage project, triggered two syncs:
- Sync 1: `changedSince=None` → full pull → 9 rows typed parquet
- Sync 2: `changedSince=last_sync - 1d window` → 9 delta rows merged with 9 existing → 9 after dedup on primary_key (PK merge confirmed)
- [x] **Browser UX** — agent-browser session against a local uvicorn: login → admin/tables → register modal → switch radios → verify field visibility per strategy → submit → edit existing row → switch to Direct/Incremental → save → confirm DB persistence
- [x] **Regression** — no regressions in the broader 3252-test suite (3 pre-v26 tests updated for the deprecation-marker removal + schema-version bump; 2 pre-existing environment-sensitive test failures unrelated to this change)
## Bugs caught + fixed during E2E
The browser + real-Keboola roundtrip exposed four bugs the unit tests missed:
1. **JS visibility race** — two competing `forEach` loops set `display=''` then `display='none'` on form elements sharing `kb-strategy-incremental kb-strategy-partitioned` classes (window_days + max_history_days are reused across strategies). Fix: single-pass selector with class-based visibility resolver.
2. **PUT cannot clear field** — pre-v26 `updates = {k: v ... if v is not None}` collapsed "omitted from body" and "sent as null" into the same case, so admin couldn't switch a partitioned row back to full_refresh and have stale `partition_by` clear. Fix: `model_dump(exclude_unset=True)`.
3. **Subprocess DB lock conflict** — `_read_last_sync` reopened `system.duckdb` while the parent server held the write lock (subprocess contract at `app/api/sync.py:_run_sync` line 260). Fix: parent injects `__last_sync__` into table_config before subprocess spawn.
4. **Wrong KBC table_id** — `extract_incremental` / `extract_partitioned` built the Storage API table_id from the registry row's slugified `id` (`circle_inc`) instead of `bucket.source_table` (`in.c-finance.circle`), producing 404s. Fix: prefer `bucket+source_table`; fall back to `id` only when bucket empty.
## Operator notes
- Existing tables stay on `full_refresh` after migration; admins opt individual tables in via `agnes admin register-table --sync-strategy ...`, the Keboola Edit modal, or `POST/PUT /api/admin/registry`.
- `merge_parquet` and `merge_partition` use `pd.concat + drop_duplicates`, loading both existing and delta into pandas RAM. For tables in the multi-million-row range this may OOM — switch to `partitioned` strategy for those (per-partition merge keeps memory bounded). Documented in `### Internal` of the changelog entry.
- Date placeholders are resolved at **sync time**, not register time — a typo'd `{{lasst_week}}` is accepted at register and surfaces only when the next sync runs. By design (rolling windows need late-binding).
## Spec source
The four corresponding plans on the `zs/keboola-connector-specs` branch under `docs/superpowers/plans/2026-05-07-0[1-4]-*.md` capture the design rationale and link back to internal repo references for each subsystem.
<!-- devin-review-badge-begin -->
---
<a href="https://app.devin.ai/review/keboola/agnes-the-ai-analyst/pull/217" target="_blank">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1" alt="Open in Devin Review">
</picture>
</a>
<!-- devin-review-badge-end -->
3440 lines
160 KiB
HTML
3440 lines
160 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en">
|
||
<head>
|
||
<meta charset="UTF-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
<title>Table Management - {{ config.INSTANCE_NAME }}</title>
|
||
{% if not config.THEME_FONT_URL %}
|
||
<link rel="preconnect" href="https://fonts.googleapis.com">
|
||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap" rel="stylesheet">
|
||
{% endif %}
|
||
<link rel="stylesheet" href="{{ url_for('static', filename='style-custom.css') }}">
|
||
<style>
|
||
:root {
|
||
/* Colors - Design System */
|
||
--primary: #0073D1;
|
||
--primary-light: rgba(0, 115, 209, 0.1);
|
||
--text-primary: #1A253C;
|
||
--text-secondary: #6B7280;
|
||
--background: #F5F7FA;
|
||
--surface: #FFFFFF;
|
||
--border: #E5E7EB;
|
||
--border-light: #F3F4F6;
|
||
--success: #10B77F;
|
||
--success-light: rgba(16, 183, 127, 0.1);
|
||
--warning: #F59F0A;
|
||
--error: #EA580C;
|
||
--error-light: rgba(234, 88, 12, 0.1);
|
||
|
||
/* Typography */
|
||
--font-primary: 'Inter', system-ui, sans-serif;
|
||
--font-mono: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace;
|
||
|
||
/* Shadows */
|
||
--shadow-sm: rgba(0, 0, 0, 0.05) 0px 1px 2px 0px;
|
||
--shadow-md: rgba(0, 0, 0, 0.1) 0px 4px 6px -1px;
|
||
}
|
||
|
||
* {
|
||
margin: 0;
|
||
padding: 0;
|
||
box-sizing: border-box;
|
||
}
|
||
|
||
body {
|
||
font-family: var(--font-primary);
|
||
font-size: 14px;
|
||
color: var(--text-primary);
|
||
background: var(--background);
|
||
line-height: 1.5;
|
||
}
|
||
|
||
/* ── Header (dashboard-style) ── */
|
||
.header {
|
||
background: var(--surface);
|
||
border-bottom: 1px solid var(--border);
|
||
padding: 0 32px;
|
||
height: 72px;
|
||
display: flex;
|
||
align-items: center;
|
||
justify-content: space-between;
|
||
position: sticky;
|
||
top: 0;
|
||
z-index: 100;
|
||
}
|
||
|
||
.header-left {
|
||
display: flex;
|
||
align-items: center;
|
||
gap: 16px;
|
||
}
|
||
|
||
.header-back {
|
||
display: flex;
|
||
align-items: center;
|
||
justify-content: center;
|
||
width: 32px;
|
||
height: 32px;
|
||
border-radius: 6px;
|
||
color: var(--text-secondary);
|
||
text-decoration: none;
|
||
transition: all 0.15s ease;
|
||
}
|
||
|
||
.header-back:hover {
|
||
background: var(--border-light);
|
||
color: var(--text-primary);
|
||
}
|
||
|
||
.header-logo-group {
|
||
display: flex;
|
||
flex-direction: column;
|
||
justify-content: center;
|
||
gap: 2px;
|
||
}
|
||
|
||
.header-logo svg {
|
||
display: block;
|
||
}
|
||
|
||
.header-subtitle {
|
||
font-size: 11px;
|
||
font-weight: 500;
|
||
color: var(--text-secondary);
|
||
letter-spacing: 0.4px;
|
||
text-transform: uppercase;
|
||
margin-top: 2px;
|
||
}
|
||
|
||
.header-right {
|
||
font-size: 12px;
|
||
color: var(--text-secondary);
|
||
}
|
||
|
||
/* ── Page Title ── */
|
||
.page-title {
|
||
max-width: 1600px;
|
||
margin: 0 auto;
|
||
padding: 32px 24px 24px;
|
||
}
|
||
|
||
.page-title h1 {
|
||
font-size: 24px;
|
||
font-weight: 600;
|
||
color: var(--text-primary);
|
||
margin-bottom: 4px;
|
||
}
|
||
|
||
.page-title p {
|
||
font-size: 14px;
|
||
color: var(--text-secondary);
|
||
}
|
||
|
||
/* ── Content Layout ── */
|
||
.content {
|
||
max-width: 1600px;
|
||
margin: 0 auto;
|
||
padding: 0 24px 32px;
|
||
display: flex;
|
||
flex-direction: column;
|
||
gap: 24px;
|
||
}
|
||
|
||
/* ── Panel (shared card style) ── */
|
||
.panel {
|
||
background: var(--surface);
|
||
border: 1px solid var(--border);
|
||
border-radius: 12px;
|
||
overflow: hidden;
|
||
box-shadow: var(--shadow-sm);
|
||
}
|
||
|
||
.panel-header {
|
||
display: flex;
|
||
align-items: center;
|
||
justify-content: space-between;
|
||
padding: 20px 24px;
|
||
border-bottom: 1px solid var(--border-light);
|
||
}
|
||
|
||
.panel-header-left {
|
||
display: flex;
|
||
align-items: center;
|
||
gap: 12px;
|
||
}
|
||
|
||
.panel-header-icon {
|
||
width: 36px;
|
||
height: 36px;
|
||
border-radius: 8px;
|
||
display: flex;
|
||
align-items: center;
|
||
justify-content: center;
|
||
flex-shrink: 0;
|
||
background: var(--primary-light);
|
||
}
|
||
|
||
.panel-title {
|
||
font-size: 16px;
|
||
font-weight: 600;
|
||
color: var(--text-primary);
|
||
}
|
||
|
||
.panel-subtitle {
|
||
font-size: 13px;
|
||
color: var(--text-secondary);
|
||
margin-top: 1px;
|
||
}
|
||
|
||
.panel-body {
|
||
padding: 20px 24px;
|
||
}
|
||
|
||
.panel-body-empty {
|
||
padding: 40px 24px;
|
||
text-align: center;
|
||
color: var(--text-secondary);
|
||
font-size: 13px;
|
||
}
|
||
|
||
/* ── Buttons ── */
|
||
.btn {
|
||
display: inline-flex;
|
||
align-items: center;
|
||
gap: 6px;
|
||
padding: 8px 16px;
|
||
border: none;
|
||
border-radius: 8px;
|
||
font-family: var(--font-primary);
|
||
font-size: 13px;
|
||
font-weight: 500;
|
||
cursor: pointer;
|
||
transition: all 0.15s ease;
|
||
white-space: nowrap;
|
||
}
|
||
|
||
.btn-primary {
|
||
background: var(--primary);
|
||
color: white;
|
||
}
|
||
|
||
.btn-primary:hover {
|
||
background: #005FA8;
|
||
}
|
||
|
||
.btn-primary:disabled {
|
||
opacity: 0.6;
|
||
cursor: not-allowed;
|
||
}
|
||
|
||
.btn-secondary {
|
||
background: var(--border-light);
|
||
color: var(--text-primary);
|
||
}
|
||
|
||
.btn-secondary:hover {
|
||
background: var(--border);
|
||
}
|
||
|
||
.btn-danger {
|
||
background: var(--error-light);
|
||
color: var(--error);
|
||
}
|
||
|
||
.btn-danger:hover {
|
||
background: rgba(234, 88, 12, 0.2);
|
||
}
|
||
|
||
.btn-sm {
|
||
padding: 5px 10px;
|
||
font-size: 12px;
|
||
border-radius: 6px;
|
||
}
|
||
|
||
.btn-icon {
|
||
width: 28px;
|
||
height: 28px;
|
||
padding: 0;
|
||
display: flex;
|
||
align-items: center;
|
||
justify-content: center;
|
||
border-radius: 6px;
|
||
background: none;
|
||
border: none;
|
||
cursor: pointer;
|
||
color: var(--text-secondary);
|
||
transition: all 0.15s ease;
|
||
}
|
||
|
||
.btn-icon:hover {
|
||
background: var(--border-light);
|
||
color: var(--text-primary);
|
||
}
|
||
|
||
.btn-icon.danger:hover {
|
||
background: var(--error-light);
|
||
color: var(--error);
|
||
}
|
||
|
||
/* ── Badges ── */
|
||
.badge {
|
||
flex-shrink: 0;
|
||
font-size: 11px;
|
||
font-weight: 500;
|
||
border-radius: 6px;
|
||
padding: 3px 8px;
|
||
white-space: nowrap;
|
||
}
|
||
|
||
.badge-registered {
|
||
color: #065F46;
|
||
background: #D1FAE5;
|
||
}
|
||
|
||
.badge-available {
|
||
color: var(--primary);
|
||
background: var(--primary-light);
|
||
}
|
||
|
||
/* ── Spinner ── */
|
||
.spinner {
|
||
display: inline-block;
|
||
width: 18px;
|
||
height: 18px;
|
||
border: 2px solid var(--border);
|
||
border-top-color: var(--primary);
|
||
border-radius: 50%;
|
||
animation: spin 0.6s linear infinite;
|
||
}
|
||
|
||
@keyframes spin {
|
||
to { transform: rotate(360deg); }
|
||
}
|
||
|
||
.spinner-lg {
|
||
width: 32px;
|
||
height: 32px;
|
||
border-width: 3px;
|
||
}
|
||
|
||
/* ── Loading state ── */
|
||
.loading-state {
|
||
display: flex;
|
||
flex-direction: column;
|
||
align-items: center;
|
||
gap: 12px;
|
||
padding: 40px 24px;
|
||
color: var(--text-secondary);
|
||
font-size: 13px;
|
||
}
|
||
|
||
/* ── Notification toast ── */
|
||
.toast {
|
||
position: fixed;
|
||
top: 84px;
|
||
right: 24px;
|
||
z-index: 200;
|
||
background: var(--surface);
|
||
border: 1px solid var(--border);
|
||
border-radius: 10px;
|
||
box-shadow: var(--shadow-md);
|
||
padding: 12px 16px;
|
||
display: flex;
|
||
align-items: center;
|
||
gap: 10px;
|
||
font-size: 13px;
|
||
transform: translateX(120%);
|
||
transition: transform 0.3s ease;
|
||
max-width: 360px;
|
||
}
|
||
|
||
.toast.visible {
|
||
transform: translateX(0);
|
||
}
|
||
|
||
.toast-success {
|
||
border-left: 3px solid var(--success);
|
||
}
|
||
|
||
.toast-error {
|
||
border-left: 3px solid var(--error);
|
||
}
|
||
|
||
.toast-icon {
|
||
width: 20px;
|
||
height: 20px;
|
||
flex-shrink: 0;
|
||
}
|
||
|
||
/* ── Bucket accordion ── */
|
||
.bucket-group {
|
||
border-top: 1px solid var(--border-light);
|
||
}
|
||
|
||
.bucket-group:first-child {
|
||
border-top: none;
|
||
}
|
||
|
||
.bucket-trigger {
|
||
width: 100%;
|
||
display: flex;
|
||
align-items: center;
|
||
gap: 10px;
|
||
padding: 12px 24px;
|
||
background: none;
|
||
border: none;
|
||
cursor: pointer;
|
||
font-family: var(--font-primary);
|
||
font-size: 14px;
|
||
font-weight: 500;
|
||
color: var(--text-primary);
|
||
text-align: left;
|
||
transition: background 0.1s ease;
|
||
}
|
||
|
||
.bucket-trigger:hover {
|
||
background: var(--border-light);
|
||
}
|
||
|
||
.bucket-chevron {
|
||
width: 16px;
|
||
height: 16px;
|
||
color: var(--text-secondary);
|
||
transition: transform 0.2s ease;
|
||
flex-shrink: 0;
|
||
}
|
||
|
||
.bucket-trigger.expanded .bucket-chevron {
|
||
transform: rotate(90deg);
|
||
}
|
||
|
||
.bucket-count {
|
||
font-size: 11px;
|
||
font-weight: 500;
|
||
color: var(--text-secondary);
|
||
background: var(--border-light);
|
||
padding: 1px 7px;
|
||
border-radius: 9999px;
|
||
margin-left: auto;
|
||
}
|
||
|
||
.bucket-content {
|
||
display: none;
|
||
}
|
||
|
||
.bucket-content.expanded {
|
||
display: block;
|
||
}
|
||
|
||
/* ── Table row (discovery results) ── */
|
||
.table-item {
|
||
display: flex;
|
||
align-items: center;
|
||
padding: 10px 24px 10px 50px;
|
||
border-top: 1px solid var(--border-light);
|
||
gap: 12px;
|
||
transition: background 0.1s ease;
|
||
}
|
||
|
||
.table-item:hover {
|
||
background: rgba(243, 244, 246, 0.5);
|
||
}
|
||
|
||
.table-item-info {
|
||
flex: 1;
|
||
min-width: 0;
|
||
}
|
||
|
||
.table-item-name {
|
||
font-weight: 500;
|
||
font-family: var(--font-mono);
|
||
font-size: 13px;
|
||
color: var(--text-primary);
|
||
}
|
||
|
||
.table-item-meta {
|
||
font-size: 12px;
|
||
color: var(--text-secondary);
|
||
margin-top: 1px;
|
||
display: flex;
|
||
gap: 12px;
|
||
flex-wrap: wrap;
|
||
}
|
||
|
||
.table-item-actions {
|
||
display: flex;
|
||
align-items: center;
|
||
gap: 8px;
|
||
flex-shrink: 0;
|
||
}
|
||
|
||
/* ── Registry table ── */
|
||
.registry-table {
|
||
width: 100%;
|
||
border-collapse: collapse;
|
||
table-layout: fixed;
|
||
}
|
||
|
||
.registry-table .col-description {
|
||
overflow: hidden;
|
||
display: -webkit-box;
|
||
-webkit-line-clamp: 2;
|
||
-webkit-box-orient: vertical;
|
||
overflow-wrap: anywhere;
|
||
line-height: 1.4;
|
||
color: var(--text-secondary);
|
||
}
|
||
|
||
.registry-table th {
|
||
text-align: left;
|
||
font-size: 11px;
|
||
font-weight: 600;
|
||
color: var(--text-secondary);
|
||
text-transform: uppercase;
|
||
letter-spacing: 0.4px;
|
||
padding: 10px 16px;
|
||
border-bottom: 1px solid var(--border);
|
||
background: var(--background);
|
||
}
|
||
|
||
.registry-table td {
|
||
padding: 12px 16px;
|
||
font-size: 13px;
|
||
border-bottom: 1px solid var(--border-light);
|
||
vertical-align: middle;
|
||
}
|
||
|
||
.registry-table tr:last-child td {
|
||
border-bottom: none;
|
||
}
|
||
|
||
.registry-table tr:hover td {
|
||
background: rgba(243, 244, 246, 0.5);
|
||
}
|
||
|
||
.registry-table .col-id {
|
||
font-family: var(--font-mono);
|
||
font-size: 12px;
|
||
max-width: 220px;
|
||
overflow: hidden;
|
||
text-overflow: ellipsis;
|
||
white-space: nowrap;
|
||
}
|
||
|
||
.registry-table .col-actions {
|
||
width: 120px;
|
||
min-width: 120px;
|
||
white-space: nowrap;
|
||
vertical-align: top;
|
||
}
|
||
|
||
/* ── Registry table — wide layout ── */
|
||
.registry-table .col-mode {
|
||
width: 100px;
|
||
}
|
||
|
||
.registry-table .col-source {
|
||
width: 200px;
|
||
font-family: var(--font-mono);
|
||
font-size: 12px;
|
||
overflow: hidden;
|
||
text-overflow: ellipsis;
|
||
white-space: nowrap;
|
||
color: var(--text-secondary);
|
||
}
|
||
|
||
.registry-table .col-pk {
|
||
width: 100px;
|
||
overflow: hidden;
|
||
text-overflow: ellipsis;
|
||
white-space: nowrap;
|
||
}
|
||
|
||
.registry-table .col-schedule {
|
||
width: 100px;
|
||
font-size: 12px;
|
||
color: var(--text-secondary);
|
||
}
|
||
|
||
.registry-table .col-folder {
|
||
width: 120px;
|
||
}
|
||
|
||
.registry-table .col-registered {
|
||
width: 160px;
|
||
font-size: 11px;
|
||
line-height: 1.4;
|
||
overflow: hidden;
|
||
}
|
||
|
||
.registry-table .col-registered .registered-by {
|
||
color: var(--text-primary);
|
||
overflow: hidden;
|
||
text-overflow: ellipsis;
|
||
white-space: nowrap;
|
||
}
|
||
|
||
.registry-table .col-registered .registered-at {
|
||
color: var(--text-secondary);
|
||
}
|
||
|
||
.registry-table .col-status {
|
||
width: 40px;
|
||
text-align: center;
|
||
}
|
||
|
||
.mode-badge {
|
||
display: inline-block;
|
||
padding: 2px 8px;
|
||
border-radius: 4px;
|
||
font-size: 11px;
|
||
font-weight: 600;
|
||
}
|
||
|
||
.mode-badge.mode-local {
|
||
background: var(--success-light);
|
||
color: var(--success);
|
||
}
|
||
|
||
.mode-badge.mode-remote {
|
||
background: var(--primary-light);
|
||
color: var(--primary);
|
||
}
|
||
|
||
.mode-badge.mode-materialized {
|
||
background: rgba(139, 92, 246, 0.1);
|
||
color: #8B5CF6;
|
||
}
|
||
|
||
.folder-badge {
|
||
display: inline-block;
|
||
padding: 1px 6px;
|
||
border-radius: 3px;
|
||
background: var(--background);
|
||
border: 1px solid var(--border);
|
||
font-size: 11px;
|
||
color: var(--text-secondary);
|
||
}
|
||
|
||
/* ── Modal overlay ── */
|
||
.modal-overlay {
|
||
display: none;
|
||
position: fixed;
|
||
top: 0;
|
||
left: 0;
|
||
right: 0;
|
||
bottom: 0;
|
||
background: rgba(0, 0, 0, 0.5);
|
||
z-index: 1000;
|
||
padding: 40px 24px;
|
||
overflow-y: auto;
|
||
}
|
||
|
||
.modal-overlay.active {
|
||
display: flex;
|
||
align-items: flex-start;
|
||
justify-content: center;
|
||
}
|
||
|
||
.modal {
|
||
max-width: 560px;
|
||
width: 100%;
|
||
background: var(--surface);
|
||
border-radius: 12px;
|
||
box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3);
|
||
overflow: hidden;
|
||
}
|
||
|
||
.modal-header {
|
||
display: flex;
|
||
justify-content: space-between;
|
||
align-items: center;
|
||
padding: 20px 24px;
|
||
border-bottom: 1px solid var(--border);
|
||
background: var(--background);
|
||
}
|
||
|
||
.modal-header h2 {
|
||
font-size: 18px;
|
||
font-weight: 600;
|
||
}
|
||
|
||
.modal-close {
|
||
width: 32px;
|
||
height: 32px;
|
||
border: none;
|
||
background: none;
|
||
cursor: pointer;
|
||
border-radius: 6px;
|
||
display: flex;
|
||
align-items: center;
|
||
justify-content: center;
|
||
color: var(--text-secondary);
|
||
transition: all 0.15s;
|
||
}
|
||
|
||
.modal-close:hover {
|
||
background: var(--border-light);
|
||
color: var(--text-primary);
|
||
}
|
||
|
||
.modal-body {
|
||
padding: 24px;
|
||
}
|
||
|
||
.modal-footer {
|
||
display: flex;
|
||
justify-content: flex-end;
|
||
gap: 10px;
|
||
padding: 16px 24px;
|
||
border-top: 1px solid var(--border);
|
||
background: var(--background);
|
||
}
|
||
|
||
/* ── Form ── */
|
||
.form-group {
|
||
margin-bottom: 18px;
|
||
}
|
||
|
||
.form-group:last-child {
|
||
margin-bottom: 0;
|
||
}
|
||
|
||
.form-label {
|
||
display: block;
|
||
font-size: 13px;
|
||
font-weight: 500;
|
||
color: var(--text-primary);
|
||
margin-bottom: 6px;
|
||
}
|
||
|
||
.form-label .optional {
|
||
font-weight: 400;
|
||
color: var(--text-secondary);
|
||
font-size: 12px;
|
||
}
|
||
|
||
.form-input,
|
||
.form-select,
|
||
.form-textarea {
|
||
width: 100%;
|
||
padding: 8px 12px;
|
||
border: 1px solid var(--border);
|
||
border-radius: 8px;
|
||
font-family: var(--font-primary);
|
||
font-size: 13px;
|
||
color: var(--text-primary);
|
||
background: var(--surface);
|
||
transition: border-color 0.15s;
|
||
}
|
||
|
||
.form-input:focus,
|
||
.form-select:focus,
|
||
.form-textarea:focus {
|
||
outline: none;
|
||
border-color: var(--primary);
|
||
box-shadow: 0 0 0 3px rgba(0, 115, 209, 0.1);
|
||
}
|
||
|
||
.form-input[readonly] {
|
||
background: var(--border-light);
|
||
color: var(--text-secondary);
|
||
cursor: not-allowed;
|
||
}
|
||
|
||
.form-textarea {
|
||
min-height: 80px;
|
||
resize: vertical;
|
||
}
|
||
|
||
.form-select {
|
||
cursor: pointer;
|
||
appearance: none;
|
||
background-image: url("data:image/svg+xml,%3Csvg width='12' height='12' viewBox='0 0 24 24' fill='none' stroke='%236B7280' stroke-width='2' stroke-linecap='round' stroke-linejoin='round' xmlns='http://www.w3.org/2000/svg'%3E%3Cpolyline points='6 9 12 15 18 9'/%3E%3C/svg%3E");
|
||
background-repeat: no-repeat;
|
||
background-position: right 12px center;
|
||
padding-right: 32px;
|
||
}
|
||
|
||
.form-hint {
|
||
font-size: 12px;
|
||
color: var(--text-secondary);
|
||
margin-top: 4px;
|
||
}
|
||
|
||
/* ── Footer ── */
|
||
.footer {
|
||
text-align: center;
|
||
padding: 24px;
|
||
color: var(--text-secondary);
|
||
font-size: 12px;
|
||
}
|
||
|
||
.footer a {
|
||
color: var(--primary);
|
||
text-decoration: none;
|
||
}
|
||
|
||
.footer a:hover {
|
||
text-decoration: underline;
|
||
}
|
||
|
||
/* ── Responsive ── */
|
||
@media (max-width: 640px) {
|
||
.header {
|
||
padding: 0 16px;
|
||
}
|
||
|
||
.page-title {
|
||
padding: 24px 16px 16px;
|
||
}
|
||
|
||
.content {
|
||
padding: 0 16px 24px;
|
||
}
|
||
|
||
.panel-header {
|
||
flex-direction: column;
|
||
align-items: flex-start;
|
||
gap: 12px;
|
||
}
|
||
|
||
.table-item {
|
||
padding-left: 24px;
|
||
flex-wrap: wrap;
|
||
}
|
||
|
||
.table-item-meta {
|
||
flex-direction: column;
|
||
gap: 4px;
|
||
}
|
||
|
||
.registry-table .col-id {
|
||
max-width: 160px;
|
||
}
|
||
|
||
.modal {
|
||
margin: 16px;
|
||
}
|
||
}
|
||
|
||
/* ── Tab nav (Phase D) ── */
|
||
.tab-nav {
|
||
display: flex;
|
||
gap: 4px;
|
||
border-bottom: 1px solid var(--border);
|
||
margin-bottom: 16px;
|
||
}
|
||
.tab {
|
||
padding: 8px 16px;
|
||
background: transparent;
|
||
border: 0;
|
||
cursor: pointer;
|
||
font-family: inherit;
|
||
font-size: inherit;
|
||
color: var(--text-secondary);
|
||
}
|
||
.tab[aria-selected="true"] {
|
||
border-bottom: 2px solid var(--primary);
|
||
color: var(--text-primary);
|
||
font-weight: 600;
|
||
}
|
||
.tab-content {
|
||
padding: 16px 0;
|
||
}
|
||
.tab-header {
|
||
display: flex;
|
||
justify-content: space-between;
|
||
align-items: center;
|
||
margin-bottom: 16px;
|
||
}
|
||
.tab-actions {
|
||
display: flex;
|
||
justify-content: flex-end;
|
||
margin-bottom: 16px;
|
||
}
|
||
</style>
|
||
{% include '_theme.html' %}
|
||
</head>
|
||
<body data-source-type="{{ data_source_type }}">
|
||
|
||
<!-- ═══════════════ HEADER ═══════════════ -->
|
||
{% include '_app_header.html' %}
|
||
|
||
<!-- ═══════════════ PAGE TITLE ═══════════════ -->
|
||
<div class="page-title">
|
||
<h1>Table Management</h1>
|
||
<p>Discover, register, and manage data tables from your source</p>
|
||
</div>
|
||
|
||
<!-- ═══════════════ CONTENT ═══════════════ -->
|
||
<div class="content">
|
||
|
||
<section id="cacheWarmupCard" class="card" style="margin-bottom: 20px;">
|
||
<header class="card-header" style="display: flex; justify-content: space-between; align-items: center;">
|
||
<h2>Cache freshness</h2>
|
||
<button class="btn btn-secondary" id="cacheWarmupRunBtn" onclick="cacheWarmupRun()">
|
||
Re-warm all
|
||
</button>
|
||
</header>
|
||
<div class="card-body">
|
||
<div id="cacheWarmupProgress" style="margin-bottom: 8px;">
|
||
<span id="cacheWarmupSummary">Loading…</span>
|
||
</div>
|
||
<progress id="cacheWarmupBar" max="100" value="0" style="width: 100%; display: none;"></progress>
|
||
<details style="margin-top: 8px;">
|
||
<summary style="cursor: pointer; user-select: none;">Show log</summary>
|
||
<pre id="cacheWarmupLog" style="background: #0a0a0a; color: #dcdcdc; font-family: ui-monospace, Menlo, monospace; font-size: 12px; padding: 8px; max-height: 240px; overflow-y: auto; margin-top: 8px; border-radius: 4px;"></pre>
|
||
</details>
|
||
</div>
|
||
</section>
|
||
|
||
{# Phase D: tab-split scaffold. Per-connector tabs (BigQuery /
|
||
Keboola / Jira) replace the single mixed form. Each tab has its
|
||
own Register button + listing div + (later) form modals. The
|
||
initial active tab matches data_source.type from instance.yaml;
|
||
the operator can still switch tabs to manage a secondary source.
|
||
|
||
Phase E moves the BQ form into #tab-content-bigquery; Phase F
|
||
builds the Keboola form inside #tab-content-keboola. For now
|
||
the existing Jinja-branched panels below stay in place. #}
|
||
{% set initial_tab = data_source_type if data_source_type in ['bigquery', 'keboola', 'jira'] else 'keboola' %}
|
||
|
||
<nav class="tab-nav" role="tablist">
|
||
<button data-tab="bigquery" aria-selected="{{ 'true' if initial_tab == 'bigquery' else 'false' }}" class="tab" onclick="switchTab('bigquery')">BigQuery</button>
|
||
<button data-tab="keboola" aria-selected="{{ 'true' if initial_tab == 'keboola' else 'false' }}" class="tab" onclick="switchTab('keboola')">Keboola</button>
|
||
<button data-tab="jira" aria-selected="{{ 'true' if initial_tab == 'jira' else 'false' }}" class="tab" onclick="switchTab('jira')">Jira</button>
|
||
</nav>
|
||
|
||
<section id="tab-content-bigquery" class="tab-content"
|
||
style="display: {% if initial_tab == 'bigquery' %}block{% else %}none{% endif %};">
|
||
<div class="tab-actions">
|
||
<button id="bqRegisterBtn" class="btn btn-primary"
|
||
onclick="openRegisterModal('bigquery')">Register BigQuery table</button>
|
||
</div>
|
||
<div id="bqTableListing"></div>
|
||
|
||
<!-- ── BigQuery Register Modal (Phase E relocation) ── -->
|
||
<div class="modal-overlay" id="registerBqModal">
|
||
<div class="modal">
|
||
<div class="modal-header">
|
||
<h2>Register BigQuery Table</h2>
|
||
<button class="modal-close" onclick="closeRegisterBqModal()">
|
||
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
|
||
<line x1="18" y1="6" x2="6" y2="18"/>
|
||
<line x1="6" y1="6" x2="18" y2="18"/>
|
||
</svg>
|
||
</button>
|
||
</div>
|
||
<div class="modal-body">
|
||
{# Two orthogonal questions: (1) live vs synced, (2) when synced,
|
||
whole table vs custom SQL. Visibility classes:
|
||
bq-access-live — only when accessMode='live'
|
||
bq-access-synced — only when accessMode='synced'
|
||
bq-source-table — only when accessMode='live' OR
|
||
(accessMode='synced' AND syncMode='whole')
|
||
bq-source-custom — only when accessMode='synced' AND syncMode='custom'
|
||
Backend payload: live → query_mode='remote'; synced/whole →
|
||
query_mode='materialized' with auto-built SELECT *; synced/custom
|
||
→ query_mode='materialized' with admin SQL. Server auto-detects
|
||
BASE TABLE vs VIEW at register time, so the UI doesn't ask. #}
|
||
<div class="form-group">
|
||
<label class="form-label">How should analysts access this data?</label>
|
||
<div class="bq-access-radio-group" style="display:flex; gap:12px; margin-top:6px;">
|
||
<label style="flex:1; padding:12px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="bqAccessMode" value="live" checked onchange="onBqAccessModeChange()">
|
||
<strong>Live from BigQuery</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Each analyst query goes straight to BQ. Always current.
|
||
Latency ≈ seconds; 0 disk on the analyst machine; cost =
|
||
bytes scanned per query. Best for huge tables or when
|
||
freshness matters.
|
||
</div>
|
||
</label>
|
||
<label style="flex:1; padding:12px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="bqAccessMode" value="synced" onchange="onBqAccessModeChange()">
|
||
<strong>Synced locally</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Agnes runs a SELECT on a schedule and ships a parquet
|
||
to analysts. Analyst-side latency <100 ms; disk =
|
||
snapshot size. Best when analysts hit the same data
|
||
often and speed beats freshness.
|
||
</div>
|
||
</label>
|
||
</div>
|
||
</div>
|
||
<div class="form-group bq-access-synced" style="display:none;">
|
||
<label class="form-label">What to sync?</label>
|
||
<div class="bq-sync-radio-group" style="display:flex; gap:12px; margin-top:6px;">
|
||
<label style="flex:1; padding:12px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="bqSyncMode" value="whole" checked onchange="onBqSyncModeChange()">
|
||
<strong>Whole table</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Agnes runs <code>SELECT *</code> automatically. No SQL
|
||
required. Disk + sync cost = full table size.
|
||
</div>
|
||
</label>
|
||
<label style="flex:1; padding:12px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="bqSyncMode" value="custom" onchange="onBqSyncModeChange()">
|
||
<strong>Custom query</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
You write the SELECT — filter, project, or aggregate
|
||
before the sync. Cuts disk + cost; cap via
|
||
<code>max_bytes_per_materialize</code> guardrail.
|
||
</div>
|
||
</label>
|
||
</div>
|
||
</div>
|
||
<div class="form-group bq-source-table">
|
||
<label class="form-label" for="bqDataset">
|
||
Dataset
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="discoverBqDatasets()" style="float:right; margin-top:-3px;">
|
||
Discover
|
||
</button>
|
||
</label>
|
||
<input type="text" class="form-input" id="bqDataset" list="bqDatasetList" placeholder="e.g. analytics">
|
||
<datalist id="bqDatasetList"></datalist>
|
||
<div class="form-hint">BigQuery dataset name (no project prefix — read from instance.yaml).
|
||
Click <strong>Discover</strong> to populate the autocomplete from the BQ project's dataset list.</div>
|
||
</div>
|
||
<div class="form-group bq-source-table">
|
||
<label class="form-label" for="bqSourceTable">
|
||
Source Table / View
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="discoverBqTables()" style="float:right; margin-top:-3px;">
|
||
List tables
|
||
</button>
|
||
</label>
|
||
<input type="text" class="form-input" id="bqSourceTable" list="bqTableList" placeholder="e.g. orders">
|
||
<datalist id="bqTableList"></datalist>
|
||
<div class="form-hint">Table or view name within the dataset. Click
|
||
<strong>List tables</strong> after filling Dataset to populate autocomplete.
|
||
<br><strong>Live access:</strong> BASE TABLEs query via
|
||
<code>bq."dataset"."table"</code> (Storage Read API; predicate pushdown).
|
||
VIEWs and MATERIALIZED_VIEWs query via the BQ jobs API (full-scan estimate;
|
||
cost-guarded by <code>bq_max_scan_bytes</code>).
|
||
<code>agnes query --remote</code> works for both.
|
||
<br><strong>Synced access:</strong> handles both table and view transparently
|
||
— the scheduler runs <code>SELECT *</code> through the jobs API and writes a
|
||
parquet.</div>
|
||
</div>
|
||
<div class="form-group bq-source-custom" style="display:none;">
|
||
<label class="form-label" for="bqSourceQuery">
|
||
SQL
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="prefillFromTable()" style="float:right; margin-top:-3px;"
|
||
title="Prefill SELECT * FROM `project.dataset.table` so you only edit the WHERE / projection">
|
||
Use table as base
|
||
</button>
|
||
</label>
|
||
<textarea class="form-textarea" id="bqSourceQuery" rows="8"
|
||
placeholder="SELECT date, SUM(revenue) AS revenue FROM `project.dataset.orders` WHERE date >= DATE_SUB(CURRENT_DATE(), INTERVAL 90 DAY) GROUP BY 1"></textarea>
|
||
<div class="form-hint">
|
||
SELECT statement, no trailing semicolon. Native BQ identifiers
|
||
(<code>`project.dataset.table`</code>) recommended — DuckDB three-part
|
||
names like <code>bq."ds"."t"</code> work for the COPY but disable the
|
||
cost guardrail's BQ dry-run.
|
||
</div>
|
||
</div>
|
||
<div class="form-group">
|
||
<label class="form-label" for="bqViewName">View Name</label>
|
||
<input type="text" class="form-input" id="bqViewName" placeholder="orders_90d">
|
||
<div class="form-hint">Name analysts use to query the data (e.g.
|
||
<code>SELECT * FROM orders_90d</code>). Required for Custom query; defaults
|
||
to the source table for the other modes.</div>
|
||
</div>
|
||
<div class="form-group">
|
||
<label class="form-label" for="bqDescription">Description <span class="optional">(optional)</span></label>
|
||
<textarea class="form-textarea" id="bqDescription" placeholder="Brief description of the table contents..."></textarea>
|
||
</div>
|
||
<div class="form-group">
|
||
<label class="form-label" for="bqFolder">Folder <span class="optional">(optional)</span></label>
|
||
<input type="text" class="form-input" id="bqFolder" placeholder="e.g. crm, finance, marketing">
|
||
<div class="form-hint">Logical grouping for catalog organization</div>
|
||
</div>
|
||
<div class="form-group bq-access-synced" style="display:none;">
|
||
<label class="form-label" for="bqSyncSchedule">Sync Schedule <span class="optional">(optional, default <code>every 1h</code>)</span></label>
|
||
<input type="text" class="form-input" id="bqSyncSchedule" placeholder="every 6h">
|
||
<div class="form-hint">
|
||
How often Agnes refreshes the local copy. Examples:
|
||
<code>every 15m</code>, <code>every 6h</code>,
|
||
<code>daily 03:00</code>, <code>daily 07:00,13:00,18:00</code> (UTC).
|
||
</div>
|
||
</div>
|
||
<div class="form-group" id="bqPrecheckSummary" style="display:none;">
|
||
<div class="form-label">Source check</div>
|
||
<div class="form-hint" id="bqPrecheckSummaryText"></div>
|
||
</div>
|
||
</div>
|
||
<div class="modal-footer">
|
||
<button class="btn btn-secondary" onclick="closeRegisterBqModal()">Cancel</button>
|
||
<button class="btn btn-primary" id="registerBqSubmitBtn" onclick="registerBqTable()">
|
||
Register Table
|
||
</button>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<!-- ── BigQuery Edit Modal (C2 — physically inside the BQ tab,
|
||
mirror of #registerBqModal placement) ── -->
|
||
<div class="modal-overlay" id="editBqModal">
|
||
<div class="modal">
|
||
<div class="modal-header">
|
||
<h2>Edit BigQuery Table</h2>
|
||
<button class="modal-close" onclick="closeEditBqModal()">
|
||
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
|
||
<line x1="18" y1="6" x2="6" y2="18"/>
|
||
<line x1="6" y1="6" x2="18" y2="18"/>
|
||
</svg>
|
||
</button>
|
||
</div>
|
||
<div class="modal-body">
|
||
<div class="form-group">
|
||
<label class="form-label" for="editBqTableId">Table ID</label>
|
||
<input type="text" class="form-input" id="editBqTableId" readonly>
|
||
<div class="form-hint">Slugified id, immutable. Source type:
|
||
<strong id="editBqSourceTypeBadge">bigquery</strong></div>
|
||
</div>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label">How should analysts access this data?
|
||
<a href="docs/admin/query-modes.md" target="_blank" title="When to use which mode" style="margin-left: 6px; text-decoration: none; cursor: help;">?</a>
|
||
</label>
|
||
<div style="display:flex; gap:12px; margin-top:6px;">
|
||
<label style="flex:1; padding:10px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="editBqAccessMode" value="live" onchange="onEditBqAccessModeChange()">
|
||
<strong>Live from BigQuery</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Each query goes to BQ. No local copy.
|
||
</div>
|
||
</label>
|
||
<label style="flex:1; padding:10px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="editBqAccessMode" value="synced" onchange="onEditBqAccessModeChange()">
|
||
<strong>Synced locally</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Scheduled SELECT → parquet, queried locally.
|
||
</div>
|
||
</label>
|
||
</div>
|
||
<div class="form-hint" id="editBqModeWarning" style="display:none;
|
||
color:#EA580C;background:rgba(234,88,12,.08);padding:8px;border-radius:6px;margin-top:8px;">
|
||
<!-- Filled by onEditBqAccessModeChange() when switching
|
||
modes on an existing row — warns about parquet
|
||
drop / scheduling impact. -->
|
||
</div>
|
||
</div>
|
||
<div class="form-group bq-edit-access-synced" style="display:none;">
|
||
<label class="form-label">What to sync?</label>
|
||
<div style="display:flex; gap:12px; margin-top:6px;">
|
||
<label style="flex:1; padding:10px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="editBqSyncMode" value="whole" onchange="onEditBqSyncModeChange()">
|
||
<strong>Whole table</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
<code>SELECT *</code> on a schedule.
|
||
</div>
|
||
</label>
|
||
<label style="flex:1; padding:10px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="editBqSyncMode" value="custom" onchange="onEditBqSyncModeChange()">
|
||
<strong>Custom query</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Filter / aggregate before sync.
|
||
</div>
|
||
</label>
|
||
</div>
|
||
</div>
|
||
<div class="form-group bq-edit-source-table" style="display:none;">
|
||
<label class="form-label" for="editBqDataset">
|
||
Dataset
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="discoverBqDatasets('editBqDatasetList')"
|
||
style="float:right; margin-top:-3px;">
|
||
Discover
|
||
</button>
|
||
</label>
|
||
<input type="text" class="form-input" id="editBqDataset"
|
||
list="editBqDatasetList" placeholder="e.g. analytics">
|
||
<datalist id="editBqDatasetList"></datalist>
|
||
</div>
|
||
<div class="form-group bq-edit-source-table" style="display:none;">
|
||
<label class="form-label" for="editBqSourceTable">
|
||
Source Table / View
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="discoverBqTables('editBqDataset', 'editBqTableList')"
|
||
style="float:right; margin-top:-3px;">
|
||
List tables
|
||
</button>
|
||
</label>
|
||
<input type="text" class="form-input" id="editBqSourceTable"
|
||
list="editBqTableList" placeholder="e.g. orders">
|
||
<datalist id="editBqTableList"></datalist>
|
||
<div class="form-hint">Table or view name within the dataset.
|
||
<br><strong>Live access:</strong> BASE TABLEs query via
|
||
<code>bq."dataset"."table"</code> (Storage Read API; predicate pushdown).
|
||
VIEWs and MATERIALIZED_VIEWs query via the BQ jobs API (full-scan estimate;
|
||
cost-guarded by <code>bq_max_scan_bytes</code>).
|
||
<code>agnes query --remote</code> works for both.
|
||
<br><strong>Synced access:</strong> handles both transparently — the
|
||
scheduler runs <code>SELECT *</code> through the jobs API and writes a
|
||
parquet.</div>
|
||
</div>
|
||
<div class="form-group bq-edit-source-custom" style="display:none;">
|
||
<label class="form-label" for="editBqSourceQuery">
|
||
SQL
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="prefillFromTable('editBqSourceQuery')"
|
||
style="float:right; margin-top:-3px;"
|
||
title="Prefill SELECT * FROM `project.dataset.table` so you only edit the WHERE / projection">
|
||
Use table as base
|
||
</button>
|
||
</label>
|
||
<textarea class="form-textarea" id="editBqSourceQuery" rows="8"></textarea>
|
||
<div class="form-hint">SELECT statement, no trailing semicolon. Native BQ
|
||
identifiers recommended for the cost guardrail to engage.</div>
|
||
</div>
|
||
<div class="form-group bq-edit-access-synced" style="display:none;">
|
||
<label class="form-label" for="editBqSyncSchedule">Sync Schedule
|
||
<span class="optional">(optional)</span></label>
|
||
<input type="text" class="form-input" id="editBqSyncSchedule" placeholder="every 6h">
|
||
<div class="form-hint">How often Agnes refreshes the local copy.
|
||
<code>every 15m</code>, <code>every 6h</code>,
|
||
<code>daily 03:00</code> (UTC).</div>
|
||
</div>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label" for="editBqDescription">Description <span class="optional">(optional)</span></label>
|
||
<textarea class="form-textarea" id="editBqDescription" placeholder="Brief description of the table contents..."></textarea>
|
||
</div>
|
||
<div class="form-group">
|
||
<label class="form-label" for="editBqFolder">Folder <span class="optional">(optional)</span></label>
|
||
<input type="text" class="form-input" id="editBqFolder" placeholder="e.g. crm, finance, marketing">
|
||
<div class="form-hint">Logical grouping for catalog organization (does not affect storage).</div>
|
||
</div>
|
||
</div>
|
||
<div class="modal-footer">
|
||
<button class="btn btn-secondary" onclick="closeEditBqModal()">Cancel</button>
|
||
<button class="btn btn-primary" id="editBqSubmitBtn" onclick="saveBqTabEdit()">
|
||
Save Changes
|
||
</button>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</section>
|
||
|
||
<section id="tab-content-keboola" class="tab-content"
|
||
style="display: {% if initial_tab == 'keboola' %}block{% else %}none{% endif %};">
|
||
<div class="tab-actions">
|
||
<button id="kbRegisterBtn" class="btn btn-primary"
|
||
onclick="openRegisterModal('keboola')">Register Keboola table</button>
|
||
</div>
|
||
<div id="kbTableListing"></div>
|
||
|
||
<!-- ── Keboola Register Modal (Phase F1) ── -->
|
||
<div class="modal-overlay" id="registerKeboolaModal">
|
||
<div class="modal">
|
||
<div class="modal-header">
|
||
<h2>Register Keboola Table</h2>
|
||
<button class="modal-close" onclick="closeRegisterKeboolaModal()">
|
||
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
|
||
<line x1="18" y1="6" x2="6" y2="18"/>
|
||
<line x1="6" y1="6" x2="18" y2="18"/>
|
||
</svg>
|
||
</button>
|
||
</div>
|
||
<div class="modal-body">
|
||
{# Three sync-mode radios:
|
||
- whole / custom → query_mode='materialized' (DuckDB Keboola
|
||
extension; whole synthesizes SELECT *, custom uses admin SQL)
|
||
- direct → query_mode='local' (Storage API SDK, supports
|
||
v26 sync strategies: incremental/partitioned + where_filters) #}
|
||
<div class="form-group">
|
||
<label class="form-label">What to sync?</label>
|
||
<div class="bq-sync-radio-group" style="display:flex; gap:12px; margin-top:6px; flex-wrap:wrap;">
|
||
<label style="flex:1; min-width:200px; padding:12px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="kbSyncMode" value="whole" checked onchange="onKbSyncModeChange()">
|
||
<strong>Whole table (extension)</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
DuckDB Keboola extension pulls the full table on
|
||
each tick. Fastest path; full overwrite each run.
|
||
</div>
|
||
</label>
|
||
<label style="flex:1; min-width:200px; padding:12px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="kbSyncMode" value="direct" onchange="onKbSyncModeChange()">
|
||
<strong>Direct extract (Storage API)</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Storage API SDK. Supports incremental sync
|
||
(changedSince + PK merge), partitioned files,
|
||
and server-side <code>where_filters</code>.
|
||
</div>
|
||
</label>
|
||
<label style="flex:1; min-width:200px; padding:12px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="kbSyncMode" value="custom" onchange="onKbSyncModeChange()">
|
||
<strong>Custom SQL</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Pre-aggregate or filter with your own SELECT
|
||
(e.g. last 30 days only, per-day rollup).
|
||
</div>
|
||
</label>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label" for="kbViewName">View name (analyst-visible)</label>
|
||
<input type="text" class="form-input" id="kbViewName"
|
||
placeholder="e.g. orders_recent">
|
||
</div>
|
||
|
||
{# Discover/List tables backend currently routes by instance's data_source.type
|
||
ignoring the `source` query param. Hiding the buttons on non-Keboola instances
|
||
prevents wrong-shape responses; inputs stay for manual entry. Future fix: make
|
||
/api/admin/discover-tables accept ?source=keboola and remove this guard. #}
|
||
<div class="form-group kb-source-table">
|
||
<label class="form-label" for="kbBucket">
|
||
Bucket
|
||
{% if data_source_type == 'keboola' %}
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="discoverKeboolaBuckets('kbBucketList')"
|
||
style="float:right; margin-top:-3px;">Discover</button>
|
||
{% endif %}
|
||
</label>
|
||
<input type="text" class="form-input" id="kbBucket"
|
||
list="kbBucketList" placeholder="e.g. in.c-sales">
|
||
<datalist id="kbBucketList"></datalist>
|
||
</div>
|
||
<div class="form-group kb-source-table">
|
||
<label class="form-label" for="kbSourceTable">
|
||
Source Table
|
||
{% if data_source_type == 'keboola' %}
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="discoverKeboolaTables('kbBucket', 'kbTableList')"
|
||
style="float:right; margin-top:-3px;">List tables</button>
|
||
{% endif %}
|
||
</label>
|
||
<input type="text" class="form-input" id="kbSourceTable"
|
||
list="kbTableList" placeholder="e.g. orders">
|
||
<datalist id="kbTableList"></datalist>
|
||
</div>
|
||
<div class="form-group kb-source-custom" style="display:none;">
|
||
<label class="form-label" for="kbSourceQuery">
|
||
SQL
|
||
{% if data_source_type == 'keboola' %}
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="prefillFromKeboolaTable('kbSourceQuery')"
|
||
style="float:right; margin-top:-3px;"
|
||
title="Prefill SELECT * FROM kbc.bucket.table so you only edit the WHERE / projection">
|
||
Use table as base
|
||
</button>
|
||
{% endif %}
|
||
</label>
|
||
<textarea class="form-textarea" id="kbSourceQuery" rows="8"></textarea>
|
||
<div class="form-hint">SELECT against <code>kbc."bucket"."table"</code>.
|
||
Result is materialized to parquet and distributed via
|
||
<code>agnes pull</code>.</div>
|
||
</div>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label" for="kbSyncSchedule">Sync Schedule
|
||
<span class="optional">(optional, default <code>every 1h</code>)</span></label>
|
||
<input type="text" class="form-input" id="kbSyncSchedule" placeholder="every 6h">
|
||
<div class="form-hint">
|
||
How often Agnes refreshes the local copy. Examples:
|
||
<code>every 15m</code>, <code>every 6h</code>,
|
||
<code>daily 03:00</code>, <code>daily 07:00,13:00,18:00</code> (UTC).
|
||
</div>
|
||
</div>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label" for="kbDescription">Description
|
||
<span class="optional">(optional)</span></label>
|
||
<textarea class="form-textarea" id="kbDescription"
|
||
placeholder="Brief description of the table contents..."></textarea>
|
||
</div>
|
||
<div class="form-group">
|
||
<label class="form-label" for="kbFolder">Folder
|
||
<span class="optional">(optional)</span></label>
|
||
<input type="text" class="form-input" id="kbFolder"
|
||
placeholder="e.g. crm, finance, marketing">
|
||
</div>
|
||
|
||
<details class="form-group">
|
||
<summary>Advanced (optional)</summary>
|
||
<div class="form-group" style="margin-top:8px;">
|
||
<label class="form-label" for="kbPrimaryKey">Primary Key</label>
|
||
<input type="text" class="form-input" id="kbPrimaryKey"
|
||
placeholder="e.g. id">
|
||
<div class="form-hint">Comma-separated list. Required for
|
||
Direct extract → Incremental (used as the dedup key on
|
||
delta merge). Auto-filled from the Keboola source when
|
||
available.</div>
|
||
</div>
|
||
</details>
|
||
|
||
<!-- v26 Direct-extract sync-strategy panel — visible only when
|
||
"Direct extract (Storage API)" is selected. Field visibility
|
||
within the panel further branches on the sync_strategy
|
||
dropdown (incremental / partitioned). -->
|
||
<div class="form-group kb-direct-only" style="display:none; padding:12px; border:1px solid var(--border); border-radius:8px; background:var(--background);">
|
||
<h3 style="margin:0 0 12px 0; font-size:14px;">Direct extract — sync strategy</h3>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label" for="kbStrategy">Sync strategy</label>
|
||
<select class="form-select" id="kbStrategy" onchange="onKbStrategyChange()">
|
||
<option value="full_refresh">Full refresh — pull entire table each tick</option>
|
||
<option value="incremental">Incremental — pull rows changed since last sync</option>
|
||
<option value="partitioned">Partitioned — per-partition files, per-month/day/year</option>
|
||
</select>
|
||
</div>
|
||
|
||
<div class="form-group kb-strategy-incremental kb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="kbIncrementalWindowDays">Incremental window (days)
|
||
<span class="optional">(default 7)</span></label>
|
||
<input type="number" class="form-input" id="kbIncrementalWindowDays" min="0" placeholder="7">
|
||
<div class="form-hint">Backtrack window applied to last_sync timestamp on each tick.
|
||
Higher = more reliable on late-arriving rows; lower = less data per tick.</div>
|
||
</div>
|
||
<div class="form-group kb-strategy-incremental kb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="kbMaxHistoryDays">Max history (days)
|
||
<span class="optional">(first sync only, default unbounded)</span></label>
|
||
<input type="number" class="form-input" id="kbMaxHistoryDays" min="1" placeholder="365">
|
||
<div class="form-hint">Cap on how far back the first-ever sync goes. Multi-year tables
|
||
without this can OOM at write — set 90/180/365 for safety.</div>
|
||
</div>
|
||
|
||
<div class="form-group kb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="kbPartitionBy">Partition by column <strong>(required)</strong></label>
|
||
<input type="text" class="form-input" id="kbPartitionBy" placeholder="e.g. event_date">
|
||
<div class="form-hint">Date / timestamp column whose value drives the partition key.
|
||
Rows with NULL or unparseable values are dropped (logged warning).</div>
|
||
</div>
|
||
<div class="form-group kb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="kbPartitionGranularity">Granularity</label>
|
||
<select class="form-select" id="kbPartitionGranularity">
|
||
<option value="month">Month — YYYY_MM.parquet (default)</option>
|
||
<option value="day">Day — YYYY_MM_DD.parquet</option>
|
||
<option value="year">Year — YYYY.parquet</option>
|
||
</select>
|
||
</div>
|
||
<div class="form-group kb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="kbInitialLoadChunkDays">Initial-load chunk size (days)
|
||
<span class="optional">(default 30)</span></label>
|
||
<input type="number" class="form-input" id="kbInitialLoadChunkDays" min="1" placeholder="30">
|
||
<div class="form-hint">First-sync chunked load step. Smaller = more API calls, less
|
||
memory per chunk. Larger = fewer calls, more memory.</div>
|
||
</div>
|
||
|
||
<div class="form-group kb-strategy-not-incremental" style="display:none;">
|
||
<label class="form-label" for="kbWhereFilters">Where filters
|
||
<span class="optional">(JSON array, optional)</span></label>
|
||
<textarea class="form-textarea" id="kbWhereFilters" rows="6"
|
||
placeholder='[{"column": "date", "operator": "ge", "values": ["{{last_3_months}}"]}]'></textarea>
|
||
<div class="form-hint">
|
||
Server-side row filter. Operators: <code>eq, ne, gt, ge, lt, le</code>.
|
||
Date placeholders resolved at sync time:
|
||
<code>{{ '{{today}}' }}</code>,
|
||
<code>{{ '{{last_week}}' }}</code>,
|
||
<code>{{ '{{last_month}}' }}</code>,
|
||
<code>{{ '{{last_2_months}}' }}</code>,
|
||
<code>{{ '{{last_3_months}}' }}</code>,
|
||
<code>{{ '{{last_6_months}}' }}</code>,
|
||
<code>{{ '{{last_year}}' }}</code>,
|
||
<code>{{ '{{last_2_years}}' }}</code>,
|
||
<code>{{ '{{start_of_3_months_ago}}' }}</code>.
|
||
Not compatible with Incremental strategy.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="modal-footer">
|
||
<button class="btn btn-secondary" onclick="closeRegisterKeboolaModal()">Cancel</button>
|
||
<button class="btn btn-primary" id="registerKeboolaSubmitBtn"
|
||
onclick="registerKeboolaTable()">Register</button>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<!-- ── Keboola Edit Modal (Phase F2) ── -->
|
||
<div class="modal-overlay" id="editKeboolaModal">
|
||
<div class="modal">
|
||
<div class="modal-header">
|
||
<h2>Edit Keboola Table</h2>
|
||
<button class="modal-close" onclick="closeEditKeboolaModal()">
|
||
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
|
||
<line x1="18" y1="6" x2="6" y2="18"/>
|
||
<line x1="6" y1="6" x2="18" y2="18"/>
|
||
</svg>
|
||
</button>
|
||
</div>
|
||
<div class="modal-body">
|
||
<div class="form-group">
|
||
<label class="form-label" for="editKbTableId">Table ID</label>
|
||
<input type="text" class="form-input" id="editKbTableId" readonly>
|
||
<div class="form-hint">Slugified id, immutable.</div>
|
||
</div>
|
||
|
||
{# Three sync-mode radios (mirror of Register). #}
|
||
<div class="form-group">
|
||
<label class="form-label">What to sync?</label>
|
||
<div class="bq-sync-radio-group" style="display:flex; gap:12px; margin-top:6px; flex-wrap:wrap;">
|
||
<label style="flex:1; min-width:200px; padding:10px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="editKbSyncMode" value="whole"
|
||
onchange="onEditKbSyncModeChange()">
|
||
<strong>Whole table (extension)</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
DuckDB Keboola extension; full overwrite each tick.
|
||
</div>
|
||
</label>
|
||
<label style="flex:1; min-width:200px; padding:10px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="editKbSyncMode" value="direct"
|
||
onchange="onEditKbSyncModeChange()">
|
||
<strong>Direct extract (Storage API)</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Storage API SDK. Supports incremental, partitioned,
|
||
<code>where_filters</code>.
|
||
</div>
|
||
</label>
|
||
<label style="flex:1; min-width:200px; padding:10px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="editKbSyncMode" value="custom"
|
||
onchange="onEditKbSyncModeChange()">
|
||
<strong>Custom SQL</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Pre-aggregate or filter with your own SELECT.
|
||
</div>
|
||
</label>
|
||
</div>
|
||
</div>
|
||
|
||
{# Discover/List tables backend currently routes by instance's data_source.type
|
||
ignoring the `source` query param. Hiding the buttons on non-Keboola instances
|
||
prevents wrong-shape responses; inputs stay for manual entry. Future fix: make
|
||
/api/admin/discover-tables accept ?source=keboola and remove this guard. #}
|
||
<div class="form-group editkb-source-table">
|
||
<label class="form-label" for="editKbBucket">
|
||
Bucket
|
||
{% if data_source_type == 'keboola' %}
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="discoverKeboolaBuckets('editKbBucketList')"
|
||
style="float:right; margin-top:-3px;">Discover</button>
|
||
{% endif %}
|
||
</label>
|
||
<input type="text" class="form-input" id="editKbBucket"
|
||
list="editKbBucketList" placeholder="e.g. in.c-sales">
|
||
<datalist id="editKbBucketList"></datalist>
|
||
</div>
|
||
<div class="form-group editkb-source-table">
|
||
<label class="form-label" for="editKbSourceTable">
|
||
Source Table
|
||
{% if data_source_type == 'keboola' %}
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="discoverKeboolaTables('editKbBucket', 'editKbTableList')"
|
||
style="float:right; margin-top:-3px;">List tables</button>
|
||
{% endif %}
|
||
</label>
|
||
<input type="text" class="form-input" id="editKbSourceTable"
|
||
list="editKbTableList" placeholder="e.g. orders">
|
||
<datalist id="editKbTableList"></datalist>
|
||
</div>
|
||
<div class="form-group editkb-source-custom" style="display:none;">
|
||
<label class="form-label" for="editKbSourceQuery">
|
||
SQL
|
||
{% if data_source_type == 'keboola' %}
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="prefillFromKeboolaTable('editKbSourceQuery')"
|
||
style="float:right; margin-top:-3px;"
|
||
title="Prefill SELECT * FROM kbc.bucket.table so you only edit the WHERE / projection">
|
||
Use table as base
|
||
</button>
|
||
{% endif %}
|
||
</label>
|
||
<textarea class="form-textarea" id="editKbSourceQuery" rows="8"></textarea>
|
||
</div>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label" for="editKbSyncSchedule">Sync Schedule
|
||
<span class="optional">(optional)</span></label>
|
||
<input type="text" class="form-input" id="editKbSyncSchedule" placeholder="every 6h">
|
||
</div>
|
||
<div class="form-group">
|
||
<label class="form-label" for="editKbDescription">Description
|
||
<span class="optional">(optional)</span></label>
|
||
<textarea class="form-textarea" id="editKbDescription"></textarea>
|
||
</div>
|
||
<div class="form-group">
|
||
<label class="form-label" for="editKbFolder">Folder
|
||
<span class="optional">(optional)</span></label>
|
||
<input type="text" class="form-input" id="editKbFolder">
|
||
</div>
|
||
|
||
<details class="form-group">
|
||
<summary>Advanced (optional)</summary>
|
||
<div class="form-group" style="margin-top:8px;">
|
||
<label class="form-label" for="editKbPrimaryKey">Primary Key</label>
|
||
<input type="text" class="form-input" id="editKbPrimaryKey"
|
||
placeholder="e.g. id">
|
||
<div class="form-hint">Comma-separated list. Required for
|
||
Direct extract → Incremental.</div>
|
||
</div>
|
||
</details>
|
||
|
||
<!-- v26 Direct-extract sync-strategy panel — mirror of Register. -->
|
||
<div class="form-group editkb-direct-only" style="display:none; padding:12px; border:1px solid var(--border); border-radius:8px; background:var(--background);">
|
||
<h3 style="margin:0 0 12px 0; font-size:14px;">Direct extract — sync strategy</h3>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label" for="editKbStrategy">Sync strategy</label>
|
||
<select class="form-select" id="editKbStrategy" onchange="onEditKbStrategyChange()">
|
||
<option value="full_refresh">Full refresh — pull entire table each tick</option>
|
||
<option value="incremental">Incremental — pull rows changed since last sync</option>
|
||
<option value="partitioned">Partitioned — per-partition files, per-month/day/year</option>
|
||
</select>
|
||
</div>
|
||
|
||
<div class="form-group editkb-strategy-incremental editkb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="editKbIncrementalWindowDays">Incremental window (days)
|
||
<span class="optional">(default 7)</span></label>
|
||
<input type="number" class="form-input" id="editKbIncrementalWindowDays" min="0" placeholder="7">
|
||
</div>
|
||
<div class="form-group editkb-strategy-incremental editkb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="editKbMaxHistoryDays">Max history (days)
|
||
<span class="optional">(first sync only)</span></label>
|
||
<input type="number" class="form-input" id="editKbMaxHistoryDays" min="1" placeholder="365">
|
||
</div>
|
||
|
||
<div class="form-group editkb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="editKbPartitionBy">Partition by column <strong>(required)</strong></label>
|
||
<input type="text" class="form-input" id="editKbPartitionBy" placeholder="e.g. event_date">
|
||
</div>
|
||
<div class="form-group editkb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="editKbPartitionGranularity">Granularity</label>
|
||
<select class="form-select" id="editKbPartitionGranularity">
|
||
<option value="month">Month (default)</option>
|
||
<option value="day">Day</option>
|
||
<option value="year">Year</option>
|
||
</select>
|
||
</div>
|
||
<div class="form-group editkb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="editKbInitialLoadChunkDays">Initial-load chunk size (days)
|
||
<span class="optional">(default 30)</span></label>
|
||
<input type="number" class="form-input" id="editKbInitialLoadChunkDays" min="1" placeholder="30">
|
||
</div>
|
||
|
||
<div class="form-group editkb-strategy-not-incremental" style="display:none;">
|
||
<label class="form-label" for="editKbWhereFilters">Where filters
|
||
<span class="optional">(JSON array, optional)</span></label>
|
||
<textarea class="form-textarea" id="editKbWhereFilters" rows="6"
|
||
placeholder='[{"column": "date", "operator": "ge", "values": ["{{last_3_months}}"]}]'></textarea>
|
||
<div class="form-hint">
|
||
Operators: <code>eq, ne, gt, ge, lt, le</code>. Date placeholders
|
||
(<code>{{ '{{today}}' }}</code>, <code>{{ '{{last_3_months}}' }}</code>, etc.) resolved at
|
||
sync time. Not compatible with Incremental.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="modal-footer">
|
||
<button class="btn btn-secondary" onclick="closeEditKeboolaModal()">Cancel</button>
|
||
<button class="btn btn-primary" id="editKeboolaSubmitBtn"
|
||
onclick="saveKeboolaTabEdit()">Save Changes</button>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</section>
|
||
|
||
<section id="tab-content-jira" class="tab-content"
|
||
style="display: {% if initial_tab == 'jira' %}block{% else %}none{% endif %};">
|
||
<p class="hint" style="margin-bottom: 16px;">Jira tables are populated by webhooks.
|
||
To register a new Jira webhook integration, see
|
||
<code>docs/connectors/jira.md</code>.</p>
|
||
<div id="jiraTableListing"></div>
|
||
</section>
|
||
|
||
{# Legacy out-of-tab panels (BQ Register card, Keboola Discovery card,
|
||
shared Registered Tables wrapper) removed — each tab now owns its
|
||
own header (with Register button) and listing div. The Refresh
|
||
action is implicit: registration / edit / delete flows already
|
||
call loadRegistry(), which re-renders all three per-tab listings. #}
|
||
|
||
</div>
|
||
|
||
{# C3: legacy #registerModal removed. The Phase E #registerBqModal
|
||
(inside #tab-content-bigquery) and Phase F #registerKeboolaModal
|
||
(inside #tab-content-keboola) own the Register flows now. The
|
||
data-source-type marker moved to <body> so DATA_SOURCE_TYPE still
|
||
has somewhere to read from. #}
|
||
|
||
<!-- ═══════════════ EDIT MODAL (legacy fallback — Keboola-only fields
|
||
remaining; the BQ Edit modal moved into #tab-content-bigquery as
|
||
#editBqModal in C2; the Keboola Edit modal is #editKeboolaModal
|
||
in #tab-content-keboola) ═══════════════ -->
|
||
<div class="modal-overlay" id="editModal">
|
||
<div class="modal">
|
||
<div class="modal-header">
|
||
<h2>Edit Table</h2>
|
||
<button class="modal-close" onclick="closeEditModal()">
|
||
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
|
||
<line x1="18" y1="6" x2="6" y2="18"/>
|
||
<line x1="6" y1="6" x2="18" y2="18"/>
|
||
</svg>
|
||
</button>
|
||
</div>
|
||
<div class="modal-body">
|
||
<div class="form-group">
|
||
<label class="form-label" for="editTableId">Table ID</label>
|
||
<input type="text" class="form-input" id="editTableId" readonly>
|
||
<div class="form-hint">Slugified id, immutable. Source type:
|
||
<strong id="editSourceTypeBadge">—</strong></div>
|
||
</div>
|
||
|
||
<!-- Keboola/Jira fallback fields. The richer Keboola modal
|
||
lives at #editKeboolaModal; this remains the catch-all
|
||
for any source_type that's neither bigquery nor keboola
|
||
(e.g. jira). -->
|
||
<div class="form-group keboola-edit-only">
|
||
<label class="form-label" for="editStrategy">Sync Strategy</label>
|
||
<select class="form-select" id="editStrategy">
|
||
<option value="full_refresh">Full Refresh</option>
|
||
<option value="incremental">Incremental</option>
|
||
<option value="partitioned">Partitioned</option>
|
||
</select>
|
||
</div>
|
||
<div class="form-group keboola-edit-only">
|
||
<label class="form-label" for="editPrimaryKey">Primary Key</label>
|
||
<input type="text" class="form-input" id="editPrimaryKey" placeholder="e.g. id">
|
||
</div>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label" for="editDescription">Description <span class="optional">(optional)</span></label>
|
||
<textarea class="form-textarea" id="editDescription" placeholder="Brief description of the table contents..."></textarea>
|
||
</div>
|
||
<div class="form-group">
|
||
<label class="form-label" for="editFolder">Folder <span class="optional">(optional)</span></label>
|
||
<input type="text" class="form-input" id="editFolder" placeholder="e.g. crm, finance, marketing">
|
||
<div class="form-hint">Logical grouping for catalog organization (does not affect storage).</div>
|
||
</div>
|
||
</div>
|
||
<div class="modal-footer">
|
||
<button class="btn btn-secondary" onclick="closeEditModal()">Cancel</button>
|
||
<button class="btn btn-primary" id="editSubmitBtn" onclick="saveTableEdit()">
|
||
Save Changes
|
||
</button>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<!-- ═══════════════ TOAST ═══════════════ -->
|
||
<div class="toast" id="toast">
|
||
<div class="toast-icon" id="toastIcon"></div>
|
||
<span id="toastMessage"></span>
|
||
</div>
|
||
|
||
<!-- ═══════════════ FOOTER ═══════════════ -->
|
||
<footer class="footer">
|
||
<a href="{{ url_for('dashboard') }}">Back to Dashboard</a>
|
||
</footer>
|
||
<script>
|
||
/* ═══════════════════════════════════════════════════════════════
|
||
Admin Tables - JavaScript
|
||
═══════════════════════════════════════════════════════════════ */
|
||
|
||
// ── Tab nav (Phase D) ───────────────────────────────────────
|
||
function switchTab(tab) {
|
||
document.querySelectorAll('.tab').forEach(function(b) {
|
||
b.setAttribute('aria-selected', b.dataset.tab === tab ? 'true' : 'false');
|
||
});
|
||
document.querySelectorAll('.tab-content').forEach(function(c) {
|
||
c.style.display = (c.id === ('tab-content-' + tab)) ? 'block' : 'none';
|
||
});
|
||
history.replaceState(null, '', '#' + tab);
|
||
}
|
||
|
||
function getActiveTabFromHash() {
|
||
var hash = window.location.hash.replace(/^#/, '');
|
||
if (hash === 'bigquery' || hash === 'keboola' || hash === 'jira') {
|
||
return hash;
|
||
}
|
||
return null;
|
||
}
|
||
|
||
(function initTabFromHash() {
|
||
var t = getActiveTabFromHash();
|
||
if (t) switchTab(t);
|
||
})();
|
||
|
||
// State
|
||
let registryData = null;
|
||
let registryVersion = null;
|
||
let currentEditTableId = null;
|
||
|
||
// ── Toast notification ──────────────────────────────────────
|
||
|
||
function showToast(message, type) {
|
||
var toast = document.getElementById('toast');
|
||
var icon = document.getElementById('toastIcon');
|
||
var msg = document.getElementById('toastMessage');
|
||
|
||
toast.className = 'toast toast-' + type;
|
||
msg.textContent = message;
|
||
|
||
if (type === 'success') {
|
||
icon.innerHTML = '<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="#10B77F" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M22 11.08V12a10 10 0 1 1-5.93-9.14"/><polyline points="22 4 12 14.01 9 11.01"/></svg>';
|
||
} else {
|
||
icon.innerHTML = '<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="#EA580C" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10"/><line x1="15" y1="9" x2="9" y2="15"/><line x1="9" y1="9" x2="15" y2="15"/></svg>';
|
||
}
|
||
|
||
// Show
|
||
requestAnimationFrame(function() {
|
||
toast.classList.add('visible');
|
||
});
|
||
|
||
// Hide after 4 seconds
|
||
setTimeout(function() {
|
||
toast.classList.remove('visible');
|
||
}, 4000);
|
||
}
|
||
|
||
// ── Format helpers ──────────────────────────────────────────
|
||
|
||
function formatNumber(n) {
|
||
if (n == null) return '-';
|
||
if (n >= 1000000) return (n / 1000000).toFixed(1) + 'M';
|
||
if (n >= 1000) return n.toLocaleString();
|
||
return String(n);
|
||
}
|
||
|
||
function formatSize(bytes) {
|
||
if (bytes == null || bytes === 0) return '-';
|
||
if (bytes >= 1073741824) return (bytes / 1073741824).toFixed(1) + ' GB';
|
||
if (bytes >= 1048576) return (bytes / 1048576).toFixed(1) + ' MB';
|
||
if (bytes >= 1024) return (bytes / 1024).toFixed(1) + ' KB';
|
||
return bytes + ' B';
|
||
}
|
||
|
||
function escapeHtml(str) {
|
||
if (!str) return '';
|
||
var div = document.createElement('div');
|
||
div.textContent = str;
|
||
return div.innerHTML;
|
||
}
|
||
|
||
// Defensive normalization for descriptions registered via shell-quoting
|
||
// tooling that injected literal backslash escapes (e.g. `Don\'t`, `\n`).
|
||
// Mirrors _unescape_shell_quoting in app/api/admin.py — applied at render
|
||
// time so already-stored corrupt rows still display readably.
|
||
function unescapeShellQuoting(s) {
|
||
if (!s) return s;
|
||
// Order matters: protect real backslashes via NUL sentinel first,
|
||
// unescape the well-known sequences, then restore real backslashes.
|
||
return s
|
||
.replace(/\\\\/g, ' ')
|
||
.replace(/\\n/g, '\n')
|
||
.replace(/\\r/g, '\r')
|
||
.replace(/\\t/g, '\t')
|
||
.replace(/\\'/g, "'")
|
||
.replace(/\\"/g, '"')
|
||
.replace(/ /g, '\\');
|
||
}
|
||
|
||
// C3: removed dead Discovery panel JS. The global Discovery card +
|
||
// its #discoverBtn / #discoveryResults DOM hooks were removed when
|
||
// the per-tab UI landed; per-tab Discover/List datalist helpers live
|
||
// in the per-source shims further down. The legacy "Register" button
|
||
// rendered per discovery row also went away — operators register
|
||
// through the per-tab Register modals.
|
||
|
||
// ── Registration Modal ──────────────────────────────────────
|
||
|
||
// Server-rendered marker so the JS knows the instance's data source
|
||
// type. Lives on <body> after C3 (previously on the now-removed
|
||
// #registerModal).
|
||
var DATA_SOURCE_TYPE = document.body.dataset.sourceType || 'keboola';
|
||
|
||
function openRegisterModal(arg) {
|
||
// Phase E + F: dispatch by string argument.
|
||
// 'bigquery' → BQ tab Register button → #registerBqModal.
|
||
// 'keboola' → Keboola tab Register button → #registerKeboolaModal.
|
||
if (arg === 'bigquery') {
|
||
return _openBqRegisterModal({});
|
||
}
|
||
if (arg === 'keboola') {
|
||
return _openKeboolaTabRegisterModal();
|
||
}
|
||
// Fallback when called with no argument — pick by instance type.
|
||
if (DATA_SOURCE_TYPE === 'bigquery') {
|
||
return _openBqRegisterModal({});
|
||
}
|
||
return _openKeboolaTabRegisterModal();
|
||
}
|
||
|
||
function _openBqRegisterModal(table) {
|
||
// BQ uses a manual-entry form (no discovery panel for BQ in M1).
|
||
// `table` may be partially populated by a future M2 prefill —
|
||
// tolerate either an empty call or a {bucket, source_table, ...}
|
||
// shape from a hypothetical future prefill.
|
||
table = table || {};
|
||
document.getElementById('bqDataset').value = table.bucket || '';
|
||
document.getElementById('bqSourceTable').value = table.source_table || table.name || '';
|
||
document.getElementById('bqViewName').value = table.name || '';
|
||
document.getElementById('bqDescription').value = '';
|
||
document.getElementById('bqFolder').value = '';
|
||
document.getElementById('bqSyncSchedule').value = '';
|
||
var summary = document.getElementById('bqPrecheckSummary');
|
||
if (summary) summary.style.display = 'none';
|
||
var btn = document.getElementById('registerBqSubmitBtn');
|
||
btn.disabled = false;
|
||
btn.textContent = 'Register Table';
|
||
btn.onclick = registerBqTable;
|
||
document.getElementById('registerBqModal').classList.add('active');
|
||
}
|
||
|
||
// C3: removed dead helpers _openKeboolaRegisterModal /
|
||
// closeRegisterModal that drove the now-deleted #registerModal.
|
||
// The Phase F #registerKeboolaModal owns the Keboola flow now.
|
||
|
||
function closeRegisterBqModal() {
|
||
document.getElementById('registerBqModal').classList.remove('active');
|
||
}
|
||
|
||
// ── Keboola tab register modal (Phase F1) ──────────────────────
|
||
|
||
function _openKeboolaTabRegisterModal() {
|
||
// Reset form to defaults each open. Whole mode is the default
|
||
// (Q2='whole'); the kb-source-table fields are visible.
|
||
var modal = document.getElementById('registerKeboolaModal');
|
||
if (!modal) return;
|
||
var radio = modal.querySelector('input[name="kbSyncMode"][value="whole"]');
|
||
if (radio) radio.checked = true;
|
||
['kbViewName', 'kbBucket', 'kbSourceTable', 'kbSourceQuery',
|
||
'kbSyncSchedule', 'kbDescription', 'kbFolder', 'kbPrimaryKey',
|
||
// v26 Direct-extract fields — reset so a stale value from a prior
|
||
// open doesn't leak into the next registration.
|
||
'kbIncrementalWindowDays', 'kbMaxHistoryDays', 'kbPartitionBy',
|
||
'kbInitialLoadChunkDays', 'kbWhereFilters'].forEach(function(id) {
|
||
var el = document.getElementById(id);
|
||
if (el) el.value = '';
|
||
});
|
||
// Default v26 dropdowns
|
||
var kbStrategyEl = document.getElementById('kbStrategy');
|
||
if (kbStrategyEl) kbStrategyEl.value = 'full_refresh';
|
||
var kbGranEl = document.getElementById('kbPartitionGranularity');
|
||
if (kbGranEl) kbGranEl.value = 'month';
|
||
onKbSyncModeChange(); // apply default visibility
|
||
var btn = document.getElementById('registerKeboolaSubmitBtn');
|
||
btn.disabled = false;
|
||
btn.textContent = 'Register';
|
||
modal.classList.add('active');
|
||
}
|
||
|
||
function closeRegisterKeboolaModal() {
|
||
document.getElementById('registerKeboolaModal').classList.remove('active');
|
||
}
|
||
|
||
function _getKbSyncMode() {
|
||
var el = document.querySelector('input[name="kbSyncMode"]:checked');
|
||
return el ? el.value : 'whole';
|
||
}
|
||
|
||
function onKbSyncModeChange() {
|
||
var mode = _getKbSyncMode();
|
||
// Whole + Direct both need bucket/source_table inputs; only Custom hides them
|
||
document.querySelectorAll('.kb-source-table').forEach(function(el) {
|
||
el.style.display = (mode === 'custom') ? 'none' : '';
|
||
});
|
||
document.querySelectorAll('.kb-source-custom').forEach(function(el) {
|
||
el.style.display = (mode === 'custom') ? '' : 'none';
|
||
});
|
||
// v26 strategy panel — Direct mode only
|
||
document.querySelectorAll('.kb-direct-only').forEach(function(el) {
|
||
el.style.display = (mode === 'direct') ? '' : 'none';
|
||
});
|
||
if (mode === 'direct') onKbStrategyChange();
|
||
}
|
||
|
||
function _getKbStrategy() {
|
||
var el = document.getElementById('kbStrategy');
|
||
return el ? el.value : 'full_refresh';
|
||
}
|
||
|
||
function _kbStrategyVisibility(el, s) {
|
||
// Single-pass visibility check that handles elements tagged with
|
||
// multiple strategy classes (e.g. window_days has both
|
||
// kb-strategy-incremental + kb-strategy-partitioned because the
|
||
// field is shared). Using two separate forEach loops would race —
|
||
// the second loop hiding what the first showed.
|
||
if (el.classList.contains('kb-strategy-not-incremental')) {
|
||
return s !== 'incremental';
|
||
}
|
||
// Element is visible if its class list mentions the active strategy
|
||
return el.classList.contains('kb-strategy-' + s);
|
||
}
|
||
|
||
function onKbStrategyChange() {
|
||
var s = _getKbStrategy();
|
||
document.querySelectorAll(
|
||
'.kb-direct-only .kb-strategy-incremental, ' +
|
||
'.kb-direct-only .kb-strategy-partitioned, ' +
|
||
'.kb-direct-only .kb-strategy-not-incremental'
|
||
).forEach(function(el) {
|
||
el.style.display = _kbStrategyVisibility(el, s) ? '' : 'none';
|
||
});
|
||
}
|
||
|
||
function _intOrNull(id) {
|
||
var raw = (document.getElementById(id).value || '').trim();
|
||
if (!raw) return null;
|
||
var n = parseInt(raw, 10);
|
||
return isNaN(n) ? null : n;
|
||
}
|
||
|
||
function _parseFiltersJSONOrThrow(id) {
|
||
var raw = (document.getElementById(id).value || '').trim();
|
||
if (!raw) return null;
|
||
try {
|
||
return JSON.parse(raw);
|
||
} catch (e) {
|
||
throw new Error('where_filters: invalid JSON — ' + e.message);
|
||
}
|
||
}
|
||
|
||
function _buildKeboolaPayload() {
|
||
// Three modes:
|
||
// whole / custom → query_mode='materialized' (DuckDB Keboola extension)
|
||
// direct → query_mode='local' + v26 sync_strategy fields
|
||
// (Storage API SDK, supports incremental/partitioned/where_filters)
|
||
var mode = _getKbSyncMode();
|
||
var viewName = (document.getElementById('kbViewName').value || '').trim();
|
||
var bucket = (document.getElementById('kbBucket').value || '').trim();
|
||
var sourceTable = (document.getElementById('kbSourceTable').value || '').trim();
|
||
var pk = (document.getElementById('kbPrimaryKey').value || '').trim();
|
||
var primaryKey = pk
|
||
? pk.split(',').map(function(s) { return s.trim(); }).filter(Boolean)
|
||
: [];
|
||
|
||
if (mode === 'direct') {
|
||
var strategy = _getKbStrategy();
|
||
var payload = {
|
||
name: viewName || sourceTable,
|
||
source_type: 'keboola',
|
||
query_mode: 'local',
|
||
bucket: bucket,
|
||
source_table: sourceTable,
|
||
primary_key: primaryKey,
|
||
sync_strategy: strategy,
|
||
sync_schedule: (document.getElementById('kbSyncSchedule').value || '').trim() || null,
|
||
description: (document.getElementById('kbDescription').value || '').trim() || null,
|
||
folder: (document.getElementById('kbFolder').value || '').trim() || null,
|
||
};
|
||
if (strategy === 'incremental' || strategy === 'partitioned') {
|
||
payload.incremental_window_days = _intOrNull('kbIncrementalWindowDays');
|
||
payload.max_history_days = _intOrNull('kbMaxHistoryDays');
|
||
}
|
||
if (strategy === 'partitioned') {
|
||
payload.partition_by = (document.getElementById('kbPartitionBy').value || '').trim() || null;
|
||
payload.partition_granularity = document.getElementById('kbPartitionGranularity').value || null;
|
||
payload.initial_load_chunk_days = _intOrNull('kbInitialLoadChunkDays');
|
||
}
|
||
if (strategy !== 'incremental') {
|
||
payload.where_filters = _parseFiltersJSONOrThrow('kbWhereFilters');
|
||
}
|
||
return payload;
|
||
}
|
||
|
||
// Materialized paths (whole / custom)
|
||
var common = {
|
||
name: viewName || sourceTable,
|
||
source_type: 'keboola',
|
||
query_mode: 'materialized',
|
||
primary_key: primaryKey,
|
||
sync_schedule: (document.getElementById('kbSyncSchedule').value || '').trim() || null,
|
||
description: (document.getElementById('kbDescription').value || '').trim() || null,
|
||
folder: (document.getElementById('kbFolder').value || '').trim() || null,
|
||
};
|
||
|
||
if (mode === 'custom') {
|
||
return Object.assign({}, common, {
|
||
source_query: (document.getElementById('kbSourceQuery').value || '').trim(),
|
||
});
|
||
}
|
||
// Whole — synthesize SELECT * FROM kbc."bucket"."table".
|
||
return Object.assign({}, common, {
|
||
bucket: bucket,
|
||
source_table: sourceTable,
|
||
source_query: 'SELECT * FROM kbc."' + bucket + '"."' + sourceTable + '"',
|
||
});
|
||
}
|
||
|
||
function registerKeboolaTable() {
|
||
var btn = document.getElementById('registerKeboolaSubmitBtn');
|
||
btn.disabled = true;
|
||
btn.textContent = 'Registering...';
|
||
var payload;
|
||
try {
|
||
payload = _buildKeboolaPayload();
|
||
} catch (e) {
|
||
// _parseFiltersJSONOrThrow surfaces malformed JSON before the
|
||
// POST so the operator sees the parse error inline rather than
|
||
// a 422 from the server. Re-enable the submit button.
|
||
showToast('' + e.message, 'error');
|
||
btn.disabled = false;
|
||
btn.textContent = 'Register';
|
||
return;
|
||
}
|
||
fetch('/api/admin/register-table', {
|
||
method: 'POST',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify(payload),
|
||
})
|
||
.then(function(r) {
|
||
if (!r.ok) {
|
||
return r.json().then(function(d) {
|
||
throw new Error(d.detail || d.error || 'Registration failed');
|
||
});
|
||
}
|
||
return r.json();
|
||
})
|
||
.then(function() {
|
||
closeRegisterKeboolaModal();
|
||
showToast('Table registered', 'success');
|
||
loadRegistry();
|
||
})
|
||
.catch(function(err) {
|
||
showToast('' + err.message, 'error');
|
||
})
|
||
.finally(function() {
|
||
btn.disabled = false;
|
||
btn.textContent = 'Register';
|
||
});
|
||
}
|
||
|
||
// Discovery shims — the existing /api/admin/discover-tables endpoint
|
||
// already routes by the instance's data_source.type (returning Keboola
|
||
// tables when the instance is Keboola-typed); the source=keboola query
|
||
// param is informational. Hidden behind `data_source_type == 'keboola'`
|
||
// because on a BQ-typed instance the endpoint would return BQ-shaped
|
||
// data (wrong shape, confusing); operators fall back to manual entry
|
||
// for cross-source registration. Future work: extend the endpoint to
|
||
// accept an explicit ?source= override so secondary-source registration
|
||
// works in both directions and we can remove this guard.
|
||
{% if data_source_type == 'keboola' %}
|
||
function discoverKeboolaBuckets(datalistId) {
|
||
fetch('/api/admin/discover-tables?source=keboola')
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) {
|
||
throw new Error(d.detail || d.error || 'Keboola discovery failed');
|
||
});
|
||
return r.json();
|
||
})
|
||
.then(function(data) {
|
||
var dl = document.getElementById(datalistId);
|
||
if (!dl) return;
|
||
dl.innerHTML = '';
|
||
// Endpoint may return either {buckets:[...]}, {datasets:[...]}
|
||
// or {tables:[...]} depending on routing; project to a flat
|
||
// bucket-id list. Keboola path returns tables → derive uniq
|
||
// bucket_ids.
|
||
var buckets = data.buckets || data.datasets;
|
||
if (!buckets && Array.isArray(data.tables)) {
|
||
var seen = {};
|
||
buckets = [];
|
||
data.tables.forEach(function(t) {
|
||
var b = t.bucket_id || (t.bucket && t.bucket.id);
|
||
if (b && !seen[b]) { seen[b] = 1; buckets.push(b); }
|
||
});
|
||
}
|
||
(buckets || []).forEach(function(b) {
|
||
var o = document.createElement('option');
|
||
o.value = (typeof b === 'string') ? b : (b.id || b.bucket_id || '');
|
||
dl.appendChild(o);
|
||
});
|
||
showToast('Loaded ' + (dl.children.length) + ' buckets', 'success');
|
||
})
|
||
.catch(function(err) {
|
||
showToast('' + err.message, 'error');
|
||
});
|
||
}
|
||
|
||
function discoverKeboolaTables(bucketInputId, tablesDatalistId) {
|
||
var bucketEl = document.getElementById(bucketInputId);
|
||
var bucket = bucketEl ? (bucketEl.value || '').trim() : '';
|
||
if (!bucket) {
|
||
showToast('Fill bucket first', 'error');
|
||
return;
|
||
}
|
||
fetch('/api/admin/discover-tables?source=keboola&bucket=' + encodeURIComponent(bucket))
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) {
|
||
throw new Error(d.detail || d.error || 'Keboola table discovery failed');
|
||
});
|
||
return r.json();
|
||
})
|
||
.then(function(data) {
|
||
var dl = document.getElementById(tablesDatalistId);
|
||
if (!dl) return;
|
||
dl.innerHTML = '';
|
||
var tables = data.tables || [];
|
||
// Filter to the selected bucket if endpoint didn't.
|
||
tables.filter(function(t) {
|
||
var b = t.bucket_id || (t.bucket && t.bucket.id);
|
||
return !bucket || !b || b === bucket;
|
||
}).forEach(function(t) {
|
||
var o = document.createElement('option');
|
||
var name = (typeof t === 'string') ? t : (t.name || t.id || '');
|
||
// Strip bucket prefix if present.
|
||
if (name.indexOf(bucket + '.') === 0) name = name.substring(bucket.length + 1);
|
||
o.value = name;
|
||
dl.appendChild(o);
|
||
});
|
||
showToast('Loaded ' + dl.children.length + ' tables in ' + bucket, 'success');
|
||
})
|
||
.catch(function(err) {
|
||
showToast('' + err.message, 'error');
|
||
});
|
||
}
|
||
|
||
// ── Keboola tab edit modal (Phase F2) ──────────────────────────
|
||
|
||
function _getEditKbSyncMode() {
|
||
var el = document.querySelector('input[name="editKbSyncMode"]:checked');
|
||
return el ? el.value : 'whole';
|
||
}
|
||
|
||
function onEditKbSyncModeChange() {
|
||
var mode = _getEditKbSyncMode();
|
||
document.querySelectorAll('.editkb-source-table').forEach(function(el) {
|
||
el.style.display = (mode === 'custom') ? 'none' : '';
|
||
});
|
||
document.querySelectorAll('.editkb-source-custom').forEach(function(el) {
|
||
el.style.display = (mode === 'custom') ? '' : 'none';
|
||
});
|
||
document.querySelectorAll('.editkb-direct-only').forEach(function(el) {
|
||
el.style.display = (mode === 'direct') ? '' : 'none';
|
||
});
|
||
if (mode === 'direct') onEditKbStrategyChange();
|
||
}
|
||
|
||
function _getEditKbStrategy() {
|
||
var el = document.getElementById('editKbStrategy');
|
||
return el ? el.value : 'full_refresh';
|
||
}
|
||
|
||
function _editKbStrategyVisibility(el, s) {
|
||
if (el.classList.contains('editkb-strategy-not-incremental')) {
|
||
return s !== 'incremental';
|
||
}
|
||
return el.classList.contains('editkb-strategy-' + s);
|
||
}
|
||
|
||
function onEditKbStrategyChange() {
|
||
var s = _getEditKbStrategy();
|
||
document.querySelectorAll(
|
||
'.editkb-direct-only .editkb-strategy-incremental, ' +
|
||
'.editkb-direct-only .editkb-strategy-partitioned, ' +
|
||
'.editkb-direct-only .editkb-strategy-not-incremental'
|
||
).forEach(function(el) {
|
||
el.style.display = _editKbStrategyVisibility(el, s) ? '' : 'none';
|
||
});
|
||
}
|
||
|
||
function _setEditKbRadio(value) {
|
||
var el = document.querySelector('input[name="editKbSyncMode"][value="' + value + '"]');
|
||
if (el) el.checked = true;
|
||
}
|
||
|
||
function openEditKeboolaModal(table) {
|
||
// Populate fields from a registry row. Mode detection:
|
||
// - query_mode='local' → 'direct' (v26 sync-strategy path)
|
||
// - query_mode='materialized' + auto SELECT * → 'whole'
|
||
// - query_mode='materialized' + custom SQL → 'custom'
|
||
table = table || {};
|
||
document.getElementById('editKbTableId').value = table.id || '';
|
||
var bucket = table.bucket || '';
|
||
var sourceTable = table.source_table || '';
|
||
var sourceQuery = table.source_query || '';
|
||
var queryMode = table.query_mode || 'materialized';
|
||
var isAutoSelectStar = false;
|
||
if (sourceQuery && bucket && sourceTable) {
|
||
var auto = 'SELECT * FROM kbc."' + bucket + '"."' + sourceTable + '"';
|
||
isAutoSelectStar = sourceQuery.replace(/\s+/g, ' ').trim() === auto;
|
||
}
|
||
var mode;
|
||
if (queryMode === 'local') {
|
||
mode = 'direct';
|
||
} else if (sourceQuery && !isAutoSelectStar) {
|
||
mode = 'custom';
|
||
} else {
|
||
mode = 'whole';
|
||
}
|
||
_setEditKbRadio(mode);
|
||
document.getElementById('editKbBucket').value = bucket;
|
||
document.getElementById('editKbSourceTable').value = sourceTable;
|
||
document.getElementById('editKbSourceQuery').value = sourceQuery;
|
||
document.getElementById('editKbSyncSchedule').value = table.sync_schedule || '';
|
||
document.getElementById('editKbDescription').value = table.description || '';
|
||
document.getElementById('editKbFolder').value = table.folder || '';
|
||
document.getElementById('editKbPrimaryKey').value = (table.primary_key || []).join(', ');
|
||
|
||
// v26 fields
|
||
document.getElementById('editKbStrategy').value = table.sync_strategy || 'full_refresh';
|
||
document.getElementById('editKbIncrementalWindowDays').value =
|
||
table.incremental_window_days != null ? table.incremental_window_days : '';
|
||
document.getElementById('editKbMaxHistoryDays').value =
|
||
table.max_history_days != null ? table.max_history_days : '';
|
||
document.getElementById('editKbPartitionBy').value = table.partition_by || '';
|
||
document.getElementById('editKbPartitionGranularity').value =
|
||
table.partition_granularity || 'month';
|
||
document.getElementById('editKbInitialLoadChunkDays').value =
|
||
table.initial_load_chunk_days != null ? table.initial_load_chunk_days : '';
|
||
document.getElementById('editKbWhereFilters').value =
|
||
table.where_filters ? JSON.stringify(table.where_filters, null, 2) : '';
|
||
|
||
onEditKbSyncModeChange();
|
||
var btn = document.getElementById('editKeboolaSubmitBtn');
|
||
btn.disabled = false;
|
||
btn.textContent = 'Save Changes';
|
||
document.getElementById('editKeboolaModal').classList.add('active');
|
||
}
|
||
|
||
function closeEditKeboolaModal() {
|
||
document.getElementById('editKeboolaModal').classList.remove('active');
|
||
}
|
||
|
||
function _editIntOrNull(id) {
|
||
var raw = (document.getElementById(id).value || '').trim();
|
||
if (!raw) return null;
|
||
var n = parseInt(raw, 10);
|
||
return isNaN(n) ? null : n;
|
||
}
|
||
|
||
function _editParseFiltersJSONOrThrow(id) {
|
||
var raw = (document.getElementById(id).value || '').trim();
|
||
if (!raw) return null;
|
||
try {
|
||
return JSON.parse(raw);
|
||
} catch (e) {
|
||
throw new Error('where_filters: invalid JSON — ' + e.message);
|
||
}
|
||
}
|
||
|
||
function _buildKeboolaEditPayload() {
|
||
var mode = _getEditKbSyncMode();
|
||
var bucket = (document.getElementById('editKbBucket').value || '').trim();
|
||
var sourceTable = (document.getElementById('editKbSourceTable').value || '').trim();
|
||
var pk = (document.getElementById('editKbPrimaryKey').value || '').trim();
|
||
var primaryKey = pk
|
||
? pk.split(',').map(function(s) { return s.trim(); }).filter(Boolean)
|
||
: [];
|
||
|
||
if (mode === 'direct') {
|
||
var strategy = _getEditKbStrategy();
|
||
var payload = {
|
||
query_mode: 'local',
|
||
bucket: bucket,
|
||
source_table: sourceTable,
|
||
primary_key: primaryKey,
|
||
sync_strategy: strategy,
|
||
sync_schedule: (document.getElementById('editKbSyncSchedule').value || '').trim() || null,
|
||
description: (document.getElementById('editKbDescription').value || '').trim() || null,
|
||
folder: (document.getElementById('editKbFolder').value || '').trim() || null,
|
||
// PUT requires explicit nulls to clear v26 fields when switching
|
||
// away from a strategy that used them — otherwise the merged-dict
|
||
// path keeps the stale value.
|
||
incremental_window_days: null,
|
||
max_history_days: null,
|
||
partition_by: null,
|
||
partition_granularity: null,
|
||
initial_load_chunk_days: null,
|
||
where_filters: null,
|
||
};
|
||
if (strategy === 'incremental' || strategy === 'partitioned') {
|
||
payload.incremental_window_days = _editIntOrNull('editKbIncrementalWindowDays');
|
||
payload.max_history_days = _editIntOrNull('editKbMaxHistoryDays');
|
||
}
|
||
if (strategy === 'partitioned') {
|
||
payload.partition_by = (document.getElementById('editKbPartitionBy').value || '').trim() || null;
|
||
payload.partition_granularity = document.getElementById('editKbPartitionGranularity').value || null;
|
||
payload.initial_load_chunk_days = _editIntOrNull('editKbInitialLoadChunkDays');
|
||
}
|
||
if (strategy !== 'incremental') {
|
||
payload.where_filters = _editParseFiltersJSONOrThrow('editKbWhereFilters');
|
||
}
|
||
return payload;
|
||
}
|
||
|
||
var common = {
|
||
query_mode: 'materialized',
|
||
primary_key: primaryKey,
|
||
sync_schedule: (document.getElementById('editKbSyncSchedule').value || '').trim() || null,
|
||
description: (document.getElementById('editKbDescription').value || '').trim() || null,
|
||
folder: (document.getElementById('editKbFolder').value || '').trim() || null,
|
||
};
|
||
if (mode === 'custom') {
|
||
return Object.assign({}, common, {
|
||
source_query: (document.getElementById('editKbSourceQuery').value || '').trim(),
|
||
});
|
||
}
|
||
return Object.assign({}, common, {
|
||
bucket: bucket,
|
||
source_table: sourceTable,
|
||
source_query: 'SELECT * FROM kbc."' + bucket + '"."' + sourceTable + '"',
|
||
});
|
||
}
|
||
|
||
function saveKeboolaTabEdit() {
|
||
var btn = document.getElementById('editKeboolaSubmitBtn');
|
||
var tableId = document.getElementById('editKbTableId').value;
|
||
if (!tableId) {
|
||
showToast('Missing table id', 'error');
|
||
return;
|
||
}
|
||
btn.disabled = true;
|
||
btn.textContent = 'Saving...';
|
||
var payload;
|
||
try {
|
||
payload = _buildKeboolaEditPayload();
|
||
} catch (e) {
|
||
showToast('' + e.message, 'error');
|
||
btn.disabled = false;
|
||
btn.textContent = 'Save Changes';
|
||
return;
|
||
}
|
||
fetch('/api/admin/registry/' + encodeURIComponent(tableId), {
|
||
method: 'PUT',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify(payload),
|
||
})
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) {
|
||
throw new Error(d.detail || d.error || 'Update failed');
|
||
});
|
||
return r.json();
|
||
})
|
||
.then(function() {
|
||
closeEditKeboolaModal();
|
||
showToast('Table updated', 'success');
|
||
loadRegistry();
|
||
})
|
||
.catch(function(err) {
|
||
showToast('' + err.message, 'error');
|
||
})
|
||
.finally(function() {
|
||
btn.disabled = false;
|
||
btn.textContent = 'Save Changes';
|
||
});
|
||
}
|
||
|
||
function prefillFromKeboolaTable(textareaId) {
|
||
// Edit-modal callers may pass an editKbBucket / editKbSourceTable
|
||
// pair instead of the Register modal's kbBucket / kbSourceTable.
|
||
// Detect the modal via the textarea id prefix.
|
||
var prefix = textareaId.indexOf('editKb') === 0 ? 'editKb' : 'kb';
|
||
var bucket = (document.getElementById(prefix + 'Bucket').value || '').trim();
|
||
var sourceTable = (document.getElementById(prefix + 'SourceTable').value || '').trim();
|
||
if (!bucket || !sourceTable) {
|
||
showToast('Fill bucket + source table first', 'error');
|
||
return;
|
||
}
|
||
var ta = document.getElementById(textareaId);
|
||
if (ta.value.trim()) {
|
||
if (!confirm('Replace existing SQL?')) return;
|
||
}
|
||
ta.value = 'SELECT *\nFROM kbc."' + bucket + '"."' + sourceTable + '"\nWHERE -- your filter here';
|
||
}
|
||
{% endif %}{# data_source_type == 'keboola' — discover/prefill JS #}
|
||
|
||
// C3: removed dead helper _buildKeboolaLegacyPayload — the Phase F
|
||
// _buildKeboolaPayload (above) replaced it.
|
||
|
||
function _getBqAccessMode() {
|
||
// Q1 radio. Default 'live' if nothing's checked yet (model-validator
|
||
// safety net for the initial render).
|
||
var el = document.querySelector('input[name="bqAccessMode"]:checked');
|
||
return el ? el.value : 'live';
|
||
}
|
||
|
||
function _getBqSyncMode() {
|
||
// Q2 radio (only meaningful when access mode is 'synced').
|
||
var el = document.querySelector('input[name="bqSyncMode"]:checked');
|
||
return el ? el.value : 'whole';
|
||
}
|
||
|
||
function _buildBigQueryPayload() {
|
||
// Two-question form maps to backend `query_mode`:
|
||
// live → query_mode='remote' (server auto-detects
|
||
// BASE TABLE vs VIEW)
|
||
// synced/whole → query_mode='materialized' (auto SELECT *)
|
||
// synced/custom → query_mode='materialized' (admin SQL)
|
||
// The UI never asks "Table vs View" — that's a server-side detail.
|
||
var accessMode = _getBqAccessMode();
|
||
var syncMode = _getBqSyncMode();
|
||
var viewName = document.getElementById('bqViewName').value.trim();
|
||
var description = document.getElementById('bqDescription').value.trim() || null;
|
||
var folder = document.getElementById('bqFolder').value.trim() || null;
|
||
var syncSchedule = document.getElementById('bqSyncSchedule').value.trim() || null;
|
||
|
||
if (accessMode === 'synced' && syncMode === 'custom') {
|
||
return {
|
||
name: viewName,
|
||
source_type: 'bigquery',
|
||
query_mode: 'materialized',
|
||
source_query: document.getElementById('bqSourceQuery').value.trim(),
|
||
profile_after_sync: false,
|
||
description: description,
|
||
folder: folder,
|
||
sync_schedule: syncSchedule,
|
||
};
|
||
}
|
||
|
||
var dataset = document.getElementById('bqDataset').value.trim();
|
||
var sourceTable = document.getElementById('bqSourceTable').value.trim();
|
||
|
||
if (accessMode === 'synced' && syncMode === 'whole') {
|
||
// Whole-table sync. We don't ship the project to the browser, so
|
||
// the SQL uses DuckDB three-part syntax against the materialize
|
||
// session's ATTACH alias. Native BQ dry-run can't parse this form
|
||
// (DuckDB identifier quoting), so the cost guardrail falls
|
||
// fail-open with a warning — operator who needs the cap to engage
|
||
// picks Custom query and writes backtick-quoted native BQ
|
||
// identifiers.
|
||
return {
|
||
name: viewName || sourceTable,
|
||
source_type: 'bigquery',
|
||
query_mode: 'materialized',
|
||
source_query: 'SELECT * FROM bq."' + dataset + '"."' + sourceTable + '"',
|
||
profile_after_sync: false,
|
||
description: description,
|
||
folder: folder,
|
||
sync_schedule: syncSchedule,
|
||
};
|
||
}
|
||
|
||
// Live access — server auto-detects BASE TABLE vs VIEW at register
|
||
// time, so the UI doesn't make the operator pick.
|
||
return {
|
||
name: viewName || sourceTable,
|
||
source_type: 'bigquery',
|
||
bucket: dataset,
|
||
source_table: sourceTable,
|
||
query_mode: 'remote',
|
||
profile_after_sync: false,
|
||
description: description,
|
||
folder: folder,
|
||
sync_schedule: syncSchedule,
|
||
};
|
||
}
|
||
|
||
function onBqAccessModeChange() {
|
||
// Q1: toggle live ↔ synced. Q2 (sync mode) is meaningful only when
|
||
// synced; default it to 'whole' on first reveal so the form stays
|
||
// consistent without forcing the operator to click twice.
|
||
var accessMode = _getBqAccessMode();
|
||
var liveFields = document.querySelectorAll('.bq-access-live');
|
||
var syncedFields = document.querySelectorAll('.bq-access-synced');
|
||
liveFields.forEach(function(el) {
|
||
el.style.display = (accessMode === 'live') ? '' : 'none';
|
||
});
|
||
syncedFields.forEach(function(el) {
|
||
el.style.display = (accessMode === 'synced') ? '' : 'none';
|
||
});
|
||
// Q2 is fresh on first reveal; trigger its handler to apply the
|
||
// source-table vs source-custom visibility rules.
|
||
if (accessMode === 'synced') {
|
||
onBqSyncModeChange();
|
||
} else {
|
||
// Live mode: show source-table fields, hide custom-SQL textarea.
|
||
document.querySelectorAll('.bq-source-table').forEach(function(el) {
|
||
el.style.display = '';
|
||
});
|
||
document.querySelectorAll('.bq-source-custom').forEach(function(el) {
|
||
el.style.display = 'none';
|
||
});
|
||
}
|
||
}
|
||
|
||
function onBqSyncModeChange() {
|
||
// Q2: toggle whole-table ↔ custom-SQL. Whole reuses the
|
||
// dataset/source-table inputs (server-side SELECT *), Custom shows
|
||
// the SQL textarea. Only fires when access mode is already 'synced'.
|
||
var syncMode = _getBqSyncMode();
|
||
var tableFields = document.querySelectorAll('.bq-source-table');
|
||
var customFields = document.querySelectorAll('.bq-source-custom');
|
||
tableFields.forEach(function(el) {
|
||
el.style.display = (syncMode === 'whole') ? '' : 'none';
|
||
});
|
||
customFields.forEach(function(el) {
|
||
el.style.display = (syncMode === 'custom') ? '' : 'none';
|
||
});
|
||
}
|
||
|
||
// The discover / list-tables / prefill helpers are shared between the
|
||
// Register and Edit modals. The default datalist + input ids match the
|
||
// Register modal; Edit-modal callers pass their own ids so the
|
||
// autocomplete populates the right datalist and reads from the right
|
||
// dataset input.
|
||
|
||
function discoverBqDatasets(datalistId) {
|
||
// GET /api/admin/discover-tables (no `dataset`) → list datasets in
|
||
// the configured BQ project. Populate the named <datalist> so the
|
||
// dataset input shows them as autocomplete suggestions. Endpoint
|
||
// routes through BqAccess so config / auth errors come back as the
|
||
// standard {error, kind, details} shape.
|
||
var dlId = datalistId || 'bqDatasetList';
|
||
fetch('/api/admin/discover-tables')
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) {
|
||
var msg = (d && d.detail && d.detail.error) || (d && d.detail) || 'BQ discovery failed';
|
||
throw new Error(msg);
|
||
});
|
||
return r.json();
|
||
})
|
||
.then(function(data) {
|
||
var dl = document.getElementById(dlId);
|
||
dl.innerHTML = '';
|
||
(data.datasets || []).forEach(function(ds) {
|
||
var opt = document.createElement('option');
|
||
opt.value = ds.dataset_id;
|
||
opt.label = ds.full_id || ds.dataset_id;
|
||
dl.appendChild(opt);
|
||
});
|
||
showToast('Loaded ' + (data.count || 0) + ' datasets', 'success');
|
||
})
|
||
.catch(function(err) {
|
||
showToast(err.message || 'Discovery failed', 'error');
|
||
});
|
||
}
|
||
|
||
function discoverBqTables(datasetInputId, tablesDatalistId) {
|
||
// GET /api/admin/discover-tables?dataset=NAME → list tables + views
|
||
// in the dataset. Two-step (dataset → tables) avoids paying the
|
||
// per-dataset list_tables() cost up front on big projects.
|
||
var inId = datasetInputId || 'bqDataset';
|
||
var dlId = tablesDatalistId || 'bqTableList';
|
||
var dataset = document.getElementById(inId).value.trim();
|
||
if (!dataset) {
|
||
showToast('Fill Dataset first', 'error');
|
||
return;
|
||
}
|
||
fetch('/api/admin/discover-tables?dataset=' + encodeURIComponent(dataset))
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) {
|
||
var msg = (d && d.detail && d.detail.error) || (d && d.detail) || 'BQ table discovery failed';
|
||
throw new Error(msg);
|
||
});
|
||
return r.json();
|
||
})
|
||
.then(function(data) {
|
||
var dl = document.getElementById(dlId);
|
||
dl.innerHTML = '';
|
||
(data.tables || []).forEach(function(t) {
|
||
var opt = document.createElement('option');
|
||
opt.value = t.table_id;
|
||
// BQ entity types: TABLE, VIEW, MATERIALIZED_VIEW,
|
||
// EXTERNAL, SNAPSHOT — surface so the operator can
|
||
// pick the right entity option.
|
||
opt.label = t.table_id + (t.table_type ? ' (' + t.table_type + ')' : '');
|
||
dl.appendChild(opt);
|
||
});
|
||
showToast('Loaded ' + (data.count || 0) + ' tables in ' + dataset, 'success');
|
||
})
|
||
.catch(function(err) {
|
||
showToast(err.message || 'Table list failed', 'error');
|
||
});
|
||
}
|
||
|
||
function prefillFromTable(textareaId) {
|
||
// Convenience for the Custom SQL path: prompt for project.dataset.table
|
||
// and prefill `SELECT * FROM \`...\`` so the operator only edits the
|
||
// WHERE / projection. We can't know the project from the form (it's
|
||
// server-side config), so the prompt accepts both `dataset.table` and
|
||
// `project.dataset.table`. Empty / cancel → no change.
|
||
var taId = textareaId || 'bqSourceQuery';
|
||
var ta = document.getElementById(taId);
|
||
var existing = ta.value.trim();
|
||
if (existing && !confirm('SQL field is not empty — overwrite with prefill?')) {
|
||
return;
|
||
}
|
||
var ref = window.prompt(
|
||
'Enter the source table as `dataset.table` or `project.dataset.table` '
|
||
+ '(it will be wrapped in backticks):',
|
||
''
|
||
);
|
||
if (!ref) return;
|
||
ref = ref.trim().replace(/^`|`$/g, '');
|
||
var stub =
|
||
'SELECT *\n'
|
||
+ 'FROM `' + ref + '`\n'
|
||
+ 'WHERE -- TODO: filter or aggregate before the COPY runs (10 GiB cap)\n'
|
||
+ 'LIMIT 1000';
|
||
ta.value = stub;
|
||
}
|
||
|
||
// C3: removed dead entry points registerTable / _registerKeboolaTable
|
||
// that submitted the now-deleted #registerModal. The Keboola Register
|
||
// flow runs through #registerKeboolaModal → registerKeboolaTable().
|
||
|
||
function registerBqTable() {
|
||
// BQ tab modal entry point — runs the same two-step precheck →
|
||
// confirm flow against #registerBqSubmitBtn / #registerBqModal.
|
||
var btn = document.getElementById('registerBqSubmitBtn');
|
||
btn.disabled = true;
|
||
_registerBigQueryTable(btn);
|
||
}
|
||
|
||
function _registerBigQueryTable(btn) {
|
||
// Step 1 of two-step flow: precheck only. After precheck succeeds
|
||
// we surface row count / size / column count in the modal AND swap
|
||
// the primary button to "Register" — operator must explicitly
|
||
// click it to fire the actual register call. Pre-fix this chained
|
||
// precheck → register in a single promise, so the operator never
|
||
// got to review the summary before the row was committed (review
|
||
// IMPORTANT 4 in #119).
|
||
btn.textContent = 'Checking source...';
|
||
var payload = _buildBigQueryPayload();
|
||
var summary = document.getElementById('bqPrecheckSummary');
|
||
var summaryText = document.getElementById('bqPrecheckSummaryText');
|
||
|
||
fetch('/api/admin/register-table/precheck', {
|
||
method: 'POST',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify(payload)
|
||
})
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) { throw new Error(d.detail || 'Precheck failed'); });
|
||
return r.json();
|
||
})
|
||
.then(function(data) {
|
||
// Surface the source-side metrics so the operator can sanity-
|
||
// check the dataset/table before committing the row.
|
||
var t = data.table || {};
|
||
if (summary && summaryText) {
|
||
summary.style.display = '';
|
||
summaryText.textContent =
|
||
formatNumber(t.rows) + ' rows, ' +
|
||
formatSize(t.size_bytes) + ', ' +
|
||
(t.column_count != null ? t.column_count : (t.columns || []).length) + ' columns';
|
||
}
|
||
// Swap the primary button from "Register Table" (which fired
|
||
// the precheck) to "Register" (which fires the actual register
|
||
// call). Reusing the same modal means the operator can still
|
||
// hit Cancel or close without committing.
|
||
btn.disabled = false;
|
||
btn.textContent = 'Register';
|
||
btn.onclick = function() { _confirmRegisterBigQueryTable(btn, payload); };
|
||
})
|
||
.catch(function(err) {
|
||
showToast('Precheck failed: ' + err.message, 'error');
|
||
btn.disabled = false;
|
||
btn.textContent = 'Register Table';
|
||
});
|
||
}
|
||
|
||
function _confirmRegisterBigQueryTable(btn, payload) {
|
||
// Step 2 of the two-step BQ flow: actually POST /register-table.
|
||
// Reached only after the operator has reviewed the precheck
|
||
// summary in the modal and clicked the (relabeled) Register button.
|
||
btn.disabled = true;
|
||
btn.textContent = 'Registering...';
|
||
fetch('/api/admin/register-table', {
|
||
method: 'POST',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify(payload)
|
||
})
|
||
.then(function(r) {
|
||
// 200 (synchronous materialize) and 202 (BG materialize) are
|
||
// success; 500 with a `rebuild_failed` body is the new failure
|
||
// mode where the row is in the registry but the view wasn't
|
||
// created. Surface the error verbatim so the operator can fix
|
||
// it (typically a missing project in the overlay or auth
|
||
// missing bigquery.metadata.get).
|
||
return r.json().then(function(d) {
|
||
if (!r.ok) {
|
||
var msg = d.detail || d.message || 'Registration failed';
|
||
if (d.errors && d.errors.length) {
|
||
msg += ' (' + d.errors.length + ' error' + (d.errors.length > 1 ? 's' : '') + ')';
|
||
}
|
||
throw new Error(msg);
|
||
}
|
||
closeRegisterBqModal();
|
||
if (r.status === 202) {
|
||
showToast('Registration accepted, materializing in background', 'success');
|
||
} else {
|
||
showToast('Queryable as ' + (d.view_name || d.id || payload.name), 'success');
|
||
}
|
||
loadRegistry();
|
||
});
|
||
})
|
||
.catch(function(err) {
|
||
showToast('Registration failed: ' + err.message, 'error');
|
||
})
|
||
.finally(function() {
|
||
// Restore the modal's default button state — the operator may
|
||
// close + reopen the modal, and openRegisterModal also resets
|
||
// onclick back to registerBqTable.
|
||
btn.disabled = false;
|
||
btn.textContent = 'Register Table';
|
||
btn.onclick = registerBqTable;
|
||
});
|
||
}
|
||
|
||
// Legacy entry point — kept for backward-compat with any external
|
||
// wiring that may still call openBigQueryRegisterModal(). Phase E:
|
||
// routes through the new BQ-tab modal.
|
||
function openBigQueryRegisterModal() {
|
||
openRegisterModal('bigquery');
|
||
}
|
||
|
||
// ── Edit Modal ──────────────────────────────────────────────
|
||
|
||
// Original mode captured at openEditModal-time so the access-mode
|
||
// change handler can detect "operator just switched it" vs "loaded
|
||
// from registry" and only surface the destructive-edit warning on a
|
||
// real change.
|
||
let _editOriginalQueryMode = null;
|
||
|
||
function _getEditBqAccessMode() {
|
||
var el = document.querySelector('input[name="editBqAccessMode"]:checked');
|
||
return el ? el.value : 'live';
|
||
}
|
||
|
||
function _getEditBqSyncMode() {
|
||
var el = document.querySelector('input[name="editBqSyncMode"]:checked');
|
||
return el ? el.value : 'whole';
|
||
}
|
||
|
||
function _setEditBqRadio(name, value) {
|
||
var el = document.querySelector('input[name="' + name + '"][value="' + value + '"]');
|
||
if (el) el.checked = true;
|
||
}
|
||
|
||
function openEditModal(table) {
|
||
// C2: dispatch by source_type. BigQuery rows go to the relocated
|
||
// #editBqModal inside the BQ tab; Keboola rows go to the existing
|
||
// #editKeboolaModal inside the Keboola tab; anything else (jira,
|
||
// future sources) falls through to the legacy #editModal which now
|
||
// only carries Description/Folder + a Keboola-style Strategy/PK
|
||
// fallback.
|
||
var sourceType = (table.source_type || '').toLowerCase();
|
||
if (sourceType === 'bigquery') {
|
||
return _openEditBqModal(table);
|
||
}
|
||
if (sourceType === 'keboola') {
|
||
return openEditKeboolaModal(table);
|
||
}
|
||
return _openEditLegacyModal(table);
|
||
}
|
||
|
||
function _openEditBqModal(table) {
|
||
// Populate the BQ-tab Edit modal from a registry row. Mirror of
|
||
// populateEditModal's BQ branch from before C2 — same query_mode →
|
||
// radio mapping, same field bindings, but writes to the new
|
||
// #editBqModal scope.
|
||
currentEditTableId = table.id;
|
||
document.getElementById('editBqTableId').value = table.id || '';
|
||
var badge = document.getElementById('editBqSourceTypeBadge');
|
||
if (badge) badge.textContent = (table.source_type || 'bigquery');
|
||
|
||
var qmode = (table.query_mode || 'remote');
|
||
var sq = (table.source_query || '');
|
||
if (qmode === 'materialized') {
|
||
_setEditBqRadio('editBqAccessMode', 'synced');
|
||
var isAutoSelectStar = /^\s*SELECT\s+\*\s+FROM\s+bq\.".+"\.".+"\s*$/i.test(sq);
|
||
_setEditBqRadio('editBqSyncMode', isAutoSelectStar ? 'whole' : 'custom');
|
||
} else {
|
||
_setEditBqRadio('editBqAccessMode', 'live');
|
||
_setEditBqRadio('editBqSyncMode', 'whole');
|
||
}
|
||
|
||
document.getElementById('editBqDataset').value = table.bucket || '';
|
||
document.getElementById('editBqSourceTable').value = table.source_table || '';
|
||
document.getElementById('editBqSourceQuery').value = sq;
|
||
document.getElementById('editBqSyncSchedule').value = table.sync_schedule || '';
|
||
document.getElementById('editBqDescription').value = table.description || '';
|
||
document.getElementById('editBqFolder').value = table.folder || '';
|
||
document.getElementById('editBqModeWarning').style.display = 'none';
|
||
_editOriginalQueryMode = qmode;
|
||
onEditBqAccessModeChange(); // fire to set field visibility
|
||
|
||
var btn = document.getElementById('editBqSubmitBtn');
|
||
btn.disabled = false;
|
||
btn.textContent = 'Save Changes';
|
||
document.getElementById('editBqModal').classList.add('active');
|
||
}
|
||
|
||
function closeEditBqModal() {
|
||
document.getElementById('editBqModal').classList.remove('active');
|
||
currentEditTableId = null;
|
||
_editOriginalQueryMode = null;
|
||
}
|
||
|
||
function _openEditLegacyModal(table) {
|
||
// Legacy fallback for source_types that aren't bigquery or keboola
|
||
// (e.g. jira). Carries the Strategy + Primary Key + Description +
|
||
// Folder fields only.
|
||
currentEditTableId = table.id;
|
||
document.getElementById('editTableId').value = table.id || '';
|
||
var sourceType = (table.source_type || '').toLowerCase();
|
||
document.getElementById('editSourceTypeBadge').textContent = sourceType || '—';
|
||
document.getElementById('editDescription').value = table.description || '';
|
||
document.getElementById('editFolder').value = table.folder || '';
|
||
document.getElementById('editStrategy').value = table.sync_strategy || 'full_refresh';
|
||
document.getElementById('editPrimaryKey').value = (table.primary_key || []).join(', ');
|
||
_editOriginalQueryMode = null;
|
||
document.getElementById('editSubmitBtn').disabled = false;
|
||
document.getElementById('editModal').classList.add('active');
|
||
}
|
||
|
||
function onEditBqAccessModeChange() {
|
||
var accessMode = _getEditBqAccessMode();
|
||
document.querySelectorAll('.bq-edit-access-synced').forEach(function(el) {
|
||
el.style.display = (accessMode === 'synced') ? '' : 'none';
|
||
});
|
||
|
||
if (accessMode === 'synced') {
|
||
onEditBqSyncModeChange();
|
||
} else {
|
||
// Live: dataset+table inputs visible, SQL textarea hidden.
|
||
document.querySelectorAll('.bq-edit-source-table').forEach(function(el) {
|
||
el.style.display = '';
|
||
});
|
||
document.querySelectorAll('.bq-edit-source-custom').forEach(function(el) {
|
||
el.style.display = 'none';
|
||
});
|
||
}
|
||
|
||
// Mode-switch warning: only fire when the operator actually flipped
|
||
// access mode from what was loaded — typo-fix edits stay quiet.
|
||
var newMode = (accessMode === 'synced') ? 'materialized' : 'remote';
|
||
var warn = document.getElementById('editBqModeWarning');
|
||
if (_editOriginalQueryMode && newMode !== _editOriginalQueryMode) {
|
||
var msg;
|
||
if (_editOriginalQueryMode === 'materialized' && newMode === 'remote') {
|
||
msg = '⚠ Switching Synced locally → Live from BigQuery will drop the materialized parquet on the next sync. Analysts who already pulled this table will start getting live BQ queries instead of a local copy; the sync schedule becomes ignored.';
|
||
} else {
|
||
msg = '⚠ Switching Live from BigQuery → Synced locally: the next scheduled sync runs a SELECT and writes a parquet. Analysts will start reading the local copy on their next `agnes pull`. Remember to set a sync schedule.';
|
||
}
|
||
warn.textContent = msg;
|
||
warn.style.display = '';
|
||
} else {
|
||
warn.style.display = 'none';
|
||
}
|
||
}
|
||
|
||
function onEditBqSyncModeChange() {
|
||
var syncMode = _getEditBqSyncMode();
|
||
document.querySelectorAll('.bq-edit-source-table').forEach(function(el) {
|
||
el.style.display = (syncMode === 'whole') ? '' : 'none';
|
||
});
|
||
document.querySelectorAll('.bq-edit-source-custom').forEach(function(el) {
|
||
el.style.display = (syncMode === 'custom') ? '' : 'none';
|
||
});
|
||
}
|
||
|
||
function closeEditModal() {
|
||
document.getElementById('editModal').classList.remove('active');
|
||
currentEditTableId = null;
|
||
_editOriginalQueryMode = null;
|
||
}
|
||
|
||
function saveTableEdit() {
|
||
// Legacy save handler — runs only for source_types that aren't
|
||
// bigquery (handled by saveBqTabEdit) or keboola (handled by
|
||
// saveKeboolaTabEdit). Currently the catch-all for jira-style rows.
|
||
if (!currentEditTableId) return;
|
||
|
||
var btn = document.getElementById('editSubmitBtn');
|
||
btn.disabled = true;
|
||
btn.textContent = 'Saving...';
|
||
|
||
var pk = document.getElementById('editPrimaryKey').value.trim();
|
||
var primaryKey = pk
|
||
? pk.split(',').map(function(s) { return s.trim(); }).filter(Boolean)
|
||
: [];
|
||
var payload = {
|
||
sync_strategy: document.getElementById('editStrategy').value,
|
||
primary_key: primaryKey,
|
||
description:
|
||
document.getElementById('editDescription').value.trim() || null,
|
||
folder: document.getElementById('editFolder').value.trim() || null,
|
||
};
|
||
|
||
fetch('/api/admin/registry/' + encodeURIComponent(currentEditTableId), {
|
||
method: 'PUT',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify(payload)
|
||
})
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) { throw new Error(d.detail || d.error || 'Update failed'); });
|
||
return r.json();
|
||
})
|
||
.then(function(data) {
|
||
closeEditModal();
|
||
showToast('Table updated successfully', 'success');
|
||
loadRegistry();
|
||
})
|
||
.catch(function(err) {
|
||
showToast('Update failed: ' + err.message, 'error');
|
||
})
|
||
.finally(function() {
|
||
btn.disabled = false;
|
||
btn.textContent = 'Save Changes';
|
||
});
|
||
}
|
||
|
||
function saveBqTabEdit() {
|
||
// Save handler for the relocated BQ Edit modal (#editBqModal).
|
||
// Mirrors the pre-C2 BQ branch of saveTableEdit() but reads from
|
||
// the BQ-modal-scoped fields (editBqDescription / editBqFolder).
|
||
if (!currentEditTableId) return;
|
||
|
||
var btn = document.getElementById('editBqSubmitBtn');
|
||
btn.disabled = true;
|
||
btn.textContent = 'Saving...';
|
||
|
||
// Two-question state machine — same as Register modal:
|
||
// live → query_mode='remote'
|
||
// synced/whole → materialized + auto SELECT *
|
||
// synced/custom → materialized + admin SQL
|
||
var accessMode = _getEditBqAccessMode();
|
||
var syncMode = _getEditBqSyncMode();
|
||
var dataset = document.getElementById('editBqDataset').value.trim();
|
||
var sourceTable = document.getElementById('editBqSourceTable').value.trim();
|
||
|
||
var payload = {
|
||
description: document.getElementById('editBqDescription').value.trim() || null,
|
||
folder: document.getElementById('editBqFolder').value.trim() || null,
|
||
source_type: 'bigquery',
|
||
sync_schedule:
|
||
document.getElementById('editBqSyncSchedule').value.trim() || null,
|
||
};
|
||
|
||
if (accessMode === 'synced' && syncMode === 'custom') {
|
||
payload.query_mode = 'materialized';
|
||
payload.source_query =
|
||
document.getElementById('editBqSourceQuery').value.trim();
|
||
// PUT semantics: explicit null clears prior value so a
|
||
// remote→materialized switch doesn't carry stale
|
||
// bucket/source_table forward through the synthetic
|
||
// RegisterTableRequest validator.
|
||
payload.bucket = null;
|
||
payload.source_table = null;
|
||
} else if (accessMode === 'synced' && syncMode === 'whole') {
|
||
payload.query_mode = 'materialized';
|
||
payload.source_query =
|
||
'SELECT * FROM bq."' + dataset + '"."' + sourceTable + '"';
|
||
payload.bucket = dataset || null;
|
||
payload.source_table = sourceTable || null;
|
||
} else {
|
||
// Live.
|
||
payload.query_mode = 'remote';
|
||
payload.bucket = dataset || null;
|
||
payload.source_table = sourceTable || null;
|
||
payload.source_query = null;
|
||
}
|
||
|
||
fetch('/api/admin/registry/' + encodeURIComponent(currentEditTableId), {
|
||
method: 'PUT',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify(payload)
|
||
})
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) { throw new Error(d.detail || d.error || 'Update failed'); });
|
||
return r.json();
|
||
})
|
||
.then(function() {
|
||
closeEditBqModal();
|
||
showToast('Table updated successfully', 'success');
|
||
loadRegistry();
|
||
})
|
||
.catch(function(err) {
|
||
showToast('Update failed: ' + err.message, 'error');
|
||
})
|
||
.finally(function() {
|
||
btn.disabled = false;
|
||
btn.textContent = 'Save Changes';
|
||
});
|
||
}
|
||
|
||
// ── Registry ────────────────────────────────────────────────
|
||
|
||
function loadRegistry() {
|
||
fetch('/api/admin/registry')
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) { throw new Error(d.error || 'Load failed'); });
|
||
return r.json();
|
||
})
|
||
.then(function(data) {
|
||
registryData = data;
|
||
registryVersion = data.version;
|
||
var tables = data.tables || [];
|
||
var subtitle = document.getElementById('registrySubtitle');
|
||
if (subtitle) {
|
||
subtitle.textContent = tables.length + ' table' + (tables.length !== 1 ? 's' : '') + ' configured for sync';
|
||
}
|
||
renderRegistryListing(
|
||
document.getElementById('bqTableListing'),
|
||
tables.filter(function(t) { return t.source_type === 'bigquery'; })
|
||
);
|
||
renderRegistryListing(
|
||
document.getElementById('kbTableListing'),
|
||
tables.filter(function(t) { return t.source_type === 'keboola'; })
|
||
);
|
||
renderRegistryListing(
|
||
document.getElementById('jiraTableListing'),
|
||
tables.filter(function(t) { return t.source_type === 'jira'; })
|
||
);
|
||
})
|
||
.catch(function(err) {
|
||
var msg = '<div class="panel-body-empty" style="color: var(--error);">Failed to load registry: ' + escapeHtml(err.message) + '</div>';
|
||
[
|
||
document.getElementById('bqTableListing'),
|
||
document.getElementById('kbTableListing'),
|
||
document.getElementById('jiraTableListing'),
|
||
].forEach(function(target) { if (target) target.innerHTML = msg; });
|
||
});
|
||
}
|
||
|
||
function renderModeBadge(mode) {
|
||
var m = (mode || 'local').toLowerCase();
|
||
var cls = 'mode-badge mode-' + (['local','remote','materialized'].indexOf(m) >= 0 ? m : 'local');
|
||
return '<span class="' + cls + '">' + escapeHtml(m) + '</span>';
|
||
}
|
||
|
||
function renderRegistryListing(target, tables) {
|
||
if (!target) return;
|
||
if (tables.length === 0) {
|
||
target.innerHTML = '<div class="panel-body-empty">No tables registered yet.</div>';
|
||
return;
|
||
}
|
||
var html = '<table class="registry-table">';
|
||
html += '<thead><tr>';
|
||
html += '<th class="col-id">Table ID</th>';
|
||
html += '<th class="col-mode">Mode</th>';
|
||
html += '<th class="col-source">Source</th>';
|
||
html += '<th class="col-pk">Primary Key</th>';
|
||
html += '<th class="col-schedule">Schedule</th>';
|
||
html += '<th class="col-folder">Folder</th>';
|
||
html += '<th>Description</th>';
|
||
html += '<th class="col-registered">Registered</th>';
|
||
html += '<th class="col-status"></th>';
|
||
html += '<th class="col-actions">Actions</th>';
|
||
html += '</tr></thead><tbody>';
|
||
tables.forEach(function(table) {
|
||
html += '<tr>';
|
||
html += '<td class="col-id" title="' + escapeHtml(table.id) + '">' + escapeHtml(table.id) + '</td>';
|
||
html += '<td class="col-mode">' + renderModeBadge(table.query_mode) + '</td>';
|
||
|
||
// Source: bucket / source_table; em-dash when both empty (custom-SQL row).
|
||
var bucket = table.bucket || '';
|
||
var srcTable = table.source_table || '';
|
||
var sourceText = '';
|
||
if (bucket && srcTable) {
|
||
sourceText = bucket + ' / ' + srcTable;
|
||
} else if (bucket || srcTable) {
|
||
sourceText = bucket || srcTable;
|
||
}
|
||
var sourceCell = sourceText ? escapeHtml(sourceText) : '—';
|
||
html += '<td class="col-source" title="' + escapeHtml(sourceText) + '">' + sourceCell + '</td>';
|
||
|
||
html += '<td class="col-pk">' + escapeHtml((table.primary_key || []).join(', ') || '—') + '</td>';
|
||
html += '<td class="col-schedule">' + escapeHtml(table.sync_schedule || '—') + '</td>';
|
||
|
||
// Folder badge — '—' when null/empty.
|
||
if (table.folder) {
|
||
html += '<td class="col-folder"><span class="folder-badge">' + escapeHtml(table.folder) + '</span></td>';
|
||
} else {
|
||
html += '<td class="col-folder">—</td>';
|
||
}
|
||
|
||
var desc = unescapeShellQuoting(table.description || '');
|
||
html += '<td class="col-description" title="' + escapeHtml(desc) + '">' + escapeHtml(desc || '—') + '</td>';
|
||
|
||
// Registered: stacked email + date (first 10 chars of ISO timestamp).
|
||
var regBy = table.registered_by || '';
|
||
var regByDisplay = regBy;
|
||
if (regBy.length > 24 && regBy.indexOf('@') > 0) {
|
||
regByDisplay = regBy.split('@')[0];
|
||
}
|
||
var regAt = table.registered_at ? String(table.registered_at).slice(0, 10) : '';
|
||
html += '<td class="col-registered">';
|
||
html += '<div class="registered-by" title="' + escapeHtml(regBy) + '">' + (regByDisplay ? escapeHtml(regByDisplay) : '—') + '</div>';
|
||
html += '<div class="registered-at">' + escapeHtml(regAt || '') + '</div>';
|
||
html += '</td>';
|
||
|
||
// Status: warning icon when the last sync errored.
|
||
html += '<td class="col-status">';
|
||
if (table.last_sync_error) {
|
||
html += '<span title="' + escapeHtml(table.last_sync_error) + '">';
|
||
html += '<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" style="color: var(--error);"><path d="M10.29 3.86 1.82 18a2 2 0 0 0 1.71 3h16.94a2 2 0 0 0 1.71-3L13.71 3.86a2 2 0 0 0-3.42 0z"/><line x1="12" y1="9" x2="12" y2="13"/><line x1="12" y1="17" x2="12.01" y2="17"/></svg>';
|
||
html += '</span>';
|
||
}
|
||
html += '</td>';
|
||
|
||
html += '<td class="col-actions"><div style="display:flex; gap:4px; justify-content:flex-end;">';
|
||
html += '<button class="btn-icon" title="Edit" onclick=\'openEditModal(' + JSON.stringify(table).replace(/'/g, "\\'") + ')\'>';
|
||
html += '<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M11 4H4a2 2 0 0 0-2 2v14a2 2 0 0 0 2 2h14a2 2 0 0 0 2-2v-7"/><path d="M18.5 2.5a2.121 2.121 0 0 1 3 3L12 15l-4 1 1-4 9.5-9.5z"/></svg>';
|
||
html += '</button>';
|
||
// Manage access: deep-link to /admin/access pre-filtered to this table.
|
||
html += '<button class="btn-icon" title="Manage access" onclick="manageAccess(\'' + escapeHtml(table.id).replace(/'/g, "\\'") + '\')">';
|
||
html += '<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect x="3" y="11" width="18" height="11" rx="2" ry="2"/><path d="M7 11V7a5 5 0 0 1 10 0v4"/></svg>';
|
||
html += '</button>';
|
||
html += '<button class="btn-icon danger" title="Delete" onclick="deleteTable(\'' + escapeHtml(table.id).replace(/'/g, "\\'") + '\')">';
|
||
html += '<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><polyline points="3 6 5 6 21 6"/><path d="M19 6v14a2 2 0 0 1-2 2H7a2 2 0 0 1-2-2V6m3 0V4a2 2 0 0 1 2-2h4a2 2 0 0 1 2 2v2"/></svg>';
|
||
html += '</button>';
|
||
html += '</div></td></tr>';
|
||
});
|
||
html += '</tbody></table>';
|
||
target.innerHTML = html;
|
||
}
|
||
|
||
// Deep-link from a table row to /admin/access scoped to this table_id.
|
||
// The /admin/access page reads window.location.hash on bootstrap and,
|
||
// when it matches `table:<id>`, pre-fills the resource filter so the
|
||
// operator lands on the just-clicked table once they pick a group.
|
||
function manageAccess(tableId) {
|
||
window.location.href = '/admin/access#table:' + encodeURIComponent(tableId);
|
||
}
|
||
|
||
function deleteTable(tableId) {
|
||
if (!confirm('Are you sure you want to unregister "' + tableId + '"?\n\nThis will remove it from sync and clean up user subscriptions.')) {
|
||
return;
|
||
}
|
||
|
||
// DELETE doesn't need a body — the API has no `version` field.
|
||
fetch('/api/admin/registry/' + encodeURIComponent(tableId), {
|
||
method: 'DELETE',
|
||
})
|
||
.then(function(r) {
|
||
// 204 No Content has an empty body, so don't .json() it.
|
||
if (r.status === 204) return null;
|
||
if (!r.ok) return r.json().then(function(d) { throw new Error(d.detail || d.error || 'Delete failed'); });
|
||
return r.json();
|
||
})
|
||
.then(function() {
|
||
showToast('Table unregistered successfully', 'success');
|
||
loadRegistry();
|
||
})
|
||
.catch(function(err) {
|
||
showToast('Delete failed: ' + err.message, 'error');
|
||
});
|
||
}
|
||
|
||
// ── Modal close on overlay click ────────────────────────────
|
||
|
||
var _registerBqModalEl = document.getElementById('registerBqModal');
|
||
if (_registerBqModalEl) {
|
||
_registerBqModalEl.addEventListener('click', function(e) {
|
||
if (e.target === this) closeRegisterBqModal();
|
||
});
|
||
}
|
||
|
||
var _registerKeboolaModalEl = document.getElementById('registerKeboolaModal');
|
||
if (_registerKeboolaModalEl) {
|
||
_registerKeboolaModalEl.addEventListener('click', function(e) {
|
||
if (e.target === this) closeRegisterKeboolaModal();
|
||
});
|
||
}
|
||
|
||
var _editKeboolaModalEl = document.getElementById('editKeboolaModal');
|
||
if (_editKeboolaModalEl) {
|
||
_editKeboolaModalEl.addEventListener('click', function(e) {
|
||
if (e.target === this) closeEditKeboolaModal();
|
||
});
|
||
}
|
||
|
||
var _editBqModalEl = document.getElementById('editBqModal');
|
||
if (_editBqModalEl) {
|
||
_editBqModalEl.addEventListener('click', function(e) {
|
||
if (e.target === this) closeEditBqModal();
|
||
});
|
||
}
|
||
|
||
document.getElementById('editModal').addEventListener('click', function(e) {
|
||
if (e.target === this) closeEditModal();
|
||
});
|
||
|
||
// Close modals on Escape key
|
||
document.addEventListener('keydown', function(e) {
|
||
if (e.key === 'Escape') {
|
||
closeRegisterBqModal();
|
||
if (typeof closeRegisterKeboolaModal === 'function') closeRegisterKeboolaModal();
|
||
if (typeof closeEditKeboolaModal === 'function') closeEditKeboolaModal();
|
||
if (typeof closeEditBqModal === 'function') closeEditBqModal();
|
||
closeEditModal();
|
||
}
|
||
});
|
||
|
||
// ── Initialize ──────────────────────────────────────────────
|
||
|
||
loadRegistry();
|
||
|
||
// ── Cache warmup toolbar (issue #155 / #156) ────────────────
|
||
let cacheWarmupSource = null;
|
||
|
||
function _cacheWarmupClearPollFallback() {
|
||
if (window._cacheWarmupPollInterval) {
|
||
clearInterval(window._cacheWarmupPollInterval);
|
||
window._cacheWarmupPollInterval = null;
|
||
}
|
||
}
|
||
|
||
function cacheWarmupInit() {
|
||
cacheWarmupRefreshSnapshot();
|
||
cacheWarmupOpenStream();
|
||
}
|
||
|
||
function cacheWarmupRefreshSnapshot() {
|
||
fetch('/api/admin/cache-warmup/status')
|
||
.then(function(r) { return r.json(); })
|
||
.then(function(state) { cacheWarmupRender(state); })
|
||
.catch(function() { /* silent */ });
|
||
}
|
||
|
||
function cacheWarmupOpenStream() {
|
||
try {
|
||
cacheWarmupSource = new EventSource('/api/admin/cache-warmup/stream');
|
||
cacheWarmupSource.addEventListener('start', cacheWarmupOnStart);
|
||
cacheWarmupSource.addEventListener('row', cacheWarmupOnRow);
|
||
cacheWarmupSource.addEventListener('complete', cacheWarmupOnComplete);
|
||
cacheWarmupSource.addEventListener('snapshot', function(e) {
|
||
_cacheWarmupClearPollFallback();
|
||
cacheWarmupRender(JSON.parse(e.data));
|
||
});
|
||
cacheWarmupSource.onerror = function() {
|
||
if (cacheWarmupSource) {
|
||
cacheWarmupSource.close();
|
||
cacheWarmupSource = null;
|
||
}
|
||
// Continuous polling fallback. Try to re-open SSE every 30 s in
|
||
// case the proxy / network heals. Only one polling interval at a
|
||
// time (prevent stacking on repeated errors).
|
||
if (!window._cacheWarmupPollInterval) {
|
||
window._cacheWarmupPollInterval = setInterval(
|
||
cacheWarmupRefreshSnapshot, 3000
|
||
);
|
||
setTimeout(function tryReconnect() {
|
||
if (cacheWarmupSource) return; // already reconnected
|
||
try {
|
||
clearInterval(window._cacheWarmupPollInterval);
|
||
window._cacheWarmupPollInterval = null;
|
||
cacheWarmupOpenStream(); // recursive — onerror retries again
|
||
} catch (e) {
|
||
window._cacheWarmupPollInterval = setInterval(
|
||
cacheWarmupRefreshSnapshot, 3000
|
||
);
|
||
setTimeout(tryReconnect, 30000);
|
||
}
|
||
}, 30000);
|
||
}
|
||
};
|
||
} catch (e) {
|
||
setInterval(cacheWarmupRefreshSnapshot, 3000);
|
||
}
|
||
}
|
||
|
||
function cacheWarmupRender(state) {
|
||
var summary = document.getElementById('cacheWarmupSummary');
|
||
var bar = document.getElementById('cacheWarmupBar');
|
||
var btn = document.getElementById('cacheWarmupRunBtn');
|
||
if (!summary) return;
|
||
|
||
if (!state || state.state === 'never_run') {
|
||
summary.textContent = 'No cache warmup yet — click Re-warm all to start.';
|
||
bar.style.display = 'none';
|
||
btn.disabled = false;
|
||
return;
|
||
}
|
||
var inProgress = state.completed_at === null || state.completed_at === undefined;
|
||
var pct = state.total > 0 ? Math.round((state.completed * 100) / state.total) : 0;
|
||
summary.textContent = inProgress
|
||
? state.completed + ' / ' + state.total + ' fresh — running…'
|
||
: 'Last run: ' + state.completed + ' ok, ' + state.failed + ' errors';
|
||
bar.style.display = 'block';
|
||
bar.value = pct;
|
||
btn.disabled = inProgress;
|
||
|
||
if (state.rows) {
|
||
for (var tid in state.rows) {
|
||
cacheWarmupSetRowBadge(tid, state.rows[tid]);
|
||
}
|
||
}
|
||
}
|
||
|
||
function cacheWarmupOnStart(e) {
|
||
_cacheWarmupClearPollFallback();
|
||
var data = JSON.parse(e.data);
|
||
var log = document.getElementById('cacheWarmupLog');
|
||
log.textContent = '';
|
||
cacheWarmupAppendLog(
|
||
'[' + nowHHMMSS() + '] start trigger=' + data.trigger + ' total=' + data.total
|
||
);
|
||
cacheWarmupRefreshSnapshot();
|
||
}
|
||
|
||
function cacheWarmupOnRow(e) {
|
||
_cacheWarmupClearPollFallback();
|
||
var rs = JSON.parse(e.data);
|
||
cacheWarmupAppendLog(
|
||
'[' + nowHHMMSS() + '] ' + rs.status.padEnd(7) + ' ' + rs.table_id +
|
||
(rs.duration_ms ? ' (' + (rs.duration_ms / 1000).toFixed(1) + ' s)' : '') +
|
||
(rs.error ? ' ' + rs.error : '')
|
||
);
|
||
cacheWarmupSetRowBadge(rs.table_id, rs);
|
||
cacheWarmupRefreshSnapshot();
|
||
}
|
||
|
||
function cacheWarmupOnComplete(e) {
|
||
_cacheWarmupClearPollFallback();
|
||
var data = JSON.parse(e.data);
|
||
cacheWarmupAppendLog(
|
||
'[' + nowHHMMSS() + '] complete total=' + data.total +
|
||
' ok=' + data.completed + ' fail=' + data.failed
|
||
);
|
||
cacheWarmupRefreshSnapshot();
|
||
}
|
||
|
||
function cacheWarmupAppendLog(line) {
|
||
var log = document.getElementById('cacheWarmupLog');
|
||
if (!log) return;
|
||
log.textContent += line + '\n';
|
||
log.scrollTop = log.scrollHeight;
|
||
}
|
||
|
||
function cacheWarmupSetRowBadge(tableId, rs) {
|
||
document.querySelectorAll('tr').forEach(function(tr) {
|
||
var idCell = tr.querySelector('td.col-id');
|
||
if (!idCell || idCell.textContent.trim() !== tableId) return;
|
||
var statusCell = tr.querySelector('td.col-status');
|
||
if (!statusCell) return;
|
||
var color = {fresh: '#10B77F', warming: '#0073D1', pending: '#9CA3AF', error: '#EA580C'}[rs.status] || '#9CA3AF';
|
||
var label = rs.status === 'fresh' ? 'fresh' : rs.status;
|
||
// Build via DOM API so rs.error escapes safely into the title
|
||
// attribute (XSS guard — rs.error is server-derived, may contain
|
||
// quotes / angle brackets).
|
||
var span = document.createElement('span');
|
||
span.style.cssText =
|
||
'display:inline-block;padding:2px 6px;border-radius:3px;' +
|
||
'font-size:11px;background:' + color + ';color:white;';
|
||
if (rs.error) span.setAttribute('title', rs.error);
|
||
span.textContent = label;
|
||
statusCell.replaceChildren(span);
|
||
});
|
||
}
|
||
|
||
function nowHHMMSS() {
|
||
var d = new Date();
|
||
return d.toTimeString().slice(0, 8);
|
||
}
|
||
|
||
function cacheWarmupRun() {
|
||
var btn = document.getElementById('cacheWarmupRunBtn');
|
||
btn.disabled = true;
|
||
fetch('/api/admin/cache-warmup/run', {method: 'POST'})
|
||
.then(function(r) { return r.json(); })
|
||
.then(function() { /* SSE stream picks up the new run */ })
|
||
.catch(function() { btn.disabled = false; });
|
||
}
|
||
|
||
document.addEventListener('DOMContentLoaded', cacheWarmupInit);
|
||
|
||
</script>
|
||
{% include "_version_badge.html" %}
|
||
</body>
|
||
</html>
|