b4d3c576af
* docs(spec): admin observability spec + Activity Center MVP plan
Parent spec (480 lines) + executable plan (2295 lines, 14 TDD tasks).
Covers Activity Center rebuild (/admin/activity), with /admin/sessions
and /admin/feedback deferred to follow-up plans.
Already incorporates reviewer-pass revisions across three angles
(security, production resilience, code architecture):
- _get_db import path corrected to app.auth.dependencies
- Test fixtures aligned with seeded_app / admin_user / get_system_db
- All new audit writes wrapped in try/except + logger.exception
- Filename sanitization on session uploads
- DuckDB DESC index behavior documented; upgrade window flagged
- Migration idempotency + evolved-DB test cases
- reveal_raw + shared-cache multi-worker explicitly deferred
Targets schema v40 (audit_log gains params_before, client_ip,
client_kind, correlation_id + 3 indices).
* feat(db): schema v40 — audit_log gains params_before, client_ip, client_kind, correlation_id + 3 indices
* chore(test): clean up Task 1 — drop unused import, rename stale test
* feat(audit): AuditRepository.log() accepts params_before/client_ip/client_kind/correlation_id
* test(audit): strengthen params_before assertion to round-trip JSON content
* feat(audit): AuditRepository.query() rich filters + keyset cursor pagination
* feat(sync): SyncStateRepository.list_recent() cross-table feed
* feat(audit): POST /api/sync/trigger writes audit_log row
* feat(audit): POST /api/scripts/run-due writes audit_log row
* feat(audit): POST /api/upload/sessions writes audit_log row + sanitizes filename
* feat(audit): GET /api/data/{table_id}/download writes audit_log row
* feat(activity): /api/admin/activity timeline + /health + /sync endpoints
* feat(ui): /admin/activity rebuilt — health pulse, timeline, sync grid; /activity-center → 308 redirect
BREAKING: removed demo executive-pulse / maturity-roadmap content from activity_center.html.
The page now reflects real audit_log + sync_history data.
* feat(ui): admin nav + dashboard widget point at /admin/activity
* feat(activity): recursive-audit suppression for AC read endpoints (60s window per actor+filter)
* feat(activity): emit PostHog events when integration enabled (no-op default)
* fix(audit): move v40 indices out of _SYSTEM_SCHEMA + update test_repositories to unpack query() tuple
_SYSTEM_SCHEMA CREATE INDEX on audit_log(timestamp) failed when migration
tests hand-roll a bare audit_log (id, action) without the timestamp column.
Fix: remove indices from _SYSTEM_SCHEMA; add ADD COLUMN IF NOT EXISTS guards
for timestamp and other pre-v40 columns in _v39_to_v40() so the upgrade path
is safe on any hand-rolled schema; call _v39_to_v40 explicitly in the
fresh-install (current==0) path to restore index creation there.
Also unpack the (rows, next_cursor) tuple from AuditRepository.query() in
the three TestAuditRepository tests that still treated it as a list.
* docs: CHANGELOG entry for Activity Center MVP
* chore: refresh stale module docstring in app/api/activity.py
* feat(cli): agnes admin activity — terminal access to Activity Center (timeline + health + sync)
* fix(db): _v39_to_v40 — add IF NOT EXISTS guard for 'action' column
The v39→v40 ladder step adds defensive ADD COLUMN IF NOT EXISTS for
every audit_log column so a hand-rolled bare audit_log (id only) is
safe through the ladder. 'action' was missing from the guard list,
causing CREATE INDEX idx_audit_action_time to fail on tests that
stub audit_log with only an id column (tests/test_e2e_extract.py::
TestSchemaMigration::test_migration_preserves_and_extends).
Local 6/6 schema tests + the previously-failing CI test pass.
* docs(spec): platform telemetry epic — Boss directive + Activity Monitoring plan rebased onto v40 (stacked on zs/spec-activity-center)
* feat(db): schema v41 — 7 usage_* tables for telemetry (events, summary, rollups, attribution)
* chore(db): tighten v41 — usage_session_summary.session_id NOT NULL + upgrade test asserts all 7 tables
* feat(usage): UsageAttributionRepository — replace/delete/lookup over usage_attribution_* tables
* refactor(marketplace): extract list_inner_skills/agents/commands to src/marketplace_listing.py for reuse
* feat(usage): explode plugin attribution on marketplace sync + store entity write; backfill script
* refactor(marketplace): finish src/marketplace_listing.py extraction — drop duplicate _list_inner_* + _parse_frontmatter from app/api/marketplace.py
* feat(usage): promote attribution helpers to src/usage_attribution_helpers.py; hook update_entity rename + bundle-swap; clarify best-effort semantics
* feat(usage): UsageProcessor real extraction + rollup rebuild + 10 fixture-driven tests
* fix(usage): include tool_id in event hash + executemany + rollup transaction (critical multi-tool-turn drop fix)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(marketplace): popularity stats — invocations_30d + trend + sort=most_used|trending + Most Popular section
* feat(admin): /admin/users/<id> Sessions section — list + single-file + bulk-zip downloads (audit-logged)
* feat(usage): admin export endpoint + CLI — csv/json/parquet streaming, filters, audit-logged
* feat(usage): agnes admin ask — LLM Text-to-SQL over usage_events with SELECT-only validator (audit-logged)
* feat(usage): reprocess + prune endpoints + scheduler daily prune job + CLI
* docs: PLATFORM_SETUP.md operator playbook + HOWTO/ cookbook (5 guides + index)
Adds docs/PLATFORM_SETUP.md as a consolidated operator playbook covering
bootstrap, TLS, marketplaces (curated + flea), scheduler env vars, telemetry
extraction/export/ask/prune, privacy posture, and daily routine.
Adds docs/HOWTO/ with 5 analyst cookbook guides: first query, snapshots for
remote tables, private sessions, feedback + admin ask, and customizing skills.
Existing setup docs (QUICKSTART, DEPLOYMENT, ONBOARDING, HEADLESS_USAGE)
get a one-line cross-reference at the top pointing to PLATFORM_SETUP.md.
* docs(changelog): platform telemetry epic — usage_* foundation + surfaces + admin access + docs
Comprehensive [Unreleased] entry covering: usage_events/session_summary/
tool_daily/plugin_daily tables (v41), attribution lookup tables, backfill
script, marketplace Most Popular + invocation chips + sort, admin Sessions
section, export/ask/reprocess/prune endpoints + CLI mirrors, Activity Center
(v40), PLATFORM_SETUP.md + HOWTO/ docs, and operations notes for v41 upgrade.
* fix(security): block DuckDB read_*/http_*/glob functions in usage_ask validator + symlink escape guard in session zip + clarify mark-private semantics
* fix(admin): parquet export tempfile cleanup on COPY failure + correct processed-first sort on /admin/users/<id>/sessions
* feat(audit): close 8 production audit gaps — query (local/remote/hybrid), catalog/schema/sample, snapshot estimate/create, check-access
* feat(ui): /admin/usage summary dashboard + per-user activity tab on /admin/users/<id>
* fix(audit): cap error messages at 200 chars + audit user_activity reads + recursion guard on usage.summary
* fix(audit): catalog.list audits on error path + clean up deferred json import
* fix(ux): client_kind=cli for PAT auth + timeline empty state + email-instead-of-uuid + nav reorder + help text + loading indicators + ask doc
* feat(observability): unify /admin/activity into single page with saved views
- KPI cards (events, users, error rate, p95) clickable as quick-filters
- Faceted filter dropdowns populated from audit_log in the current window
- Sortable audit table, cursor pagination, per-row JSON side panel
- Saved views (schema v43: user_observability_views) — per-user state
- Top bar: window selector + 30s Live toggle + saved views dropdown
- /admin/scheduler-runs → 308 redirect (source=scheduler filter)
- New endpoints: /api/admin/observability/{facets,kpis,views}
* test: update activity + scheduler-runs tests for unified page
- test_admin_activity_page_renders asserts new structural anchors
- test_admin_scheduler_runs_page_admin_only asserts 308 redirect
* fix(observability): respect [hidden] on modal + side panel
CSS `display: flex` on .obs-modal beat the [hidden] attribute's UA
display:none, so the save-view modal rendered on page load and Cancel
clicks couldn't dismiss it. Gate the modal's flex layout on
:not([hidden]); add the same display:none guard prophylactically to
.obs-panel and .obs-views-panel.
* feat(observability): user enrichment in audit + interactive /admin/usage
Activity:
- /api/admin/activity now joins users for user_email + user_name per row
- User column renders "name (id-prefix)" or "email (id-prefix)" instead
of an opaque truncated UUID; falls back to id when the user record is
missing
Usage:
- /admin/usage rewritten as the same filter/group-by/search pattern as
/admin/activity. Faceted dropdowns (User / Tool / Source / Event type)
populated from usage_events; debounced free-text search across
tool_name / skill_name / subagent_type / command_name
- New endpoints /api/admin/usage/{facets,kpis,query}; the query endpoint
supports group_by in {day, username, tool_name, source, ref_id} with
sort + offset pagination, plus an ungrouped raw-events mode
- 4 KPI cards (events, distinct users, distinct tools, error rate) are
clickable quick-filters; clicking a grouped row applies the bucket as
a filter
- Old static `?window=7d|30d|all` server preload removed; all state is
client-side via since_minutes + group_by + filters in the URL
* fix(observability): clearer labels, all-column sort, drop saved views UI
- Rename page titles: "Activity" → "Server activity", "Usage" → "Tool usage"
with a one-line subtitle on each explaining what the page covers and
linking the other one. The two pages source different data (audit_log
vs usage_events) and the previous labels conflated them.
- Drop the saved-views dropdown + save modal from /admin/activity. The
modal pop-open bug was the trigger; the value wasn't there yet. The
/api/admin/observability/views CRUD + DuckDB table stay in place.
- Rename "Live (30s)" to "Auto-refresh (30s)" with a tooltip clarifying
that it's the re-fetch rate, not the time range. Time range now
labeled "Time range" instead of "Window".
- All audit-table columns are sortable (User, Source, Action, Resource,
Result added); sort is page-local with a Jinja comment explaining the
trade-off. Same for raw usage rows.
- Fix duplicate sort-arrow bug — the literal "▼" in the Time th HTML was
rendering alongside the CSS ::before arrow. Removed the literal; CSS
is the single source of truth.
* feat(observability): global Sessions browser + transcript viewer + CLI
Web:
- /admin/sessions — list every collected session JSONL across all users
with time-range, user, model, errors-only and free-text filters. Default
sort surfaces error-heavy sessions first. KPI cards (sessions, distinct
users, sessions w/ errors, tool error rate) clickable as quick-filters.
- /admin/sessions/<username>/<file> — transcript viewer rendering the
JSONL chronologically: user prompts, assistant text, tool calls (with
JSON input) and tool results (with flattened output). Errors get a red
border + chip and a "Next error" navigation button at the top.
- Admin dropdown gains a "Sessions" link.
API:
- GET /api/admin/sessions/{list,kpis,facets} — filtered cross-user reads
off usage_session_summary
- GET /api/admin/sessions/{username}/{file}/transcript — parses JSONL via
the existing services.session_pipeline.lib, returns chronological events
- GET /api/admin/sessions/{username}/{file}/download — JSONL stream, same
path-safety guards as the per-user endpoint, audit-logged
CLI:
- `agnes admin sessions list [--user X] [--errors] [--since 7d]` — table
output with `!` prefix on rows that hit a tool error
- `agnes admin sessions show <username> <file>` — transcript dump, with
`--errors` to print only the failed tool_result blocks
- `agnes admin sessions download <username> <file> [-o path]`
- `agnes admin sessions kpis` — top-level numbers
* feat(internal): expose telemetry tables to agnes query with row-level RBAC
Three new registered tables backed by system.duckdb, queryable through
the same /api/query plumbing analysts use for Keboola / BigQuery /
local sources:
agnes_sessions → usage_session_summary (filter: username)
agnes_usage → usage_events (filter: username)
agnes_audit → audit_log (filter: user_id)
RBAC is per-row, not per-table: admins see every user's rows; non-admins
see only their own. The filter is built server-side from the auth user
dict; non-admin filter values are regex-validated before SQL interpolation.
Implementation:
- new connector connectors/internal/ with access (filter+exec) + registry
(idempotent table_registry seed at startup)
- /api/query detects internal table refs and short-circuits to a CTE
wrapper that prepends "WITH agnes_x AS (SELECT * FROM <src> WHERE …),
…" then "SELECT * FROM (<user_sql>) AS _q". DuckDB cursor on the
shared system.duckdb handle — opening parallel handles / ATTACH on the
same file is blocked process-wide.
- mixing internal + BQ / registered local tables in one SELECT is
rejected (v1 limitation)
- src.rbac.can_access_table waves internal tables through for all
authenticated users; row scoping is the actual security control
- /api/v2/schema and /api/v2/sample gained internal branches; sample
intentionally skips its cache because rows are RBAC-scoped per caller
- audit row written as action='query.internal' with is_admin flag
Tests: connectors/internal/access — RBAC, filter clause, schema, CTE
wrapper coexistence with user-supplied aggregations, unsafe-username
rejection. 16/16 passing.
Motivating queries this enables:
SELECT tool_name, COUNT(*) FROM agnes_usage
WHERE is_error GROUP BY 1 ORDER BY 2 DESC
-- analyst self-introspection: which tools fail for me?
SELECT user_id, COUNT(*) FROM agnes_audit
WHERE action = 'session.transcript_view' GROUP BY 1
-- admin: who's been looking at whose session transcripts?
* feat(admin): group dropdown into 5 named sections + internal tables in /catalog
Admin dropdown gains section headers so admins can land on the right
page without re-reading the full menu:
Activity Center Server activity / Tool usage / Sessions
Users & Access Users / Groups / Resource access / Tokens
Data Tables
Agent Experience Curated Marketplaces / Flea Submissions /
Agent Setup Prompt / Agent Workspace Prompt
Server Server config
"Agent Experience" frames the curated content + prompts as one cluster
— it's all admin-controlled material that shapes what an analyst's AI
agent encounters. "Configuration" → "Server" since only one item lives
there now.
Renamed the section's first two items:
"Activity" → "Server activity" (matches page H1)
"Usage" → "Tool usage"
Also fixes /catalog visibility of the internal tables (agnes_sessions /
_usage / _audit) for non-admin users: ``app.auth.access.can_access``
short-circuits to True for resource_type='table' + an internal-table id.
Without this, non-admins saw the tables in /api/v2/catalog (which uses
the same RBAC bypass) but not on the /catalog HTML page (which calls
can_access directly, requiring a resource_grants row internal tables
don't have).
CSS for `.app-nav-menu-section`: small caps, muted, non-clickable; first
section trims top padding so the panel doesn't open with an awkward gap.
* refactor(admin): move corporate memory into Admin > Agent Experience
Memory link was the only admin-only entry in the primary nav (gated by
session.user.is_admin). Moves it into the Admin dropdown under Agent
Experience, alongside Curated Marketplaces / Flea Submissions / Prompts
— all admin-curated content that shapes what an analyst's AI agent
encounters.
Renamed the nav label to "Shared Knowledge" to match what the page
actually is (admin-curated organisational knowledge from session
verification, surfaced to agents). URL stays at /corporate-memory; the
route still gates on require_admin per the existing comment.
Side effect: primary nav (Home / Marketplace / Data Packages) is now
uniform for every authenticated user — no conditional admin-only entry.
* ui: rename admin entries to Curated Knowledge / Init Prompt / Workspace Prompt
- "Shared Knowledge" → "Curated Knowledge" (parallel with "Curated
Marketplaces" in the same Agent Experience section; "curated" tells
the admin what they do there — review + approve)
- "Agent Setup Prompt" → "Init Prompt" (matches the `agnes init` flow
it actually drives)
- "Agent Workspace Prompt" → "Workspace Prompt" (the "Agent" prefix
was redundant — every item in the section is agent-facing)
Renames page titles + H1s on /admin/agent-prompt and
/admin/workspace-prompt to match.
* refactor: rename Usage → Telemetry across user-facing surfaces
External surfaces all switch; internal Python module / file names and the
physical DB tables (usage_events, usage_session_summary, usage_tool_daily,
usage_plugin_daily) stay — renaming them would force a schema migration
+ a redo of the LLM Text-to-SQL prompt for no analyst-visible win.
Changes:
- Admin dropdown: "Tool usage" → "Telemetry"
- Page H1 / <title>: same
- URL: /admin/usage → /admin/telemetry; old URL 308-redirects
- API prefix: /api/admin/usage/* → /api/admin/telemetry/*
- CLI: primary command `agnes admin telemetry …`; `agnes admin usage` kept
as a deprecated alias so existing operator scripts keep working
- Internal data-source table id: agnes_usage → agnes_telemetry. The
registry seed now evicts any stale internal-source row whose id no
longer matches INTERNAL_TABLES, so the old `agnes_usage` row is
removed from table_registry on next app boot
- All tests + JS endpoint paths updated
* test(rbac): include auto-appended internal tables in expectations
get_accessible_tables now appends agnes_sessions / agnes_telemetry /
agnes_audit to every authenticated user's accessible-tables list so the
internal data source shows up in /catalog. The two existing rbac tests
asserted hardcoded list shapes that pre-dated the change.
Rewritten to assert "granted tables + the canonical internal-table set"
instead of literal lists, so the test stays correct if the internal
table roster changes again later.
* ui: visual dividers between admin-dropdown sections
Adds a 1px top border + 6px top margin to every section header except
the first, so the five named groups (Activity Center, Users & Access,
Data, Agent Experience, Server) read as visually separated clusters.
The header itself stays small-caps + muted as before — the border is
additive.
* ui(memory): match obs-topbar visual on /corporate-memory
The Curated Knowledge page (linked from the admin dropdown's Agent
Experience section) opened straight into the stats bar — no title,
no subtitle, no shared chrome with the other admin pages. Adds an
obs-topbar-style header at the top of .container-memory:
- H1 "Curated Knowledge"
- subtitle explaining what the page is + how AI agents pull from it
The `.ck-*` class set duplicates the inline obs-* styles from
/admin/activity etc. for this one page; promoting the obs-* class set
to style-custom.css for shared reuse is the obvious next step (4 pages
already inline the same CSS), tracked as a follow-up.
Page <title> also renamed from "Corporate Memory" → "Curated Knowledge".
* ui(tables): list Agnes internal tables in /admin/tables + group in /catalog
/admin/tables previously rendered three per-source-type listings
(BQ / Keboola / Jira) and dropped any row whose source_type didn't
match — so the agnes_sessions / agnes_telemetry / agnes_audit rows
seeded into table_registry were invisible. Adds a fourth read-only
section "Agnes internal tables" that filters source_type === 'internal'
and renders the same registry-table layout the other sections use,
with two changes:
- no Register button (these rows are seeded on every app boot from
connectors/internal/registry.py)
- Edit + Delete actions hidden (any change would be reverted on the
next start). Manage access stays so admins can still inspect.
Mode badge picks up a new mode-internal CSS class (teal accent) so the
display doesn't lie and call it "local".
In /catalog, internal tables now group under an "agnes" accordion
section (bucket="agnes" on seed) instead of falling into the catch-all
"default". Single source of truth for which tables exist; admins find
them where they expect.
* ui(tables): Agnes internal as a 4th tab next to BQ/Keboola/Jira
Previous iteration mounted the internal-table listing as a separate
standalone card under the tab strip. Reshapes it to a proper
tab-content section so admins switch between data sources via one
consistent nav (BigQuery / Keboola / Jira / Agnes internal).
- New tab button "Agnes internal" in the tab-nav.
- The listing card becomes <section id="tab-content-internal"
class="tab-content">; switchTab() already routes by id so no JS
change beyond extending the hash allowlist for direct #internal
links.
- Tab content keeps the read-only treatment from the previous commit
(no Register button, no Edit / Delete in renderRegistryListing).
* ui: rename Curated Knowledge → Curated Memory
Settles the naming back on "Curated Memory" — parallel structure with
"Curated Marketplaces" in the same Agent Experience section, and zero
rename ripple: URL (/corporate-memory), API (/api/memory/*), CLI
(agnes admin memory), and Python modules all stay on "memory" so the
admin label finally lines up with the underlying surfaces.
The "Curated" prefix still tells admins what they do on the page
(review pending → approve / mandate / reject) and reads as a sibling
of "Curated Marketplaces" right next to it in the dropdown.
Touches: admin dropdown label, page <title>, page H1. DB tables stay
on knowledge_* (already the canonical naming for the data shape).
* ui: rename "Server activity" → "Audit log"
"Audit log" is what the page actually is — server-side audit_log table
rendered with KPI cards + filter bar + sortable table. The "Server
activity" label confused the term with Claude Code session telemetry
(Telemetry page) and didn't make the source/concept clear.
Touches:
- Admin dropdown nav label
- /admin/activity page H1 + subtitle
- /admin/telemetry subtitle cross-link
- test_activity_api page-renders assertion
URL (/admin/activity) and API (/api/admin/activity/*) stay — the
"activity" name has stuck at the route layer for a year; rerouting
those would churn dashboards/bookmarks for zero analyst-visible win.
* ui(admin-nav): gray band on each section header for clearer separation
Previous iteration used a 1px top border between section labels — the
labels still blended into the items above/below at a glance. Switches
to a light gray background band per section header, extended edge-to-
edge inside the panel via negative horizontal margins. Bolder
font-weight (700) reinforces the separation; bumping the font color
isn't needed because the band itself does the work.
First section's header tucks into the panel's top border-radius so the
band reaches the corners without a gap.
* ui(catalog): rename internal-table category to "Agnes Internal"
`bucket` is what /catalog renders as the accordion category header
verbatim — "agnes" lowercase didn't read as a real category name and
got confused with a system identifier. Bumps to "Agnes Internal".
Seed re-applies on every app boot so existing rows pick up the new
bucket value via `ON CONFLICT (id) DO UPDATE`.
* ui(catalog): split Agnes Internal into its own card on /catalog
Previously the three internal tables landed inside the "Core Business
Data" card under an "Agnes Internal" accordion alongside Keboola / BQ
buckets — readers conflated system telemetry with business datasets,
and the data_stats header counter ("3 tables · ~X rows total") only
ever counted synced rows so internal tables looked invisible.
Split the catalog page into two cards:
- Core Business Data: only non-internal source_types (Keboola, BQ,
Jira). Accordions group by bucket as before. Stats counter reflects
this card's tables.
- Agnes Internal: a dedicated card with its own visual treatment
(teal accent matching the mode-internal badge in /admin/tables).
Flat list (no accordion — only 3 rows, never grows here), each
row carries the canonical `agnes query` snippet. Read-only — no
profiler click, no In-stack toggle, no sync metadata.
Route adds `internal_card` context object; template renders the new
card only when it's non-None.
* fix(rbac): hide internal tables from /admin/access + drop "my" framing
Two related cleanups for the Agnes-internal tables:
1. /admin/access (resource grants) no longer lists them. The
`can_access` check has a hardcoded internal-table bypass — security
is row-level (per-request view filter), so a table-grain
`resource_grants` row would do nothing. Surfacing them in the UI
let admins set up grants that silently no-op. Filter at the
`_table_blocks` projection so the UI tree never sees them.
2. Display names drop the analyst-perspective "my" framing:
"Agnes — my sessions" → "Agnes sessions"
"Agnes — my telemetry events" → "Agnes telemetry events"
"Agnes — my audit log" → "Agnes audit log"
The "my" only makes sense from the querying analyst's seat
(`SELECT … FROM agnes_sessions` returns *their* rows); on /admin/*
pages where admin sees / configures them across users, the
pronoun was misleading. Description text now spells out the
row-level RBAC contract explicitly.
Display names update via TableRegistryRepository.register's ON CONFLICT
UPDATE on next app boot; no manual cleanup needed.
* ui: subtitle notes about agnes_* tables on each Activity Center page
The recursive observability story — Agnes serves its own audit /
telemetry / session data through the same `agnes query` plumbing
analysts use for business data — wasn't surfaced anywhere on the
admin pages that show that data. Three pages get a one-liner with
the canonical `agnes query` snippet + the RBAC contract (analysts
see their own rows, admin sees all):
- /admin/activity (Audit log) → agnes_audit
- /admin/telemetry (Tool usage) → agnes_telemetry
- /admin/sessions → agnes_sessions
Sets up the discovery moment for admins: they're reading the page,
they see "you can query this from Claude Code", they remember it
when an analyst asks "how do I find my own failed tool calls?".
* ui(tables): explain "Show log" empty-state on /admin/tables
Cache warmup log <pre> renders with a dark background and is only
populated by the SSE stream during a Re-warm all run. Opening the
page cold + clicking Show log just revealed a black bar with no
context — admins couldn't tell what they were looking at.
Adds an inline paragraph above the <pre> explaining what the log is,
the row format, when it fills in, and where to find the historical
audit trail (/admin/activity). The actual <pre> stays empty until
SSE events arrive, but the surrounding copy carries the meaning.
* ui(tables): auto-open cache-warmup log on Re-warm all click
A Re-warm all run takes ~24s per remote BQ row. With the <details>
collapsed by default, operators saw the button disable, watched a
quiet ~24s pass, and assumed nothing had happened — the streaming
log was hidden behind a closed disclosure.
Two small JS tweaks:
- cacheWarmupRun() opens the details on click, so streamed lines
appear without an extra interaction
- cacheWarmupOnStart() hides the inline hint paragraph the moment
real log content lands, so the dark log block isn't competing
with redundant context
Hint paragraph also clarifies that only `query_mode='remote'` BQ
rows are warmed — operators with only materialized/internal tables
would see total=0 and the page would "do nothing" by spec.
* ui: trim Agnes internal copy across surfaces
Descriptions had grown to explain the extraction pipeline ("parsed
out of session JSONLs"), the underlying table ("Backed by
usage_session_summary"), the RBAC mechanic ("row-level RBAC at query
time — analysts see their own; admin sees all"), and the SQL snippet.
Every implementation detail meant another rewrite on the next iter.
Strips to one stable line per surface: what the data is, plus
"Also available locally for analysis". Mechanics live in code +
docs; the page copy says what the user needs to know.
Touched:
- connectors/internal/access.py: INTERNAL_TABLES descriptions
- activity_center.html / admin_usage.html / admin_sessions.html
subtitles
- catalog.html Agnes Internal card description + row strip
- admin_tables.html "Agnes internal" tab hint
* fix(internal): is_user_admin arity bugs + + saved-view payload cap
Round-1 code review (PR #278) caught two blocking bugs and three nits.
Blocking — both `is_user_admin(user)` (single dict arg) calls raised
TypeError. is_user_admin signature is `(user_id, conn)`. Affected:
- app/api/query.py:_run_internal_query — every POST /api/query that
references agnes_sessions / agnes_telemetry / agnes_audit blew up
with a 500. The headline analyst-facing feature of this PR was
unusable through the API.
- app/api/v2_sample.py — same shape; `GET /api/v2/sample/agnes_*`
returned 500.
Both fixed to call `is_user_admin(user.get("id"), conn)`. Added two
FastAPI-level tests in test_internal_data_source.py that go through
the TestClient — the existing unit tests on `execute_internal_query`
and `build_filter_clause` skipped the request-handler layer where the
bugs lived, which is why this landed.
Nits also closed:
- connectors/internal/access.py: `+` allowed in _USERNAME_RE /
_USER_ID_RE so RFC 5321 email local-parts (alice+test@x) resolve
correctly without hitting InternalAccessError.
- app/api/observability.py: saved-view payload capped at 64 KiB to
prevent an admin from bloating system.duckdb with a malformed save.
* fix(security): close non-admin data-leak via underlying-table refs
PR #278 R2 review surfaced a non-admin-exploitable bypass: SQL whose
string literal contains 'agnes_sessions' routed into the privileged
internal-query path, then queried the underlying physical table
(usage_session_summary / usage_events / audit_log) directly, escaping
the CTE wrapper's row filter. Two reinforcing defenses:
1. find_internal_refs() now strips single-quoted string literals
before scanning for alias names — a literal alone no longer
routes the request into the privileged code path.
2. execute_internal_query() rejects non-admin SQL that references
the underlying physical tables (usage_*, audit_log). The CTE
wrapper only scopes the agnes_* aliases; a direct FROM on the
base table — or a shadowing inner WITH that still has to read
the base table — bypasses RBAC. Block before execution with an
actionable error pointing to the agnes_* alias. Admins are
unaffected (god-mode short-circuit on the filter clause).
3. tests/test_internal_data_source.py — three new negative tests
covering literal-only matches, direct-table refs, and CTE
shadow attempts.
Also tightens usage_ask.py's SELECT-only validator: pragma_table_info,
pragma_storage_info, pragma_database_*, and duckdb_tables / columns /
views / indexes / schemas are reflection functions that leak metadata
the analyst question shouldn't reach. \bPRAGMA\b in _FORBIDDEN never
matched the function-call form (word-boundary between `A` and `_`).
* fix(security): dynamic denylist for non-admin internal queries
R3 review (PR #278) caught a wider data-leak than R2: the underlying-
physical-table guard listed only the 7 usage_* + audit_log tables,
but system.duckdb has 30+ other sensitive tables — users (emails +
ids), personal_access_tokens, resource_grants, user_groups,
user_observability_views, store_*, marketplace_*, knowledge_*, etc.
A non-admin SQL like
SELECT * FROM agnes_sessions
UNION ALL SELECT email, id, … FROM users LIMIT 1
would leak every user's row.
Replaces the hardcoded denylist with a **dynamic allowlist** —
non-admin SQL may reference ONLY the registered agnes_* aliases.
Every other table in `information_schema.tables` (main schema) is
rejected. Future migrations that add a new sensitive table are
automatically covered without re-editing this module.
Also strips SQL comments (`/* */` and `--`) before the identifier
scan so a comment-wrapped table name (`/**/users/**/`) can't slip
past the regex.
Four new negative tests pin: `users`, `personal_access_tokens`,
block-comment wrap, line-comment wrap.
Plus: per-user view-count cap (100) on /api/admin/observability/views
so an admin can't fill system.duckdb with thousands of saved views.
* release: 0.54.0 — Activity Center + Telemetry + Sessions + internal datasource
Cuts the work shipped across this PR (Activity Center build, recursive
internal data source) into a versioned release. Bumps pyproject.toml
to 0.54.0; renames the top of CHANGELOG.md from [Unreleased] to
[0.54.0] — 2026-05-12 with a header summary; opens a fresh
[Unreleased] section for the next round.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
3563 lines
167 KiB
HTML
3563 lines
167 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en">
|
||
<head>
|
||
<meta charset="UTF-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
<title>Table Management - {{ config.INSTANCE_NAME }}</title>
|
||
{% if not config.THEME_FONT_URL %}
|
||
<link rel="preconnect" href="https://fonts.googleapis.com">
|
||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap" rel="stylesheet">
|
||
{% endif %}
|
||
<link rel="stylesheet" href="{{ url_for('static', filename='style-custom.css') }}">
|
||
<style>
|
||
:root {
|
||
/* Colors - Design System */
|
||
--primary: #0073D1;
|
||
--primary-light: rgba(0, 115, 209, 0.1);
|
||
--text-primary: #1A253C;
|
||
--text-secondary: #6B7280;
|
||
--background: #F5F7FA;
|
||
--surface: #FFFFFF;
|
||
--border: #E5E7EB;
|
||
--border-light: #F3F4F6;
|
||
--success: #10B77F;
|
||
--success-light: rgba(16, 183, 127, 0.1);
|
||
--warning: #F59F0A;
|
||
--error: #EA580C;
|
||
--error-light: rgba(234, 88, 12, 0.1);
|
||
|
||
/* Typography */
|
||
--font-primary: 'Inter', system-ui, sans-serif;
|
||
--font-mono: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace;
|
||
|
||
/* Shadows */
|
||
--shadow-sm: rgba(0, 0, 0, 0.05) 0px 1px 2px 0px;
|
||
--shadow-md: rgba(0, 0, 0, 0.1) 0px 4px 6px -1px;
|
||
}
|
||
|
||
* {
|
||
margin: 0;
|
||
padding: 0;
|
||
box-sizing: border-box;
|
||
}
|
||
|
||
body {
|
||
font-family: var(--font-primary);
|
||
font-size: 14px;
|
||
color: var(--text-primary);
|
||
background: var(--background);
|
||
line-height: 1.5;
|
||
}
|
||
|
||
/* ── Header (dashboard-style) ── */
|
||
.header {
|
||
background: var(--surface);
|
||
border-bottom: 1px solid var(--border);
|
||
padding: 0 32px;
|
||
height: 72px;
|
||
display: flex;
|
||
align-items: center;
|
||
justify-content: space-between;
|
||
position: sticky;
|
||
top: 0;
|
||
z-index: 100;
|
||
}
|
||
|
||
.header-left {
|
||
display: flex;
|
||
align-items: center;
|
||
gap: 16px;
|
||
}
|
||
|
||
.header-back {
|
||
display: flex;
|
||
align-items: center;
|
||
justify-content: center;
|
||
width: 32px;
|
||
height: 32px;
|
||
border-radius: 6px;
|
||
color: var(--text-secondary);
|
||
text-decoration: none;
|
||
transition: all 0.15s ease;
|
||
}
|
||
|
||
.header-back:hover {
|
||
background: var(--border-light);
|
||
color: var(--text-primary);
|
||
}
|
||
|
||
.header-logo-group {
|
||
display: flex;
|
||
flex-direction: column;
|
||
justify-content: center;
|
||
gap: 2px;
|
||
}
|
||
|
||
.header-logo svg {
|
||
display: block;
|
||
}
|
||
|
||
.header-subtitle {
|
||
font-size: 11px;
|
||
font-weight: 500;
|
||
color: var(--text-secondary);
|
||
letter-spacing: 0.4px;
|
||
text-transform: uppercase;
|
||
margin-top: 2px;
|
||
}
|
||
|
||
.header-right {
|
||
font-size: 12px;
|
||
color: var(--text-secondary);
|
||
}
|
||
|
||
/* ── Page Title ── */
|
||
.page-title {
|
||
max-width: 1600px;
|
||
margin: 0 auto;
|
||
padding: 32px 24px 24px;
|
||
}
|
||
|
||
.page-title h1 {
|
||
font-size: 24px;
|
||
font-weight: 600;
|
||
color: var(--text-primary);
|
||
margin-bottom: 4px;
|
||
}
|
||
|
||
.page-title p {
|
||
font-size: 14px;
|
||
color: var(--text-secondary);
|
||
}
|
||
|
||
/* ── Content Layout ── */
|
||
.content {
|
||
max-width: 1600px;
|
||
margin: 0 auto;
|
||
padding: 0 24px 32px;
|
||
display: flex;
|
||
flex-direction: column;
|
||
gap: 24px;
|
||
}
|
||
|
||
/* ── Panel (shared card style) ── */
|
||
.panel {
|
||
background: var(--surface);
|
||
border: 1px solid var(--border);
|
||
border-radius: 12px;
|
||
overflow: hidden;
|
||
box-shadow: var(--shadow-sm);
|
||
}
|
||
|
||
.panel-header {
|
||
display: flex;
|
||
align-items: center;
|
||
justify-content: space-between;
|
||
padding: 20px 24px;
|
||
border-bottom: 1px solid var(--border-light);
|
||
}
|
||
|
||
.panel-header-left {
|
||
display: flex;
|
||
align-items: center;
|
||
gap: 12px;
|
||
}
|
||
|
||
.panel-header-icon {
|
||
width: 36px;
|
||
height: 36px;
|
||
border-radius: 8px;
|
||
display: flex;
|
||
align-items: center;
|
||
justify-content: center;
|
||
flex-shrink: 0;
|
||
background: var(--primary-light);
|
||
}
|
||
|
||
.panel-title {
|
||
font-size: 16px;
|
||
font-weight: 600;
|
||
color: var(--text-primary);
|
||
}
|
||
|
||
.panel-subtitle {
|
||
font-size: 13px;
|
||
color: var(--text-secondary);
|
||
margin-top: 1px;
|
||
}
|
||
|
||
.panel-body {
|
||
padding: 20px 24px;
|
||
}
|
||
|
||
.panel-body-empty {
|
||
padding: 40px 24px;
|
||
text-align: center;
|
||
color: var(--text-secondary);
|
||
font-size: 13px;
|
||
}
|
||
|
||
/* ── Buttons ── */
|
||
.btn {
|
||
display: inline-flex;
|
||
align-items: center;
|
||
gap: 6px;
|
||
padding: 8px 16px;
|
||
border: none;
|
||
border-radius: 8px;
|
||
font-family: var(--font-primary);
|
||
font-size: 13px;
|
||
font-weight: 500;
|
||
cursor: pointer;
|
||
transition: all 0.15s ease;
|
||
white-space: nowrap;
|
||
}
|
||
|
||
.btn-primary {
|
||
background: var(--primary);
|
||
color: white;
|
||
}
|
||
|
||
.btn-primary:hover {
|
||
background: #005FA8;
|
||
}
|
||
|
||
.btn-primary:disabled {
|
||
opacity: 0.6;
|
||
cursor: not-allowed;
|
||
}
|
||
|
||
.btn-secondary {
|
||
background: var(--border-light);
|
||
color: var(--text-primary);
|
||
}
|
||
|
||
.btn-secondary:hover {
|
||
background: var(--border);
|
||
}
|
||
|
||
.btn-danger {
|
||
background: var(--error-light);
|
||
color: var(--error);
|
||
}
|
||
|
||
.btn-danger:hover {
|
||
background: rgba(234, 88, 12, 0.2);
|
||
}
|
||
|
||
.btn-sm {
|
||
padding: 5px 10px;
|
||
font-size: 12px;
|
||
border-radius: 6px;
|
||
}
|
||
|
||
.btn-icon {
|
||
width: 28px;
|
||
height: 28px;
|
||
padding: 0;
|
||
display: flex;
|
||
align-items: center;
|
||
justify-content: center;
|
||
border-radius: 6px;
|
||
background: none;
|
||
border: none;
|
||
cursor: pointer;
|
||
color: var(--text-secondary);
|
||
transition: all 0.15s ease;
|
||
}
|
||
|
||
.btn-icon:hover {
|
||
background: var(--border-light);
|
||
color: var(--text-primary);
|
||
}
|
||
|
||
.btn-icon.danger:hover {
|
||
background: var(--error-light);
|
||
color: var(--error);
|
||
}
|
||
|
||
/* ── Badges ── */
|
||
.badge {
|
||
flex-shrink: 0;
|
||
font-size: 11px;
|
||
font-weight: 500;
|
||
border-radius: 6px;
|
||
padding: 3px 8px;
|
||
white-space: nowrap;
|
||
}
|
||
|
||
.badge-registered {
|
||
color: #065F46;
|
||
background: #D1FAE5;
|
||
}
|
||
|
||
.badge-available {
|
||
color: var(--primary);
|
||
background: var(--primary-light);
|
||
}
|
||
|
||
/* ── Spinner ── */
|
||
.spinner {
|
||
display: inline-block;
|
||
width: 18px;
|
||
height: 18px;
|
||
border: 2px solid var(--border);
|
||
border-top-color: var(--primary);
|
||
border-radius: 50%;
|
||
animation: spin 0.6s linear infinite;
|
||
}
|
||
|
||
@keyframes spin {
|
||
to { transform: rotate(360deg); }
|
||
}
|
||
|
||
.spinner-lg {
|
||
width: 32px;
|
||
height: 32px;
|
||
border-width: 3px;
|
||
}
|
||
|
||
/* ── Loading state ── */
|
||
.loading-state {
|
||
display: flex;
|
||
flex-direction: column;
|
||
align-items: center;
|
||
gap: 12px;
|
||
padding: 40px 24px;
|
||
color: var(--text-secondary);
|
||
font-size: 13px;
|
||
}
|
||
|
||
/* ── Notification toast ── */
|
||
.toast {
|
||
position: fixed;
|
||
top: 84px;
|
||
right: 24px;
|
||
z-index: 200;
|
||
background: var(--surface);
|
||
border: 1px solid var(--border);
|
||
border-radius: 10px;
|
||
box-shadow: var(--shadow-md);
|
||
padding: 12px 16px;
|
||
display: flex;
|
||
align-items: center;
|
||
gap: 10px;
|
||
font-size: 13px;
|
||
transform: translateX(120%);
|
||
transition: transform 0.3s ease;
|
||
max-width: 360px;
|
||
}
|
||
|
||
.toast.visible {
|
||
transform: translateX(0);
|
||
}
|
||
|
||
.toast-success {
|
||
border-left: 3px solid var(--success);
|
||
}
|
||
|
||
.toast-error {
|
||
border-left: 3px solid var(--error);
|
||
}
|
||
|
||
.toast-icon {
|
||
width: 20px;
|
||
height: 20px;
|
||
flex-shrink: 0;
|
||
}
|
||
|
||
/* ── Bucket accordion ── */
|
||
.bucket-group {
|
||
border-top: 1px solid var(--border-light);
|
||
}
|
||
|
||
.bucket-group:first-child {
|
||
border-top: none;
|
||
}
|
||
|
||
.bucket-trigger {
|
||
width: 100%;
|
||
display: flex;
|
||
align-items: center;
|
||
gap: 10px;
|
||
padding: 12px 24px;
|
||
background: none;
|
||
border: none;
|
||
cursor: pointer;
|
||
font-family: var(--font-primary);
|
||
font-size: 14px;
|
||
font-weight: 500;
|
||
color: var(--text-primary);
|
||
text-align: left;
|
||
transition: background 0.1s ease;
|
||
}
|
||
|
||
.bucket-trigger:hover {
|
||
background: var(--border-light);
|
||
}
|
||
|
||
.bucket-chevron {
|
||
width: 16px;
|
||
height: 16px;
|
||
color: var(--text-secondary);
|
||
transition: transform 0.2s ease;
|
||
flex-shrink: 0;
|
||
}
|
||
|
||
.bucket-trigger.expanded .bucket-chevron {
|
||
transform: rotate(90deg);
|
||
}
|
||
|
||
.bucket-count {
|
||
font-size: 11px;
|
||
font-weight: 500;
|
||
color: var(--text-secondary);
|
||
background: var(--border-light);
|
||
padding: 1px 7px;
|
||
border-radius: 9999px;
|
||
margin-left: auto;
|
||
}
|
||
|
||
.bucket-content {
|
||
display: none;
|
||
}
|
||
|
||
.bucket-content.expanded {
|
||
display: block;
|
||
}
|
||
|
||
/* ── Table row (discovery results) ── */
|
||
.table-item {
|
||
display: flex;
|
||
align-items: center;
|
||
padding: 10px 24px 10px 50px;
|
||
border-top: 1px solid var(--border-light);
|
||
gap: 12px;
|
||
transition: background 0.1s ease;
|
||
}
|
||
|
||
.table-item:hover {
|
||
background: rgba(243, 244, 246, 0.5);
|
||
}
|
||
|
||
.table-item-info {
|
||
flex: 1;
|
||
min-width: 0;
|
||
}
|
||
|
||
.table-item-name {
|
||
font-weight: 500;
|
||
font-family: var(--font-mono);
|
||
font-size: 13px;
|
||
color: var(--text-primary);
|
||
}
|
||
|
||
.table-item-meta {
|
||
font-size: 12px;
|
||
color: var(--text-secondary);
|
||
margin-top: 1px;
|
||
display: flex;
|
||
gap: 12px;
|
||
flex-wrap: wrap;
|
||
}
|
||
|
||
.table-item-actions {
|
||
display: flex;
|
||
align-items: center;
|
||
gap: 8px;
|
||
flex-shrink: 0;
|
||
}
|
||
|
||
/* ── Registry table ── */
|
||
.registry-table {
|
||
width: 100%;
|
||
border-collapse: collapse;
|
||
table-layout: fixed;
|
||
}
|
||
|
||
.registry-table .col-description {
|
||
overflow: hidden;
|
||
display: -webkit-box;
|
||
-webkit-line-clamp: 2;
|
||
-webkit-box-orient: vertical;
|
||
overflow-wrap: anywhere;
|
||
line-height: 1.4;
|
||
color: var(--text-secondary);
|
||
}
|
||
|
||
.registry-table th {
|
||
text-align: left;
|
||
font-size: 11px;
|
||
font-weight: 600;
|
||
color: var(--text-secondary);
|
||
text-transform: uppercase;
|
||
letter-spacing: 0.4px;
|
||
padding: 10px 16px;
|
||
border-bottom: 1px solid var(--border);
|
||
background: var(--background);
|
||
}
|
||
|
||
.registry-table td {
|
||
padding: 12px 16px;
|
||
font-size: 13px;
|
||
border-bottom: 1px solid var(--border-light);
|
||
vertical-align: middle;
|
||
}
|
||
|
||
.registry-table tr:last-child td {
|
||
border-bottom: none;
|
||
}
|
||
|
||
.registry-table tr:hover td {
|
||
background: rgba(243, 244, 246, 0.5);
|
||
}
|
||
|
||
.registry-table .col-id {
|
||
font-family: var(--font-mono);
|
||
font-size: 12px;
|
||
max-width: 220px;
|
||
overflow: hidden;
|
||
text-overflow: ellipsis;
|
||
white-space: nowrap;
|
||
}
|
||
|
||
.registry-table .col-actions {
|
||
width: 120px;
|
||
min-width: 120px;
|
||
white-space: nowrap;
|
||
vertical-align: top;
|
||
}
|
||
|
||
/* ── Registry table — wide layout ── */
|
||
.registry-table .col-mode {
|
||
width: 100px;
|
||
}
|
||
|
||
.registry-table .col-source {
|
||
width: 200px;
|
||
font-family: var(--font-mono);
|
||
font-size: 12px;
|
||
overflow: hidden;
|
||
text-overflow: ellipsis;
|
||
white-space: nowrap;
|
||
color: var(--text-secondary);
|
||
}
|
||
|
||
.registry-table .col-pk {
|
||
width: 100px;
|
||
overflow: hidden;
|
||
text-overflow: ellipsis;
|
||
white-space: nowrap;
|
||
}
|
||
|
||
.registry-table .col-schedule {
|
||
width: 100px;
|
||
font-size: 12px;
|
||
color: var(--text-secondary);
|
||
}
|
||
|
||
.registry-table .col-folder {
|
||
width: 120px;
|
||
}
|
||
|
||
.registry-table .col-registered {
|
||
width: 160px;
|
||
font-size: 11px;
|
||
line-height: 1.4;
|
||
overflow: hidden;
|
||
}
|
||
|
||
.registry-table .col-registered .registered-by {
|
||
color: var(--text-primary);
|
||
overflow: hidden;
|
||
text-overflow: ellipsis;
|
||
white-space: nowrap;
|
||
}
|
||
|
||
.registry-table .col-registered .registered-at {
|
||
color: var(--text-secondary);
|
||
}
|
||
|
||
.registry-table .col-status {
|
||
width: 40px;
|
||
text-align: center;
|
||
}
|
||
|
||
.mode-badge {
|
||
display: inline-block;
|
||
padding: 2px 8px;
|
||
border-radius: 4px;
|
||
font-size: 11px;
|
||
font-weight: 600;
|
||
}
|
||
|
||
.mode-badge.mode-local {
|
||
background: var(--success-light);
|
||
color: var(--success);
|
||
}
|
||
|
||
.mode-badge.mode-remote {
|
||
background: var(--primary-light);
|
||
color: var(--primary);
|
||
}
|
||
|
||
.mode-badge.mode-materialized {
|
||
background: rgba(139, 92, 246, 0.1);
|
||
color: #8B5CF6;
|
||
}
|
||
|
||
.mode-badge.mode-internal {
|
||
background: rgba(15, 118, 110, 0.1);
|
||
color: #0F766E;
|
||
}
|
||
|
||
.folder-badge {
|
||
display: inline-block;
|
||
padding: 1px 6px;
|
||
border-radius: 3px;
|
||
background: var(--background);
|
||
border: 1px solid var(--border);
|
||
font-size: 11px;
|
||
color: var(--text-secondary);
|
||
}
|
||
|
||
/* ── Modal overlay ── */
|
||
.modal-overlay {
|
||
display: none;
|
||
position: fixed;
|
||
top: 0;
|
||
left: 0;
|
||
right: 0;
|
||
bottom: 0;
|
||
background: rgba(0, 0, 0, 0.5);
|
||
z-index: 1000;
|
||
padding: 40px 24px;
|
||
overflow-y: auto;
|
||
}
|
||
|
||
.modal-overlay.active {
|
||
display: flex;
|
||
align-items: flex-start;
|
||
justify-content: center;
|
||
}
|
||
|
||
.modal {
|
||
max-width: 560px;
|
||
width: 100%;
|
||
background: var(--surface);
|
||
border-radius: 12px;
|
||
box-shadow: 0 20px 60px rgba(0, 0, 0, 0.3);
|
||
overflow: hidden;
|
||
}
|
||
|
||
.modal-header {
|
||
display: flex;
|
||
justify-content: space-between;
|
||
align-items: center;
|
||
padding: 20px 24px;
|
||
border-bottom: 1px solid var(--border);
|
||
background: var(--background);
|
||
}
|
||
|
||
.modal-header h2 {
|
||
font-size: 18px;
|
||
font-weight: 600;
|
||
}
|
||
|
||
.modal-close {
|
||
width: 32px;
|
||
height: 32px;
|
||
border: none;
|
||
background: none;
|
||
cursor: pointer;
|
||
border-radius: 6px;
|
||
display: flex;
|
||
align-items: center;
|
||
justify-content: center;
|
||
color: var(--text-secondary);
|
||
transition: all 0.15s;
|
||
}
|
||
|
||
.modal-close:hover {
|
||
background: var(--border-light);
|
||
color: var(--text-primary);
|
||
}
|
||
|
||
.modal-body {
|
||
padding: 24px;
|
||
}
|
||
|
||
.modal-footer {
|
||
display: flex;
|
||
justify-content: flex-end;
|
||
gap: 10px;
|
||
padding: 16px 24px;
|
||
border-top: 1px solid var(--border);
|
||
background: var(--background);
|
||
}
|
||
|
||
/* ── Form ── */
|
||
.form-group {
|
||
margin-bottom: 18px;
|
||
}
|
||
|
||
.form-group:last-child {
|
||
margin-bottom: 0;
|
||
}
|
||
|
||
.form-label {
|
||
display: block;
|
||
font-size: 13px;
|
||
font-weight: 500;
|
||
color: var(--text-primary);
|
||
margin-bottom: 6px;
|
||
}
|
||
|
||
.form-label .optional {
|
||
font-weight: 400;
|
||
color: var(--text-secondary);
|
||
font-size: 12px;
|
||
}
|
||
|
||
.form-input,
|
||
.form-select,
|
||
.form-textarea {
|
||
width: 100%;
|
||
padding: 8px 12px;
|
||
border: 1px solid var(--border);
|
||
border-radius: 8px;
|
||
font-family: var(--font-primary);
|
||
font-size: 13px;
|
||
color: var(--text-primary);
|
||
background: var(--surface);
|
||
transition: border-color 0.15s;
|
||
}
|
||
|
||
.form-input:focus,
|
||
.form-select:focus,
|
||
.form-textarea:focus {
|
||
outline: none;
|
||
border-color: var(--primary);
|
||
box-shadow: 0 0 0 3px rgba(0, 115, 209, 0.1);
|
||
}
|
||
|
||
.form-input[readonly] {
|
||
background: var(--border-light);
|
||
color: var(--text-secondary);
|
||
cursor: not-allowed;
|
||
}
|
||
|
||
.form-textarea {
|
||
min-height: 80px;
|
||
resize: vertical;
|
||
}
|
||
|
||
.form-select {
|
||
cursor: pointer;
|
||
appearance: none;
|
||
background-image: url("data:image/svg+xml,%3Csvg width='12' height='12' viewBox='0 0 24 24' fill='none' stroke='%236B7280' stroke-width='2' stroke-linecap='round' stroke-linejoin='round' xmlns='http://www.w3.org/2000/svg'%3E%3Cpolyline points='6 9 12 15 18 9'/%3E%3C/svg%3E");
|
||
background-repeat: no-repeat;
|
||
background-position: right 12px center;
|
||
padding-right: 32px;
|
||
}
|
||
|
||
.form-hint {
|
||
font-size: 12px;
|
||
color: var(--text-secondary);
|
||
margin-top: 4px;
|
||
}
|
||
|
||
/* ── Footer ── */
|
||
.footer {
|
||
text-align: center;
|
||
padding: 24px;
|
||
color: var(--text-secondary);
|
||
font-size: 12px;
|
||
}
|
||
|
||
.footer a {
|
||
color: var(--primary);
|
||
text-decoration: none;
|
||
}
|
||
|
||
.footer a:hover {
|
||
text-decoration: underline;
|
||
}
|
||
|
||
/* ── Responsive ── */
|
||
@media (max-width: 640px) {
|
||
.header {
|
||
padding: 0 16px;
|
||
}
|
||
|
||
.page-title {
|
||
padding: 24px 16px 16px;
|
||
}
|
||
|
||
.content {
|
||
padding: 0 16px 24px;
|
||
}
|
||
|
||
.panel-header {
|
||
flex-direction: column;
|
||
align-items: flex-start;
|
||
gap: 12px;
|
||
}
|
||
|
||
.table-item {
|
||
padding-left: 24px;
|
||
flex-wrap: wrap;
|
||
}
|
||
|
||
.table-item-meta {
|
||
flex-direction: column;
|
||
gap: 4px;
|
||
}
|
||
|
||
.registry-table .col-id {
|
||
max-width: 160px;
|
||
}
|
||
|
||
.modal {
|
||
margin: 16px;
|
||
}
|
||
}
|
||
|
||
/* ── Tab nav (Phase D) ── */
|
||
.tab-nav {
|
||
display: flex;
|
||
gap: 4px;
|
||
border-bottom: 1px solid var(--border);
|
||
margin-bottom: 16px;
|
||
}
|
||
.tab {
|
||
padding: 8px 16px;
|
||
background: transparent;
|
||
border: 0;
|
||
cursor: pointer;
|
||
font-family: inherit;
|
||
font-size: inherit;
|
||
color: var(--text-secondary);
|
||
}
|
||
.tab[aria-selected="true"] {
|
||
border-bottom: 2px solid var(--primary);
|
||
color: var(--text-primary);
|
||
font-weight: 600;
|
||
}
|
||
.tab-content {
|
||
padding: 16px 0;
|
||
}
|
||
.tab-header {
|
||
display: flex;
|
||
justify-content: space-between;
|
||
align-items: center;
|
||
margin-bottom: 16px;
|
||
}
|
||
.tab-actions {
|
||
display: flex;
|
||
justify-content: flex-end;
|
||
margin-bottom: 16px;
|
||
}
|
||
</style>
|
||
{% include '_theme.html' %}
|
||
</head>
|
||
<body data-source-type="{{ data_source_type }}">
|
||
|
||
<!-- ═══════════════ HEADER ═══════════════ -->
|
||
{% include '_app_header.html' %}
|
||
|
||
<!-- ═══════════════ PAGE TITLE ═══════════════ -->
|
||
<div class="page-title">
|
||
<h1>Table Management</h1>
|
||
<p>Discover, register, and manage data tables from your source</p>
|
||
</div>
|
||
|
||
<!-- ═══════════════ CONTENT ═══════════════ -->
|
||
<div class="content">
|
||
|
||
<section id="cacheWarmupCard" class="card" style="margin-bottom: 20px;">
|
||
<header class="card-header" style="display: flex; justify-content: space-between; align-items: center;">
|
||
<h2>Cache freshness</h2>
|
||
<button class="btn btn-secondary" id="cacheWarmupRunBtn" onclick="cacheWarmupRun()">
|
||
Re-warm all
|
||
</button>
|
||
</header>
|
||
<div class="card-body">
|
||
<div id="cacheWarmupProgress" style="margin-bottom: 8px;">
|
||
<span id="cacheWarmupSummary">Loading…</span>
|
||
</div>
|
||
<progress id="cacheWarmupBar" max="100" value="0" style="width: 100%; display: none;"></progress>
|
||
<details id="cacheWarmupDetails" style="margin-top: 8px;">
|
||
<summary style="cursor: pointer; user-select: none;">Show log</summary>
|
||
<p id="cacheWarmupHint" style="font-size: 12px; color: var(--text-secondary, #6b7280); margin: 8px 0 0;">
|
||
Live log from <em>Re-warm all</em>. Each row reports
|
||
<code>[hh:mm:ss] ok/error <table_id> (duration)</code>
|
||
as the BQ metadata refresh streams in via SSE. Only
|
||
<code>query_mode='remote'</code> BigQuery tables are
|
||
warmed (e.g. <code>product_inventory_remote</code>);
|
||
materialized + internal tables don't need it.
|
||
Historical runs aren't persisted; check
|
||
<a href="/admin/activity">/admin/activity</a> for the
|
||
audit trail.
|
||
</p>
|
||
<pre id="cacheWarmupLog" style="background: #0a0a0a; color: #dcdcdc; font-family: ui-monospace, Menlo, monospace; font-size: 12px; padding: 8px; max-height: 240px; overflow-y: auto; margin-top: 8px; border-radius: 4px;"></pre>
|
||
</details>
|
||
</div>
|
||
</section>
|
||
|
||
{# Phase D: tab-split scaffold. Per-connector tabs (BigQuery /
|
||
Keboola / Jira) replace the single mixed form. Each tab has its
|
||
own Register button + listing div + (later) form modals. The
|
||
initial active tab matches data_source.type from instance.yaml;
|
||
the operator can still switch tabs to manage a secondary source.
|
||
|
||
Phase E moves the BQ form into #tab-content-bigquery; Phase F
|
||
builds the Keboola form inside #tab-content-keboola. For now
|
||
the existing Jinja-branched panels below stay in place. #}
|
||
{% set initial_tab = data_source_type if data_source_type in ['bigquery', 'keboola', 'jira'] else 'keboola' %}
|
||
|
||
<nav class="tab-nav" role="tablist">
|
||
<button data-tab="bigquery" aria-selected="{{ 'true' if initial_tab == 'bigquery' else 'false' }}" class="tab" onclick="switchTab('bigquery')">BigQuery</button>
|
||
<button data-tab="keboola" aria-selected="{{ 'true' if initial_tab == 'keboola' else 'false' }}" class="tab" onclick="switchTab('keboola')">Keboola</button>
|
||
<button data-tab="jira" aria-selected="{{ 'true' if initial_tab == 'jira' else 'false' }}" class="tab" onclick="switchTab('jira')">Jira</button>
|
||
<button data-tab="internal" aria-selected="false" class="tab" onclick="switchTab('internal')">Agnes internal</button>
|
||
</nav>
|
||
|
||
<section id="tab-content-bigquery" class="tab-content"
|
||
style="display: {% if initial_tab == 'bigquery' %}block{% else %}none{% endif %};">
|
||
<div class="tab-actions">
|
||
<button id="bqRegisterBtn" class="btn btn-primary"
|
||
onclick="openRegisterModal('bigquery')">Register BigQuery table</button>
|
||
</div>
|
||
<div id="bqTableListing"></div>
|
||
|
||
<!-- ── BigQuery Register Modal (Phase E relocation) ── -->
|
||
<div class="modal-overlay" id="registerBqModal">
|
||
<div class="modal">
|
||
<div class="modal-header">
|
||
<h2>Register BigQuery Table</h2>
|
||
<button class="modal-close" onclick="closeRegisterBqModal()">
|
||
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
|
||
<line x1="18" y1="6" x2="6" y2="18"/>
|
||
<line x1="6" y1="6" x2="18" y2="18"/>
|
||
</svg>
|
||
</button>
|
||
</div>
|
||
<div class="modal-body">
|
||
{# Two orthogonal questions: (1) live vs synced, (2) when synced,
|
||
whole table vs custom SQL. Visibility classes:
|
||
bq-access-live — only when accessMode='live'
|
||
bq-access-synced — only when accessMode='synced'
|
||
bq-source-table — only when accessMode='live' OR
|
||
(accessMode='synced' AND syncMode='whole')
|
||
bq-source-custom — only when accessMode='synced' AND syncMode='custom'
|
||
Backend payload: live → query_mode='remote'; synced/whole →
|
||
query_mode='materialized' with auto-built SELECT *; synced/custom
|
||
→ query_mode='materialized' with admin SQL. Server auto-detects
|
||
BASE TABLE vs VIEW at register time, so the UI doesn't ask. #}
|
||
<div class="form-group">
|
||
<label class="form-label">How should analysts access this data?</label>
|
||
<div class="bq-access-radio-group" style="display:flex; gap:12px; margin-top:6px;">
|
||
<label style="flex:1; padding:12px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="bqAccessMode" value="live" checked onchange="onBqAccessModeChange()">
|
||
<strong>Live from BigQuery</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Each analyst query goes straight to BQ. Always current.
|
||
Latency ≈ seconds; 0 disk on the analyst machine; cost =
|
||
bytes scanned per query. Best for huge tables or when
|
||
freshness matters.
|
||
</div>
|
||
</label>
|
||
<label style="flex:1; padding:12px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="bqAccessMode" value="synced" onchange="onBqAccessModeChange()">
|
||
<strong>Synced locally</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Agnes runs a SELECT on a schedule and ships a parquet
|
||
to analysts. Analyst-side latency <100 ms; disk =
|
||
snapshot size. Best when analysts hit the same data
|
||
often and speed beats freshness.
|
||
</div>
|
||
</label>
|
||
</div>
|
||
</div>
|
||
<div class="form-group bq-access-synced" style="display:none;">
|
||
<label class="form-label">What to sync?</label>
|
||
<div class="bq-sync-radio-group" style="display:flex; gap:12px; margin-top:6px;">
|
||
<label style="flex:1; padding:12px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="bqSyncMode" value="whole" checked onchange="onBqSyncModeChange()">
|
||
<strong>Whole table</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Agnes runs <code>SELECT *</code> automatically. No SQL
|
||
required. Disk + sync cost = full table size.
|
||
</div>
|
||
</label>
|
||
<label style="flex:1; padding:12px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="bqSyncMode" value="custom" onchange="onBqSyncModeChange()">
|
||
<strong>Custom query</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
You write the SELECT — filter, project, or aggregate
|
||
before the sync. Cuts disk + cost; cap via
|
||
<code>max_bytes_per_materialize</code> guardrail.
|
||
</div>
|
||
</label>
|
||
</div>
|
||
</div>
|
||
<div class="form-group bq-source-table">
|
||
<label class="form-label" for="bqDataset">
|
||
Dataset
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="discoverBqDatasets()" style="float:right; margin-top:-3px;">
|
||
Discover
|
||
</button>
|
||
</label>
|
||
<input type="text" class="form-input" id="bqDataset" list="bqDatasetList" placeholder="e.g. analytics">
|
||
<datalist id="bqDatasetList"></datalist>
|
||
<div class="form-hint">BigQuery dataset name (no project prefix — read from instance.yaml).
|
||
Click <strong>Discover</strong> to populate the autocomplete from the BQ project's dataset list.</div>
|
||
</div>
|
||
<div class="form-group bq-source-table">
|
||
<label class="form-label" for="bqSourceTable">
|
||
Source Table / View
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="discoverBqTables()" style="float:right; margin-top:-3px;">
|
||
List tables
|
||
</button>
|
||
</label>
|
||
<input type="text" class="form-input" id="bqSourceTable" list="bqTableList" placeholder="e.g. orders">
|
||
<datalist id="bqTableList"></datalist>
|
||
<div class="form-hint">Table or view name within the dataset. Click
|
||
<strong>List tables</strong> after filling Dataset to populate autocomplete.
|
||
<br><strong>Live access:</strong> BASE TABLEs query via
|
||
<code>bq."dataset"."table"</code> (Storage Read API; predicate pushdown).
|
||
VIEWs and MATERIALIZED_VIEWs query via the BQ jobs API (full-scan estimate;
|
||
cost-guarded by <code>bq_max_scan_bytes</code>).
|
||
<code>agnes query --remote</code> works for both.
|
||
<br><strong>Synced access:</strong> handles both table and view transparently
|
||
— the scheduler runs <code>SELECT *</code> through the jobs API and writes a
|
||
parquet.</div>
|
||
</div>
|
||
<div class="form-group bq-source-custom" style="display:none;">
|
||
<label class="form-label" for="bqSourceQuery">
|
||
SQL
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="prefillFromTable()" style="float:right; margin-top:-3px;"
|
||
title="Prefill SELECT * FROM `project.dataset.table` so you only edit the WHERE / projection">
|
||
Use table as base
|
||
</button>
|
||
</label>
|
||
<textarea class="form-textarea" id="bqSourceQuery" rows="8"
|
||
placeholder="SELECT date, SUM(revenue) AS revenue FROM `project.dataset.orders` WHERE date >= DATE_SUB(CURRENT_DATE(), INTERVAL 90 DAY) GROUP BY 1"></textarea>
|
||
<div class="form-hint">
|
||
SELECT statement, no trailing semicolon. Native BQ identifiers
|
||
(<code>`project.dataset.table`</code>) recommended — DuckDB three-part
|
||
names like <code>bq."ds"."t"</code> work for the COPY but disable the
|
||
cost guardrail's BQ dry-run.
|
||
</div>
|
||
</div>
|
||
<div class="form-group">
|
||
<label class="form-label" for="bqViewName">View Name</label>
|
||
<input type="text" class="form-input" id="bqViewName" placeholder="orders_90d">
|
||
<div class="form-hint">Name analysts use to query the data (e.g.
|
||
<code>SELECT * FROM orders_90d</code>). Required for Custom query; defaults
|
||
to the source table for the other modes.</div>
|
||
</div>
|
||
<div class="form-group">
|
||
<label class="form-label" for="bqDescription">Description <span class="optional">(optional)</span></label>
|
||
<textarea class="form-textarea" id="bqDescription" placeholder="Brief description of the table contents..."></textarea>
|
||
</div>
|
||
<div class="form-group">
|
||
<label class="form-label" for="bqFolder">Folder <span class="optional">(optional)</span></label>
|
||
<input type="text" class="form-input" id="bqFolder" placeholder="e.g. crm, finance, marketing">
|
||
<div class="form-hint">Logical grouping for catalog organization</div>
|
||
</div>
|
||
<div class="form-group bq-access-synced" style="display:none;">
|
||
<label class="form-label" for="bqSyncSchedule">Sync Schedule <span class="optional">(optional, default <code>every 1h</code>)</span></label>
|
||
<input type="text" class="form-input" id="bqSyncSchedule" placeholder="every 6h">
|
||
<div class="form-hint">
|
||
How often Agnes refreshes the local copy. Examples:
|
||
<code>every 15m</code>, <code>every 6h</code>,
|
||
<code>daily 03:00</code>, <code>daily 07:00,13:00,18:00</code> (UTC).
|
||
</div>
|
||
</div>
|
||
<div class="form-group" id="bqPrecheckSummary" style="display:none;">
|
||
<div class="form-label">Source check</div>
|
||
<div class="form-hint" id="bqPrecheckSummaryText"></div>
|
||
</div>
|
||
</div>
|
||
<div class="modal-footer">
|
||
<button class="btn btn-secondary" onclick="closeRegisterBqModal()">Cancel</button>
|
||
<button class="btn btn-primary" id="registerBqSubmitBtn" onclick="registerBqTable()">
|
||
Register Table
|
||
</button>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<!-- ── BigQuery Edit Modal (C2 — physically inside the BQ tab,
|
||
mirror of #registerBqModal placement) ── -->
|
||
<div class="modal-overlay" id="editBqModal">
|
||
<div class="modal">
|
||
<div class="modal-header">
|
||
<h2>Edit BigQuery Table</h2>
|
||
<button class="modal-close" onclick="closeEditBqModal()">
|
||
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
|
||
<line x1="18" y1="6" x2="6" y2="18"/>
|
||
<line x1="6" y1="6" x2="18" y2="18"/>
|
||
</svg>
|
||
</button>
|
||
</div>
|
||
<div class="modal-body">
|
||
<div class="form-group">
|
||
<label class="form-label" for="editBqTableId">Table ID</label>
|
||
<input type="text" class="form-input" id="editBqTableId" readonly>
|
||
<div class="form-hint">Slugified id, immutable. Source type:
|
||
<strong id="editBqSourceTypeBadge">bigquery</strong></div>
|
||
</div>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label">How should analysts access this data?
|
||
<a href="docs/admin/query-modes.md" target="_blank" title="When to use which mode" style="margin-left: 6px; text-decoration: none; cursor: help;">?</a>
|
||
</label>
|
||
<div style="display:flex; gap:12px; margin-top:6px;">
|
||
<label style="flex:1; padding:10px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="editBqAccessMode" value="live" onchange="onEditBqAccessModeChange()">
|
||
<strong>Live from BigQuery</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Each query goes to BQ. No local copy.
|
||
</div>
|
||
</label>
|
||
<label style="flex:1; padding:10px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="editBqAccessMode" value="synced" onchange="onEditBqAccessModeChange()">
|
||
<strong>Synced locally</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Scheduled SELECT → parquet, queried locally.
|
||
</div>
|
||
</label>
|
||
</div>
|
||
<div class="form-hint" id="editBqModeWarning" style="display:none;
|
||
color:#EA580C;background:rgba(234,88,12,.08);padding:8px;border-radius:6px;margin-top:8px;">
|
||
<!-- Filled by onEditBqAccessModeChange() when switching
|
||
modes on an existing row — warns about parquet
|
||
drop / scheduling impact. -->
|
||
</div>
|
||
</div>
|
||
<div class="form-group bq-edit-access-synced" style="display:none;">
|
||
<label class="form-label">What to sync?</label>
|
||
<div style="display:flex; gap:12px; margin-top:6px;">
|
||
<label style="flex:1; padding:10px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="editBqSyncMode" value="whole" onchange="onEditBqSyncModeChange()">
|
||
<strong>Whole table</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
<code>SELECT *</code> on a schedule.
|
||
</div>
|
||
</label>
|
||
<label style="flex:1; padding:10px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="editBqSyncMode" value="custom" onchange="onEditBqSyncModeChange()">
|
||
<strong>Custom query</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Filter / aggregate before sync.
|
||
</div>
|
||
</label>
|
||
</div>
|
||
</div>
|
||
<div class="form-group bq-edit-source-table" style="display:none;">
|
||
<label class="form-label" for="editBqDataset">
|
||
Dataset
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="discoverBqDatasets('editBqDatasetList')"
|
||
style="float:right; margin-top:-3px;">
|
||
Discover
|
||
</button>
|
||
</label>
|
||
<input type="text" class="form-input" id="editBqDataset"
|
||
list="editBqDatasetList" placeholder="e.g. analytics">
|
||
<datalist id="editBqDatasetList"></datalist>
|
||
</div>
|
||
<div class="form-group bq-edit-source-table" style="display:none;">
|
||
<label class="form-label" for="editBqSourceTable">
|
||
Source Table / View
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="discoverBqTables('editBqDataset', 'editBqTableList')"
|
||
style="float:right; margin-top:-3px;">
|
||
List tables
|
||
</button>
|
||
</label>
|
||
<input type="text" class="form-input" id="editBqSourceTable"
|
||
list="editBqTableList" placeholder="e.g. orders">
|
||
<datalist id="editBqTableList"></datalist>
|
||
<div class="form-hint">Table or view name within the dataset.
|
||
<br><strong>Live access:</strong> BASE TABLEs query via
|
||
<code>bq."dataset"."table"</code> (Storage Read API; predicate pushdown).
|
||
VIEWs and MATERIALIZED_VIEWs query via the BQ jobs API (full-scan estimate;
|
||
cost-guarded by <code>bq_max_scan_bytes</code>).
|
||
<code>agnes query --remote</code> works for both.
|
||
<br><strong>Synced access:</strong> handles both transparently — the
|
||
scheduler runs <code>SELECT *</code> through the jobs API and writes a
|
||
parquet.</div>
|
||
</div>
|
||
<div class="form-group bq-edit-source-custom" style="display:none;">
|
||
<label class="form-label" for="editBqSourceQuery">
|
||
SQL
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="prefillFromTable('editBqSourceQuery')"
|
||
style="float:right; margin-top:-3px;"
|
||
title="Prefill SELECT * FROM `project.dataset.table` so you only edit the WHERE / projection">
|
||
Use table as base
|
||
</button>
|
||
</label>
|
||
<textarea class="form-textarea" id="editBqSourceQuery" rows="8"></textarea>
|
||
<div class="form-hint">SELECT statement, no trailing semicolon. Native BQ
|
||
identifiers recommended for the cost guardrail to engage.</div>
|
||
</div>
|
||
<div class="form-group bq-edit-access-synced" style="display:none;">
|
||
<label class="form-label" for="editBqSyncSchedule">Sync Schedule
|
||
<span class="optional">(optional)</span></label>
|
||
<input type="text" class="form-input" id="editBqSyncSchedule" placeholder="every 6h">
|
||
<div class="form-hint">How often Agnes refreshes the local copy.
|
||
<code>every 15m</code>, <code>every 6h</code>,
|
||
<code>daily 03:00</code> (UTC).</div>
|
||
</div>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label" for="editBqDescription">Description <span class="optional">(optional)</span></label>
|
||
<textarea class="form-textarea" id="editBqDescription" placeholder="Brief description of the table contents..."></textarea>
|
||
</div>
|
||
<div class="form-group">
|
||
<label class="form-label" for="editBqFolder">Folder <span class="optional">(optional)</span></label>
|
||
<input type="text" class="form-input" id="editBqFolder" placeholder="e.g. crm, finance, marketing">
|
||
<div class="form-hint">Logical grouping for catalog organization (does not affect storage).</div>
|
||
</div>
|
||
</div>
|
||
<div class="modal-footer">
|
||
<button class="btn btn-secondary" onclick="closeEditBqModal()">Cancel</button>
|
||
<button class="btn btn-primary" id="editBqSubmitBtn" onclick="saveBqTabEdit()">
|
||
Save Changes
|
||
</button>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</section>
|
||
|
||
<section id="tab-content-keboola" class="tab-content"
|
||
style="display: {% if initial_tab == 'keboola' %}block{% else %}none{% endif %};">
|
||
<div class="tab-actions">
|
||
<button id="kbRegisterBtn" class="btn btn-primary"
|
||
onclick="openRegisterModal('keboola')">Register Keboola table</button>
|
||
</div>
|
||
<div id="kbTableListing"></div>
|
||
|
||
<!-- ── Keboola Register Modal (Phase F1) ── -->
|
||
<div class="modal-overlay" id="registerKeboolaModal">
|
||
<div class="modal">
|
||
<div class="modal-header">
|
||
<h2>Register Keboola Table</h2>
|
||
<button class="modal-close" onclick="closeRegisterKeboolaModal()">
|
||
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
|
||
<line x1="18" y1="6" x2="6" y2="18"/>
|
||
<line x1="6" y1="6" x2="18" y2="18"/>
|
||
</svg>
|
||
</button>
|
||
</div>
|
||
<div class="modal-body">
|
||
{# Three sync-mode radios:
|
||
- whole / custom → query_mode='materialized' (DuckDB Keboola
|
||
extension; whole synthesizes SELECT *, custom uses admin SQL)
|
||
- direct → query_mode='local' (Storage API SDK, supports
|
||
v26 sync strategies: incremental/partitioned + where_filters) #}
|
||
<div class="form-group">
|
||
<label class="form-label">What to sync?</label>
|
||
<div class="bq-sync-radio-group" style="display:flex; gap:12px; margin-top:6px; flex-wrap:wrap;">
|
||
<label style="flex:1; min-width:200px; padding:12px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="kbSyncMode" value="whole" checked onchange="onKbSyncModeChange()">
|
||
<strong>Whole table (extension)</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
DuckDB Keboola extension pulls the full table on
|
||
each tick. Fastest path; full overwrite each run.
|
||
</div>
|
||
</label>
|
||
<label style="flex:1; min-width:200px; padding:12px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="kbSyncMode" value="direct" onchange="onKbSyncModeChange()">
|
||
<strong>Direct extract (Storage API)</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Storage API SDK. Supports incremental sync
|
||
(changedSince + PK merge), partitioned files,
|
||
and server-side <code>where_filters</code>.
|
||
</div>
|
||
</label>
|
||
<label style="flex:1; min-width:200px; padding:12px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="kbSyncMode" value="custom" onchange="onKbSyncModeChange()">
|
||
<strong>Custom SQL</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Pre-aggregate or filter with your own SELECT
|
||
(e.g. last 30 days only, per-day rollup).
|
||
</div>
|
||
</label>
|
||
</div>
|
||
</div>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label" for="kbViewName">View name (analyst-visible)</label>
|
||
<input type="text" class="form-input" id="kbViewName"
|
||
placeholder="e.g. orders_recent">
|
||
</div>
|
||
|
||
{# Discover/List tables backend currently routes by instance's data_source.type
|
||
ignoring the `source` query param. Hiding the buttons on non-Keboola instances
|
||
prevents wrong-shape responses; inputs stay for manual entry. Future fix: make
|
||
/api/admin/discover-tables accept ?source=keboola and remove this guard. #}
|
||
<div class="form-group kb-source-table">
|
||
<label class="form-label" for="kbBucket">
|
||
Bucket
|
||
{% if data_source_type == 'keboola' %}
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="discoverKeboolaBuckets('kbBucketList')"
|
||
style="float:right; margin-top:-3px;">Discover</button>
|
||
{% endif %}
|
||
</label>
|
||
<input type="text" class="form-input" id="kbBucket"
|
||
list="kbBucketList" placeholder="e.g. in.c-sales">
|
||
<datalist id="kbBucketList"></datalist>
|
||
</div>
|
||
<div class="form-group kb-source-table">
|
||
<label class="form-label" for="kbSourceTable">
|
||
Source Table
|
||
{% if data_source_type == 'keboola' %}
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="discoverKeboolaTables('kbBucket', 'kbTableList')"
|
||
style="float:right; margin-top:-3px;">List tables</button>
|
||
{% endif %}
|
||
</label>
|
||
<input type="text" class="form-input" id="kbSourceTable"
|
||
list="kbTableList" placeholder="e.g. orders">
|
||
<datalist id="kbTableList"></datalist>
|
||
</div>
|
||
<div class="form-group kb-source-custom" style="display:none;">
|
||
<label class="form-label" for="kbSourceQuery">
|
||
SQL
|
||
{% if data_source_type == 'keboola' %}
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="prefillFromKeboolaTable('kbSourceQuery')"
|
||
style="float:right; margin-top:-3px;"
|
||
title="Prefill SELECT * FROM kbc.bucket.table so you only edit the WHERE / projection">
|
||
Use table as base
|
||
</button>
|
||
{% endif %}
|
||
</label>
|
||
<textarea class="form-textarea" id="kbSourceQuery" rows="8"></textarea>
|
||
<div class="form-hint">SELECT against <code>kbc."bucket"."table"</code>.
|
||
Result is materialized to parquet and distributed via
|
||
<code>agnes pull</code>.</div>
|
||
</div>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label" for="kbSyncSchedule">Sync Schedule
|
||
<span class="optional">(optional, default <code>every 1h</code>)</span></label>
|
||
<input type="text" class="form-input" id="kbSyncSchedule" placeholder="every 6h">
|
||
<div class="form-hint">
|
||
How often Agnes refreshes the local copy. Examples:
|
||
<code>every 15m</code>, <code>every 6h</code>,
|
||
<code>daily 03:00</code>, <code>daily 07:00,13:00,18:00</code> (UTC).
|
||
</div>
|
||
</div>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label" for="kbDescription">Description
|
||
<span class="optional">(optional)</span></label>
|
||
<textarea class="form-textarea" id="kbDescription"
|
||
placeholder="Brief description of the table contents..."></textarea>
|
||
</div>
|
||
<div class="form-group">
|
||
<label class="form-label" for="kbFolder">Folder
|
||
<span class="optional">(optional)</span></label>
|
||
<input type="text" class="form-input" id="kbFolder"
|
||
placeholder="e.g. crm, finance, marketing">
|
||
</div>
|
||
|
||
<details class="form-group">
|
||
<summary>Advanced (optional)</summary>
|
||
<div class="form-group" style="margin-top:8px;">
|
||
<label class="form-label" for="kbPrimaryKey">Primary Key</label>
|
||
<input type="text" class="form-input" id="kbPrimaryKey"
|
||
placeholder="e.g. id">
|
||
<div class="form-hint">Comma-separated list. Required for
|
||
Direct extract → Incremental (used as the dedup key on
|
||
delta merge). Auto-filled from the Keboola source when
|
||
available.</div>
|
||
</div>
|
||
</details>
|
||
|
||
<!-- v26 Direct-extract sync-strategy panel — visible only when
|
||
"Direct extract (Storage API)" is selected. Field visibility
|
||
within the panel further branches on the sync_strategy
|
||
dropdown (incremental / partitioned). -->
|
||
<div class="form-group kb-direct-only" style="display:none; padding:12px; border:1px solid var(--border); border-radius:8px; background:var(--background);">
|
||
<h3 style="margin:0 0 12px 0; font-size:14px;">Direct extract — sync strategy</h3>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label" for="kbStrategy">Sync strategy</label>
|
||
<select class="form-select" id="kbStrategy" onchange="onKbStrategyChange()">
|
||
<option value="full_refresh">Full refresh — pull entire table each tick</option>
|
||
<option value="incremental">Incremental — pull rows changed since last sync</option>
|
||
<option value="partitioned">Partitioned — per-partition files, per-month/day/year</option>
|
||
</select>
|
||
</div>
|
||
|
||
<div class="form-group kb-strategy-incremental kb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="kbIncrementalWindowDays">Incremental window (days)
|
||
<span class="optional">(default 7)</span></label>
|
||
<input type="number" class="form-input" id="kbIncrementalWindowDays" min="0" placeholder="7">
|
||
<div class="form-hint">Backtrack window applied to last_sync timestamp on each tick.
|
||
Higher = more reliable on late-arriving rows; lower = less data per tick.</div>
|
||
</div>
|
||
<div class="form-group kb-strategy-incremental kb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="kbMaxHistoryDays">Max history (days)
|
||
<span class="optional">(first sync only, default unbounded)</span></label>
|
||
<input type="number" class="form-input" id="kbMaxHistoryDays" min="1" placeholder="365">
|
||
<div class="form-hint">Cap on how far back the first-ever sync goes. Multi-year tables
|
||
without this can OOM at write — set 90/180/365 for safety.</div>
|
||
</div>
|
||
|
||
<div class="form-group kb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="kbPartitionBy">Partition by column <strong>(required)</strong></label>
|
||
<input type="text" class="form-input" id="kbPartitionBy" placeholder="e.g. event_date">
|
||
<div class="form-hint">Date / timestamp column whose value drives the partition key.
|
||
Rows with NULL or unparseable values are dropped (logged warning).</div>
|
||
</div>
|
||
<div class="form-group kb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="kbPartitionGranularity">Granularity</label>
|
||
<select class="form-select" id="kbPartitionGranularity">
|
||
<option value="month">Month — YYYY_MM.parquet (default)</option>
|
||
<option value="day">Day — YYYY_MM_DD.parquet</option>
|
||
<option value="year">Year — YYYY.parquet</option>
|
||
</select>
|
||
</div>
|
||
<div class="form-group kb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="kbInitialLoadChunkDays">Initial-load chunk size (days)
|
||
<span class="optional">(default 30)</span></label>
|
||
<input type="number" class="form-input" id="kbInitialLoadChunkDays" min="1" placeholder="30">
|
||
<div class="form-hint">First-sync chunked load step. Smaller = more API calls, less
|
||
memory per chunk. Larger = fewer calls, more memory.</div>
|
||
</div>
|
||
|
||
<div class="form-group kb-strategy-not-incremental" style="display:none;">
|
||
<label class="form-label" for="kbWhereFilters">Where filters
|
||
<span class="optional">(JSON array, optional)</span></label>
|
||
<textarea class="form-textarea" id="kbWhereFilters" rows="6"
|
||
placeholder='[{"column": "date", "operator": "ge", "values": ["{{last_3_months}}"]}]'></textarea>
|
||
<div class="form-hint">
|
||
Server-side row filter. Operators: <code>eq, ne, gt, ge, lt, le</code>.
|
||
Date placeholders resolved at sync time:
|
||
<code>{{ '{{today}}' }}</code>,
|
||
<code>{{ '{{last_week}}' }}</code>,
|
||
<code>{{ '{{last_month}}' }}</code>,
|
||
<code>{{ '{{last_2_months}}' }}</code>,
|
||
<code>{{ '{{last_3_months}}' }}</code>,
|
||
<code>{{ '{{last_6_months}}' }}</code>,
|
||
<code>{{ '{{last_year}}' }}</code>,
|
||
<code>{{ '{{last_2_years}}' }}</code>,
|
||
<code>{{ '{{start_of_3_months_ago}}' }}</code>.
|
||
Not compatible with Incremental strategy.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="modal-footer">
|
||
<button class="btn btn-secondary" onclick="closeRegisterKeboolaModal()">Cancel</button>
|
||
<button class="btn btn-primary" id="registerKeboolaSubmitBtn"
|
||
onclick="registerKeboolaTable()">Register</button>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<!-- ── Keboola Edit Modal (Phase F2) ── -->
|
||
<div class="modal-overlay" id="editKeboolaModal">
|
||
<div class="modal">
|
||
<div class="modal-header">
|
||
<h2>Edit Keboola Table</h2>
|
||
<button class="modal-close" onclick="closeEditKeboolaModal()">
|
||
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
|
||
<line x1="18" y1="6" x2="6" y2="18"/>
|
||
<line x1="6" y1="6" x2="18" y2="18"/>
|
||
</svg>
|
||
</button>
|
||
</div>
|
||
<div class="modal-body">
|
||
<div class="form-group">
|
||
<label class="form-label" for="editKbTableId">Table ID</label>
|
||
<input type="text" class="form-input" id="editKbTableId" readonly>
|
||
<div class="form-hint">Slugified id, immutable.</div>
|
||
</div>
|
||
|
||
{# Three sync-mode radios (mirror of Register). #}
|
||
<div class="form-group">
|
||
<label class="form-label">What to sync?</label>
|
||
<div class="bq-sync-radio-group" style="display:flex; gap:12px; margin-top:6px; flex-wrap:wrap;">
|
||
<label style="flex:1; min-width:200px; padding:10px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="editKbSyncMode" value="whole"
|
||
onchange="onEditKbSyncModeChange()">
|
||
<strong>Whole table (extension)</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
DuckDB Keboola extension; full overwrite each tick.
|
||
</div>
|
||
</label>
|
||
<label style="flex:1; min-width:200px; padding:10px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="editKbSyncMode" value="direct"
|
||
onchange="onEditKbSyncModeChange()">
|
||
<strong>Direct extract (Storage API)</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Storage API SDK. Supports incremental, partitioned,
|
||
<code>where_filters</code>.
|
||
</div>
|
||
</label>
|
||
<label style="flex:1; min-width:200px; padding:10px; border:1px solid var(--border); border-radius:8px; cursor:pointer;">
|
||
<input type="radio" name="editKbSyncMode" value="custom"
|
||
onchange="onEditKbSyncModeChange()">
|
||
<strong>Custom SQL</strong>
|
||
<div style="font-size:12px; color:var(--text-secondary); margin-top:4px;">
|
||
Pre-aggregate or filter with your own SELECT.
|
||
</div>
|
||
</label>
|
||
</div>
|
||
</div>
|
||
|
||
{# Discover/List tables backend currently routes by instance's data_source.type
|
||
ignoring the `source` query param. Hiding the buttons on non-Keboola instances
|
||
prevents wrong-shape responses; inputs stay for manual entry. Future fix: make
|
||
/api/admin/discover-tables accept ?source=keboola and remove this guard. #}
|
||
<div class="form-group editkb-source-table">
|
||
<label class="form-label" for="editKbBucket">
|
||
Bucket
|
||
{% if data_source_type == 'keboola' %}
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="discoverKeboolaBuckets('editKbBucketList')"
|
||
style="float:right; margin-top:-3px;">Discover</button>
|
||
{% endif %}
|
||
</label>
|
||
<input type="text" class="form-input" id="editKbBucket"
|
||
list="editKbBucketList" placeholder="e.g. in.c-sales">
|
||
<datalist id="editKbBucketList"></datalist>
|
||
</div>
|
||
<div class="form-group editkb-source-table">
|
||
<label class="form-label" for="editKbSourceTable">
|
||
Source Table
|
||
{% if data_source_type == 'keboola' %}
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="discoverKeboolaTables('editKbBucket', 'editKbTableList')"
|
||
style="float:right; margin-top:-3px;">List tables</button>
|
||
{% endif %}
|
||
</label>
|
||
<input type="text" class="form-input" id="editKbSourceTable"
|
||
list="editKbTableList" placeholder="e.g. orders">
|
||
<datalist id="editKbTableList"></datalist>
|
||
</div>
|
||
<div class="form-group editkb-source-custom" style="display:none;">
|
||
<label class="form-label" for="editKbSourceQuery">
|
||
SQL
|
||
{% if data_source_type == 'keboola' %}
|
||
<button type="button" class="btn btn-secondary btn-sm"
|
||
onclick="prefillFromKeboolaTable('editKbSourceQuery')"
|
||
style="float:right; margin-top:-3px;"
|
||
title="Prefill SELECT * FROM kbc.bucket.table so you only edit the WHERE / projection">
|
||
Use table as base
|
||
</button>
|
||
{% endif %}
|
||
</label>
|
||
<textarea class="form-textarea" id="editKbSourceQuery" rows="8"></textarea>
|
||
</div>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label" for="editKbSyncSchedule">Sync Schedule
|
||
<span class="optional">(optional)</span></label>
|
||
<input type="text" class="form-input" id="editKbSyncSchedule" placeholder="every 6h">
|
||
</div>
|
||
<div class="form-group">
|
||
<label class="form-label" for="editKbDescription">Description
|
||
<span class="optional">(optional)</span></label>
|
||
<textarea class="form-textarea" id="editKbDescription"></textarea>
|
||
</div>
|
||
<div class="form-group">
|
||
<label class="form-label" for="editKbFolder">Folder
|
||
<span class="optional">(optional)</span></label>
|
||
<input type="text" class="form-input" id="editKbFolder">
|
||
</div>
|
||
|
||
<details class="form-group">
|
||
<summary>Advanced (optional)</summary>
|
||
<div class="form-group" style="margin-top:8px;">
|
||
<label class="form-label" for="editKbPrimaryKey">Primary Key</label>
|
||
<input type="text" class="form-input" id="editKbPrimaryKey"
|
||
placeholder="e.g. id">
|
||
<div class="form-hint">Comma-separated list. Required for
|
||
Direct extract → Incremental.</div>
|
||
</div>
|
||
</details>
|
||
|
||
<!-- v26 Direct-extract sync-strategy panel — mirror of Register. -->
|
||
<div class="form-group editkb-direct-only" style="display:none; padding:12px; border:1px solid var(--border); border-radius:8px; background:var(--background);">
|
||
<h3 style="margin:0 0 12px 0; font-size:14px;">Direct extract — sync strategy</h3>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label" for="editKbStrategy">Sync strategy</label>
|
||
<select class="form-select" id="editKbStrategy" onchange="onEditKbStrategyChange()">
|
||
<option value="full_refresh">Full refresh — pull entire table each tick</option>
|
||
<option value="incremental">Incremental — pull rows changed since last sync</option>
|
||
<option value="partitioned">Partitioned — per-partition files, per-month/day/year</option>
|
||
</select>
|
||
</div>
|
||
|
||
<div class="form-group editkb-strategy-incremental editkb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="editKbIncrementalWindowDays">Incremental window (days)
|
||
<span class="optional">(default 7)</span></label>
|
||
<input type="number" class="form-input" id="editKbIncrementalWindowDays" min="0" placeholder="7">
|
||
</div>
|
||
<div class="form-group editkb-strategy-incremental editkb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="editKbMaxHistoryDays">Max history (days)
|
||
<span class="optional">(first sync only)</span></label>
|
||
<input type="number" class="form-input" id="editKbMaxHistoryDays" min="1" placeholder="365">
|
||
</div>
|
||
|
||
<div class="form-group editkb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="editKbPartitionBy">Partition by column <strong>(required)</strong></label>
|
||
<input type="text" class="form-input" id="editKbPartitionBy" placeholder="e.g. event_date">
|
||
</div>
|
||
<div class="form-group editkb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="editKbPartitionGranularity">Granularity</label>
|
||
<select class="form-select" id="editKbPartitionGranularity">
|
||
<option value="month">Month (default)</option>
|
||
<option value="day">Day</option>
|
||
<option value="year">Year</option>
|
||
</select>
|
||
</div>
|
||
<div class="form-group editkb-strategy-partitioned" style="display:none;">
|
||
<label class="form-label" for="editKbInitialLoadChunkDays">Initial-load chunk size (days)
|
||
<span class="optional">(default 30)</span></label>
|
||
<input type="number" class="form-input" id="editKbInitialLoadChunkDays" min="1" placeholder="30">
|
||
</div>
|
||
|
||
<div class="form-group editkb-strategy-not-incremental" style="display:none;">
|
||
<label class="form-label" for="editKbWhereFilters">Where filters
|
||
<span class="optional">(JSON array, optional)</span></label>
|
||
<textarea class="form-textarea" id="editKbWhereFilters" rows="6"
|
||
placeholder='[{"column": "date", "operator": "ge", "values": ["{{last_3_months}}"]}]'></textarea>
|
||
<div class="form-hint">
|
||
Operators: <code>eq, ne, gt, ge, lt, le</code>. Date placeholders
|
||
(<code>{{ '{{today}}' }}</code>, <code>{{ '{{last_3_months}}' }}</code>, etc.) resolved at
|
||
sync time. Not compatible with Incremental.
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
<div class="modal-footer">
|
||
<button class="btn btn-secondary" onclick="closeEditKeboolaModal()">Cancel</button>
|
||
<button class="btn btn-primary" id="editKeboolaSubmitBtn"
|
||
onclick="saveKeboolaTabEdit()">Save Changes</button>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
</section>
|
||
|
||
<section id="tab-content-jira" class="tab-content"
|
||
style="display: {% if initial_tab == 'jira' %}block{% else %}none{% endif %};">
|
||
<p class="hint" style="margin-bottom: 16px;">Jira tables are populated by webhooks.
|
||
To register a new Jira webhook integration, see
|
||
<code>docs/connectors/jira.md</code>.</p>
|
||
<div id="jiraTableListing"></div>
|
||
</section>
|
||
|
||
{# Fourth tab: Agnes internal data source. Auto-seeded read-only
|
||
rows backed by system.duckdb (agnes_sessions / agnes_telemetry /
|
||
agnes_audit). No Register button — these are seeded on every
|
||
app boot from connectors/internal/registry.py. The
|
||
tab-content visibility matches the BQ/Keboola/Jira pattern
|
||
above so switchTab() handles it without code changes. #}
|
||
<section id="tab-content-internal" class="tab-content" style="display: none;">
|
||
<p class="hint" style="margin-bottom: 16px;">
|
||
System tables — sessions, telemetry, audit. Also available
|
||
locally for analysis. Read-only here; Edit / Delete are
|
||
hidden because the rows are seeded on every app boot.
|
||
</p>
|
||
<div id="internalTableListing"></div>
|
||
</section>
|
||
|
||
{# Legacy out-of-tab panels (BQ Register card, Keboola Discovery card,
|
||
shared Registered Tables wrapper) removed — each tab now owns its
|
||
own header (with Register button) and listing div. The Refresh
|
||
action is implicit: registration / edit / delete flows already
|
||
call loadRegistry(), which re-renders all three per-tab listings. #}
|
||
|
||
</div>
|
||
|
||
{# C3: legacy #registerModal removed. The Phase E #registerBqModal
|
||
(inside #tab-content-bigquery) and Phase F #registerKeboolaModal
|
||
(inside #tab-content-keboola) own the Register flows now. The
|
||
data-source-type marker moved to <body> so DATA_SOURCE_TYPE still
|
||
has somewhere to read from. #}
|
||
|
||
<!-- ═══════════════ EDIT MODAL (legacy fallback — Keboola-only fields
|
||
remaining; the BQ Edit modal moved into #tab-content-bigquery as
|
||
#editBqModal in C2; the Keboola Edit modal is #editKeboolaModal
|
||
in #tab-content-keboola) ═══════════════ -->
|
||
<div class="modal-overlay" id="editModal">
|
||
<div class="modal">
|
||
<div class="modal-header">
|
||
<h2>Edit Table</h2>
|
||
<button class="modal-close" onclick="closeEditModal()">
|
||
<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
|
||
<line x1="18" y1="6" x2="6" y2="18"/>
|
||
<line x1="6" y1="6" x2="18" y2="18"/>
|
||
</svg>
|
||
</button>
|
||
</div>
|
||
<div class="modal-body">
|
||
<div class="form-group">
|
||
<label class="form-label" for="editTableId">Table ID</label>
|
||
<input type="text" class="form-input" id="editTableId" readonly>
|
||
<div class="form-hint">Slugified id, immutable. Source type:
|
||
<strong id="editSourceTypeBadge">—</strong></div>
|
||
</div>
|
||
|
||
<!-- Keboola/Jira fallback fields. The richer Keboola modal
|
||
lives at #editKeboolaModal; this remains the catch-all
|
||
for any source_type that's neither bigquery nor keboola
|
||
(e.g. jira). -->
|
||
<div class="form-group keboola-edit-only">
|
||
<label class="form-label" for="editStrategy">Sync Strategy</label>
|
||
<select class="form-select" id="editStrategy">
|
||
<option value="full_refresh">Full Refresh</option>
|
||
<option value="incremental">Incremental</option>
|
||
<option value="partitioned">Partitioned</option>
|
||
</select>
|
||
</div>
|
||
<div class="form-group keboola-edit-only">
|
||
<label class="form-label" for="editPrimaryKey">Primary Key</label>
|
||
<input type="text" class="form-input" id="editPrimaryKey" placeholder="e.g. id">
|
||
</div>
|
||
|
||
<div class="form-group">
|
||
<label class="form-label" for="editDescription">Description <span class="optional">(optional)</span></label>
|
||
<textarea class="form-textarea" id="editDescription" placeholder="Brief description of the table contents..."></textarea>
|
||
</div>
|
||
<div class="form-group">
|
||
<label class="form-label" for="editFolder">Folder <span class="optional">(optional)</span></label>
|
||
<input type="text" class="form-input" id="editFolder" placeholder="e.g. crm, finance, marketing">
|
||
<div class="form-hint">Logical grouping for catalog organization (does not affect storage).</div>
|
||
</div>
|
||
</div>
|
||
<div class="modal-footer">
|
||
<button class="btn btn-secondary" onclick="closeEditModal()">Cancel</button>
|
||
<button class="btn btn-primary" id="editSubmitBtn" onclick="saveTableEdit()">
|
||
Save Changes
|
||
</button>
|
||
</div>
|
||
</div>
|
||
</div>
|
||
|
||
<!-- ═══════════════ TOAST ═══════════════ -->
|
||
<div class="toast" id="toast">
|
||
<div class="toast-icon" id="toastIcon"></div>
|
||
<span id="toastMessage"></span>
|
||
</div>
|
||
|
||
<!-- ═══════════════ FOOTER ═══════════════ -->
|
||
<footer class="footer">
|
||
<a href="{{ url_for('dashboard') }}">Back to Dashboard</a>
|
||
</footer>
|
||
<script>
|
||
/* ═══════════════════════════════════════════════════════════════
|
||
Admin Tables - JavaScript
|
||
═══════════════════════════════════════════════════════════════ */
|
||
|
||
// ── Tab nav (Phase D) ───────────────────────────────────────
|
||
function switchTab(tab) {
|
||
document.querySelectorAll('.tab').forEach(function(b) {
|
||
b.setAttribute('aria-selected', b.dataset.tab === tab ? 'true' : 'false');
|
||
});
|
||
document.querySelectorAll('.tab-content').forEach(function(c) {
|
||
c.style.display = (c.id === ('tab-content-' + tab)) ? 'block' : 'none';
|
||
});
|
||
history.replaceState(null, '', '#' + tab);
|
||
}
|
||
|
||
function getActiveTabFromHash() {
|
||
var hash = window.location.hash.replace(/^#/, '');
|
||
if (hash === 'bigquery' || hash === 'keboola' || hash === 'jira' || hash === 'internal') {
|
||
return hash;
|
||
}
|
||
return null;
|
||
}
|
||
|
||
(function initTabFromHash() {
|
||
var t = getActiveTabFromHash();
|
||
if (t) switchTab(t);
|
||
})();
|
||
|
||
// State
|
||
let registryData = null;
|
||
let registryVersion = null;
|
||
let currentEditTableId = null;
|
||
|
||
// ── Toast notification ──────────────────────────────────────
|
||
|
||
function showToast(message, type) {
|
||
var toast = document.getElementById('toast');
|
||
var icon = document.getElementById('toastIcon');
|
||
var msg = document.getElementById('toastMessage');
|
||
|
||
toast.className = 'toast toast-' + type;
|
||
msg.textContent = message;
|
||
|
||
if (type === 'success') {
|
||
icon.innerHTML = '<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="#10B77F" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M22 11.08V12a10 10 0 1 1-5.93-9.14"/><polyline points="22 4 12 14.01 9 11.01"/></svg>';
|
||
} else {
|
||
icon.innerHTML = '<svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="#EA580C" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10"/><line x1="15" y1="9" x2="9" y2="15"/><line x1="9" y1="9" x2="15" y2="15"/></svg>';
|
||
}
|
||
|
||
// Show
|
||
requestAnimationFrame(function() {
|
||
toast.classList.add('visible');
|
||
});
|
||
|
||
// Hide after 4 seconds
|
||
setTimeout(function() {
|
||
toast.classList.remove('visible');
|
||
}, 4000);
|
||
}
|
||
|
||
// ── Format helpers ──────────────────────────────────────────
|
||
|
||
function formatNumber(n) {
|
||
if (n == null) return '-';
|
||
if (n >= 1000000) return (n / 1000000).toFixed(1) + 'M';
|
||
if (n >= 1000) return n.toLocaleString();
|
||
return String(n);
|
||
}
|
||
|
||
function formatSize(bytes) {
|
||
if (bytes == null || bytes === 0) return '-';
|
||
if (bytes >= 1073741824) return (bytes / 1073741824).toFixed(1) + ' GB';
|
||
if (bytes >= 1048576) return (bytes / 1048576).toFixed(1) + ' MB';
|
||
if (bytes >= 1024) return (bytes / 1024).toFixed(1) + ' KB';
|
||
return bytes + ' B';
|
||
}
|
||
|
||
function escapeHtml(str) {
|
||
if (!str) return '';
|
||
var div = document.createElement('div');
|
||
div.textContent = str;
|
||
return div.innerHTML;
|
||
}
|
||
|
||
/**
|
||
* Escape a string for safe inclusion inside a single- OR double-quoted
|
||
* HTML attribute. Unlike `escapeHtml` (which goes through textContent →
|
||
* innerHTML and only escapes `<`/`>`/`&`), this also escapes both quote
|
||
* characters so the value can't break out of the attribute. Issue #265.
|
||
*/
|
||
function escapeHtmlAttr(str) {
|
||
if (str === null || str === undefined) return '';
|
||
return String(str)
|
||
.replace(/&/g, '&')
|
||
.replace(/</g, '<')
|
||
.replace(/>/g, '>')
|
||
.replace(/"/g, '"')
|
||
.replace(/'/g, ''');
|
||
}
|
||
|
||
// Defensive normalization for descriptions registered via shell-quoting
|
||
// tooling that injected literal backslash escapes (e.g. `Don\'t`, `\n`).
|
||
// Mirrors _unescape_shell_quoting in app/api/admin.py — applied at render
|
||
// time so already-stored corrupt rows still display readably.
|
||
function unescapeShellQuoting(s) {
|
||
if (!s) return s;
|
||
// Order matters: protect real backslashes via NUL sentinel first,
|
||
// unescape the well-known sequences, then restore real backslashes.
|
||
return s
|
||
.replace(/\\\\/g, ' ')
|
||
.replace(/\\n/g, '\n')
|
||
.replace(/\\r/g, '\r')
|
||
.replace(/\\t/g, '\t')
|
||
.replace(/\\'/g, "'")
|
||
.replace(/\\"/g, '"')
|
||
.replace(/ /g, '\\');
|
||
}
|
||
|
||
// C3: removed dead Discovery panel JS. The global Discovery card +
|
||
// its #discoverBtn / #discoveryResults DOM hooks were removed when
|
||
// the per-tab UI landed; per-tab Discover/List datalist helpers live
|
||
// in the per-source shims further down. The legacy "Register" button
|
||
// rendered per discovery row also went away — operators register
|
||
// through the per-tab Register modals.
|
||
|
||
// ── Registration Modal ──────────────────────────────────────
|
||
|
||
// Server-rendered marker so the JS knows the instance's data source
|
||
// type. Lives on <body> after C3 (previously on the now-removed
|
||
// #registerModal).
|
||
var DATA_SOURCE_TYPE = document.body.dataset.sourceType || 'keboola';
|
||
|
||
function openRegisterModal(arg) {
|
||
// Phase E + F: dispatch by string argument.
|
||
// 'bigquery' → BQ tab Register button → #registerBqModal.
|
||
// 'keboola' → Keboola tab Register button → #registerKeboolaModal.
|
||
if (arg === 'bigquery') {
|
||
return _openBqRegisterModal({});
|
||
}
|
||
if (arg === 'keboola') {
|
||
return _openKeboolaTabRegisterModal();
|
||
}
|
||
// Fallback when called with no argument — pick by instance type.
|
||
if (DATA_SOURCE_TYPE === 'bigquery') {
|
||
return _openBqRegisterModal({});
|
||
}
|
||
return _openKeboolaTabRegisterModal();
|
||
}
|
||
|
||
function _openBqRegisterModal(table) {
|
||
// BQ uses a manual-entry form (no discovery panel for BQ in M1).
|
||
// `table` may be partially populated by a future M2 prefill —
|
||
// tolerate either an empty call or a {bucket, source_table, ...}
|
||
// shape from a hypothetical future prefill.
|
||
table = table || {};
|
||
document.getElementById('bqDataset').value = table.bucket || '';
|
||
document.getElementById('bqSourceTable').value = table.source_table || table.name || '';
|
||
document.getElementById('bqViewName').value = table.name || '';
|
||
document.getElementById('bqDescription').value = '';
|
||
document.getElementById('bqFolder').value = '';
|
||
document.getElementById('bqSyncSchedule').value = '';
|
||
var summary = document.getElementById('bqPrecheckSummary');
|
||
if (summary) summary.style.display = 'none';
|
||
var btn = document.getElementById('registerBqSubmitBtn');
|
||
btn.disabled = false;
|
||
btn.textContent = 'Register Table';
|
||
btn.onclick = registerBqTable;
|
||
document.getElementById('registerBqModal').classList.add('active');
|
||
}
|
||
|
||
// C3: removed dead helpers _openKeboolaRegisterModal /
|
||
// closeRegisterModal that drove the now-deleted #registerModal.
|
||
// The Phase F #registerKeboolaModal owns the Keboola flow now.
|
||
|
||
function closeRegisterBqModal() {
|
||
document.getElementById('registerBqModal').classList.remove('active');
|
||
}
|
||
|
||
// ── Keboola tab register modal (Phase F1) ──────────────────────
|
||
|
||
function _openKeboolaTabRegisterModal() {
|
||
// Reset form to defaults each open. Whole mode is the default
|
||
// (Q2='whole'); the kb-source-table fields are visible.
|
||
var modal = document.getElementById('registerKeboolaModal');
|
||
if (!modal) return;
|
||
var radio = modal.querySelector('input[name="kbSyncMode"][value="whole"]');
|
||
if (radio) radio.checked = true;
|
||
['kbViewName', 'kbBucket', 'kbSourceTable', 'kbSourceQuery',
|
||
'kbSyncSchedule', 'kbDescription', 'kbFolder', 'kbPrimaryKey',
|
||
// v26 Direct-extract fields — reset so a stale value from a prior
|
||
// open doesn't leak into the next registration.
|
||
'kbIncrementalWindowDays', 'kbMaxHistoryDays', 'kbPartitionBy',
|
||
'kbInitialLoadChunkDays', 'kbWhereFilters'].forEach(function(id) {
|
||
var el = document.getElementById(id);
|
||
if (el) el.value = '';
|
||
});
|
||
// Default v26 dropdowns
|
||
var kbStrategyEl = document.getElementById('kbStrategy');
|
||
if (kbStrategyEl) kbStrategyEl.value = 'full_refresh';
|
||
var kbGranEl = document.getElementById('kbPartitionGranularity');
|
||
if (kbGranEl) kbGranEl.value = 'month';
|
||
onKbSyncModeChange(); // apply default visibility
|
||
var btn = document.getElementById('registerKeboolaSubmitBtn');
|
||
btn.disabled = false;
|
||
btn.textContent = 'Register';
|
||
modal.classList.add('active');
|
||
}
|
||
|
||
function closeRegisterKeboolaModal() {
|
||
document.getElementById('registerKeboolaModal').classList.remove('active');
|
||
}
|
||
|
||
function _getKbSyncMode() {
|
||
var el = document.querySelector('input[name="kbSyncMode"]:checked');
|
||
return el ? el.value : 'whole';
|
||
}
|
||
|
||
function onKbSyncModeChange() {
|
||
var mode = _getKbSyncMode();
|
||
// Whole + Direct both need bucket/source_table inputs; only Custom hides them
|
||
document.querySelectorAll('.kb-source-table').forEach(function(el) {
|
||
el.style.display = (mode === 'custom') ? 'none' : '';
|
||
});
|
||
document.querySelectorAll('.kb-source-custom').forEach(function(el) {
|
||
el.style.display = (mode === 'custom') ? '' : 'none';
|
||
});
|
||
// v26 strategy panel — Direct mode only
|
||
document.querySelectorAll('.kb-direct-only').forEach(function(el) {
|
||
el.style.display = (mode === 'direct') ? '' : 'none';
|
||
});
|
||
if (mode === 'direct') onKbStrategyChange();
|
||
}
|
||
|
||
function _getKbStrategy() {
|
||
var el = document.getElementById('kbStrategy');
|
||
return el ? el.value : 'full_refresh';
|
||
}
|
||
|
||
function _kbStrategyVisibility(el, s) {
|
||
// Single-pass visibility check that handles elements tagged with
|
||
// multiple strategy classes (e.g. window_days has both
|
||
// kb-strategy-incremental + kb-strategy-partitioned because the
|
||
// field is shared). Using two separate forEach loops would race —
|
||
// the second loop hiding what the first showed.
|
||
if (el.classList.contains('kb-strategy-not-incremental')) {
|
||
return s !== 'incremental';
|
||
}
|
||
// Element is visible if its class list mentions the active strategy
|
||
return el.classList.contains('kb-strategy-' + s);
|
||
}
|
||
|
||
function onKbStrategyChange() {
|
||
var s = _getKbStrategy();
|
||
document.querySelectorAll(
|
||
'.kb-direct-only .kb-strategy-incremental, ' +
|
||
'.kb-direct-only .kb-strategy-partitioned, ' +
|
||
'.kb-direct-only .kb-strategy-not-incremental'
|
||
).forEach(function(el) {
|
||
el.style.display = _kbStrategyVisibility(el, s) ? '' : 'none';
|
||
});
|
||
}
|
||
|
||
function _intOrNull(id) {
|
||
var raw = (document.getElementById(id).value || '').trim();
|
||
if (!raw) return null;
|
||
var n = parseInt(raw, 10);
|
||
return isNaN(n) ? null : n;
|
||
}
|
||
|
||
function _parseFiltersJSONOrThrow(id) {
|
||
var raw = (document.getElementById(id).value || '').trim();
|
||
if (!raw) return null;
|
||
try {
|
||
return JSON.parse(raw);
|
||
} catch (e) {
|
||
throw new Error('where_filters: invalid JSON — ' + e.message);
|
||
}
|
||
}
|
||
|
||
function _buildKeboolaPayload() {
|
||
// Three modes:
|
||
// whole / custom → query_mode='materialized' (DuckDB Keboola extension)
|
||
// direct → query_mode='local' + v26 sync_strategy fields
|
||
// (Storage API SDK, supports incremental/partitioned/where_filters)
|
||
var mode = _getKbSyncMode();
|
||
var viewName = (document.getElementById('kbViewName').value || '').trim();
|
||
var bucket = (document.getElementById('kbBucket').value || '').trim();
|
||
var sourceTable = (document.getElementById('kbSourceTable').value || '').trim();
|
||
var pk = (document.getElementById('kbPrimaryKey').value || '').trim();
|
||
var primaryKey = pk
|
||
? pk.split(',').map(function(s) { return s.trim(); }).filter(Boolean)
|
||
: [];
|
||
|
||
if (mode === 'direct') {
|
||
var strategy = _getKbStrategy();
|
||
var payload = {
|
||
name: viewName || sourceTable,
|
||
source_type: 'keboola',
|
||
query_mode: 'local',
|
||
bucket: bucket,
|
||
source_table: sourceTable,
|
||
primary_key: primaryKey,
|
||
sync_strategy: strategy,
|
||
sync_schedule: (document.getElementById('kbSyncSchedule').value || '').trim() || null,
|
||
description: (document.getElementById('kbDescription').value || '').trim() || null,
|
||
folder: (document.getElementById('kbFolder').value || '').trim() || null,
|
||
};
|
||
if (strategy === 'incremental' || strategy === 'partitioned') {
|
||
payload.incremental_window_days = _intOrNull('kbIncrementalWindowDays');
|
||
payload.max_history_days = _intOrNull('kbMaxHistoryDays');
|
||
}
|
||
if (strategy === 'partitioned') {
|
||
payload.partition_by = (document.getElementById('kbPartitionBy').value || '').trim() || null;
|
||
payload.partition_granularity = document.getElementById('kbPartitionGranularity').value || null;
|
||
payload.initial_load_chunk_days = _intOrNull('kbInitialLoadChunkDays');
|
||
}
|
||
if (strategy !== 'incremental') {
|
||
payload.where_filters = _parseFiltersJSONOrThrow('kbWhereFilters');
|
||
}
|
||
return payload;
|
||
}
|
||
|
||
// Materialized paths (whole / custom)
|
||
var common = {
|
||
name: viewName || sourceTable,
|
||
source_type: 'keboola',
|
||
query_mode: 'materialized',
|
||
primary_key: primaryKey,
|
||
sync_schedule: (document.getElementById('kbSyncSchedule').value || '').trim() || null,
|
||
description: (document.getElementById('kbDescription').value || '').trim() || null,
|
||
folder: (document.getElementById('kbFolder').value || '').trim() || null,
|
||
};
|
||
|
||
if (mode === 'custom') {
|
||
return Object.assign({}, common, {
|
||
source_query: (document.getElementById('kbSourceQuery').value || '').trim(),
|
||
});
|
||
}
|
||
// Whole — synthesize SELECT * FROM kbc."bucket"."table".
|
||
return Object.assign({}, common, {
|
||
bucket: bucket,
|
||
source_table: sourceTable,
|
||
source_query: 'SELECT * FROM kbc."' + bucket + '"."' + sourceTable + '"',
|
||
});
|
||
}
|
||
|
||
function registerKeboolaTable() {
|
||
var btn = document.getElementById('registerKeboolaSubmitBtn');
|
||
btn.disabled = true;
|
||
btn.textContent = 'Registering...';
|
||
var payload;
|
||
try {
|
||
payload = _buildKeboolaPayload();
|
||
} catch (e) {
|
||
// _parseFiltersJSONOrThrow surfaces malformed JSON before the
|
||
// POST so the operator sees the parse error inline rather than
|
||
// a 422 from the server. Re-enable the submit button.
|
||
showToast('' + e.message, 'error');
|
||
btn.disabled = false;
|
||
btn.textContent = 'Register';
|
||
return;
|
||
}
|
||
fetch('/api/admin/register-table', {
|
||
method: 'POST',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify(payload),
|
||
})
|
||
.then(function(r) {
|
||
if (!r.ok) {
|
||
return r.json().then(function(d) {
|
||
throw new Error(d.detail || d.error || 'Registration failed');
|
||
});
|
||
}
|
||
return r.json();
|
||
})
|
||
.then(function() {
|
||
closeRegisterKeboolaModal();
|
||
showToast('Table registered', 'success');
|
||
loadRegistry();
|
||
})
|
||
.catch(function(err) {
|
||
showToast('' + err.message, 'error');
|
||
})
|
||
.finally(function() {
|
||
btn.disabled = false;
|
||
btn.textContent = 'Register';
|
||
});
|
||
}
|
||
|
||
// Discovery shims — the existing /api/admin/discover-tables endpoint
|
||
// already routes by the instance's data_source.type (returning Keboola
|
||
// tables when the instance is Keboola-typed); the source=keboola query
|
||
// param is informational. Hidden behind `data_source_type == 'keboola'`
|
||
// because on a BQ-typed instance the endpoint would return BQ-shaped
|
||
// data (wrong shape, confusing); operators fall back to manual entry
|
||
// for cross-source registration. Future work: extend the endpoint to
|
||
// accept an explicit ?source= override so secondary-source registration
|
||
// works in both directions and we can remove this guard.
|
||
{% if data_source_type == 'keboola' %}
|
||
function discoverKeboolaBuckets(datalistId) {
|
||
fetch('/api/admin/discover-tables?source=keboola')
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) {
|
||
throw new Error(d.detail || d.error || 'Keboola discovery failed');
|
||
});
|
||
return r.json();
|
||
})
|
||
.then(function(data) {
|
||
var dl = document.getElementById(datalistId);
|
||
if (!dl) return;
|
||
dl.innerHTML = '';
|
||
// Endpoint may return either {buckets:[...]}, {datasets:[...]}
|
||
// or {tables:[...]} depending on routing; project to a flat
|
||
// bucket-id list. Keboola path returns tables → derive uniq
|
||
// bucket_ids.
|
||
var buckets = data.buckets || data.datasets;
|
||
if (!buckets && Array.isArray(data.tables)) {
|
||
var seen = {};
|
||
buckets = [];
|
||
data.tables.forEach(function(t) {
|
||
var b = t.bucket_id || (t.bucket && t.bucket.id);
|
||
if (b && !seen[b]) { seen[b] = 1; buckets.push(b); }
|
||
});
|
||
}
|
||
(buckets || []).forEach(function(b) {
|
||
var o = document.createElement('option');
|
||
o.value = (typeof b === 'string') ? b : (b.id || b.bucket_id || '');
|
||
dl.appendChild(o);
|
||
});
|
||
showToast('Loaded ' + (dl.children.length) + ' buckets', 'success');
|
||
})
|
||
.catch(function(err) {
|
||
showToast('' + err.message, 'error');
|
||
});
|
||
}
|
||
|
||
function discoverKeboolaTables(bucketInputId, tablesDatalistId) {
|
||
var bucketEl = document.getElementById(bucketInputId);
|
||
var bucket = bucketEl ? (bucketEl.value || '').trim() : '';
|
||
if (!bucket) {
|
||
showToast('Fill bucket first', 'error');
|
||
return;
|
||
}
|
||
fetch('/api/admin/discover-tables?source=keboola&bucket=' + encodeURIComponent(bucket))
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) {
|
||
throw new Error(d.detail || d.error || 'Keboola table discovery failed');
|
||
});
|
||
return r.json();
|
||
})
|
||
.then(function(data) {
|
||
var dl = document.getElementById(tablesDatalistId);
|
||
if (!dl) return;
|
||
dl.innerHTML = '';
|
||
var tables = data.tables || [];
|
||
// Filter to the selected bucket if endpoint didn't.
|
||
tables.filter(function(t) {
|
||
var b = t.bucket_id || (t.bucket && t.bucket.id);
|
||
return !bucket || !b || b === bucket;
|
||
}).forEach(function(t) {
|
||
var o = document.createElement('option');
|
||
var name = (typeof t === 'string') ? t : (t.name || t.id || '');
|
||
// Strip bucket prefix if present.
|
||
if (name.indexOf(bucket + '.') === 0) name = name.substring(bucket.length + 1);
|
||
o.value = name;
|
||
dl.appendChild(o);
|
||
});
|
||
showToast('Loaded ' + dl.children.length + ' tables in ' + bucket, 'success');
|
||
})
|
||
.catch(function(err) {
|
||
showToast('' + err.message, 'error');
|
||
});
|
||
}
|
||
|
||
// ── Keboola tab edit modal (Phase F2) ──────────────────────────
|
||
|
||
function _getEditKbSyncMode() {
|
||
var el = document.querySelector('input[name="editKbSyncMode"]:checked');
|
||
return el ? el.value : 'whole';
|
||
}
|
||
|
||
function onEditKbSyncModeChange() {
|
||
var mode = _getEditKbSyncMode();
|
||
document.querySelectorAll('.editkb-source-table').forEach(function(el) {
|
||
el.style.display = (mode === 'custom') ? 'none' : '';
|
||
});
|
||
document.querySelectorAll('.editkb-source-custom').forEach(function(el) {
|
||
el.style.display = (mode === 'custom') ? '' : 'none';
|
||
});
|
||
document.querySelectorAll('.editkb-direct-only').forEach(function(el) {
|
||
el.style.display = (mode === 'direct') ? '' : 'none';
|
||
});
|
||
if (mode === 'direct') onEditKbStrategyChange();
|
||
}
|
||
|
||
function _getEditKbStrategy() {
|
||
var el = document.getElementById('editKbStrategy');
|
||
return el ? el.value : 'full_refresh';
|
||
}
|
||
|
||
function _editKbStrategyVisibility(el, s) {
|
||
if (el.classList.contains('editkb-strategy-not-incremental')) {
|
||
return s !== 'incremental';
|
||
}
|
||
return el.classList.contains('editkb-strategy-' + s);
|
||
}
|
||
|
||
function onEditKbStrategyChange() {
|
||
var s = _getEditKbStrategy();
|
||
document.querySelectorAll(
|
||
'.editkb-direct-only .editkb-strategy-incremental, ' +
|
||
'.editkb-direct-only .editkb-strategy-partitioned, ' +
|
||
'.editkb-direct-only .editkb-strategy-not-incremental'
|
||
).forEach(function(el) {
|
||
el.style.display = _editKbStrategyVisibility(el, s) ? '' : 'none';
|
||
});
|
||
}
|
||
|
||
function _setEditKbRadio(value) {
|
||
var el = document.querySelector('input[name="editKbSyncMode"][value="' + value + '"]');
|
||
if (el) el.checked = true;
|
||
}
|
||
|
||
function openEditKeboolaModal(table) {
|
||
// Populate fields from a registry row. Mode detection:
|
||
// - query_mode='local' → 'direct' (v26 sync-strategy path)
|
||
// - query_mode='materialized' + auto SELECT * → 'whole'
|
||
// - query_mode='materialized' + custom SQL → 'custom'
|
||
table = table || {};
|
||
document.getElementById('editKbTableId').value = table.id || '';
|
||
var bucket = table.bucket || '';
|
||
var sourceTable = table.source_table || '';
|
||
var sourceQuery = table.source_query || '';
|
||
var queryMode = table.query_mode || 'materialized';
|
||
var isAutoSelectStar = false;
|
||
if (sourceQuery && bucket && sourceTable) {
|
||
var auto = 'SELECT * FROM kbc."' + bucket + '"."' + sourceTable + '"';
|
||
isAutoSelectStar = sourceQuery.replace(/\s+/g, ' ').trim() === auto;
|
||
}
|
||
var mode;
|
||
if (queryMode === 'local') {
|
||
mode = 'direct';
|
||
} else if (sourceQuery && !isAutoSelectStar) {
|
||
mode = 'custom';
|
||
} else {
|
||
mode = 'whole';
|
||
}
|
||
_setEditKbRadio(mode);
|
||
document.getElementById('editKbBucket').value = bucket;
|
||
document.getElementById('editKbSourceTable').value = sourceTable;
|
||
document.getElementById('editKbSourceQuery').value = sourceQuery;
|
||
document.getElementById('editKbSyncSchedule').value = table.sync_schedule || '';
|
||
document.getElementById('editKbDescription').value = table.description || '';
|
||
document.getElementById('editKbFolder').value = table.folder || '';
|
||
document.getElementById('editKbPrimaryKey').value = (table.primary_key || []).join(', ');
|
||
|
||
// v26 fields
|
||
document.getElementById('editKbStrategy').value = table.sync_strategy || 'full_refresh';
|
||
document.getElementById('editKbIncrementalWindowDays').value =
|
||
table.incremental_window_days != null ? table.incremental_window_days : '';
|
||
document.getElementById('editKbMaxHistoryDays').value =
|
||
table.max_history_days != null ? table.max_history_days : '';
|
||
document.getElementById('editKbPartitionBy').value = table.partition_by || '';
|
||
document.getElementById('editKbPartitionGranularity').value =
|
||
table.partition_granularity || 'month';
|
||
document.getElementById('editKbInitialLoadChunkDays').value =
|
||
table.initial_load_chunk_days != null ? table.initial_load_chunk_days : '';
|
||
document.getElementById('editKbWhereFilters').value =
|
||
table.where_filters ? JSON.stringify(table.where_filters, null, 2) : '';
|
||
|
||
onEditKbSyncModeChange();
|
||
var btn = document.getElementById('editKeboolaSubmitBtn');
|
||
btn.disabled = false;
|
||
btn.textContent = 'Save Changes';
|
||
document.getElementById('editKeboolaModal').classList.add('active');
|
||
}
|
||
|
||
function closeEditKeboolaModal() {
|
||
document.getElementById('editKeboolaModal').classList.remove('active');
|
||
}
|
||
|
||
function _editIntOrNull(id) {
|
||
var raw = (document.getElementById(id).value || '').trim();
|
||
if (!raw) return null;
|
||
var n = parseInt(raw, 10);
|
||
return isNaN(n) ? null : n;
|
||
}
|
||
|
||
function _editParseFiltersJSONOrThrow(id) {
|
||
var raw = (document.getElementById(id).value || '').trim();
|
||
if (!raw) return null;
|
||
try {
|
||
return JSON.parse(raw);
|
||
} catch (e) {
|
||
throw new Error('where_filters: invalid JSON — ' + e.message);
|
||
}
|
||
}
|
||
|
||
function _buildKeboolaEditPayload() {
|
||
var mode = _getEditKbSyncMode();
|
||
var bucket = (document.getElementById('editKbBucket').value || '').trim();
|
||
var sourceTable = (document.getElementById('editKbSourceTable').value || '').trim();
|
||
var pk = (document.getElementById('editKbPrimaryKey').value || '').trim();
|
||
var primaryKey = pk
|
||
? pk.split(',').map(function(s) { return s.trim(); }).filter(Boolean)
|
||
: [];
|
||
|
||
if (mode === 'direct') {
|
||
var strategy = _getEditKbStrategy();
|
||
var payload = {
|
||
query_mode: 'local',
|
||
bucket: bucket,
|
||
source_table: sourceTable,
|
||
primary_key: primaryKey,
|
||
sync_strategy: strategy,
|
||
sync_schedule: (document.getElementById('editKbSyncSchedule').value || '').trim() || null,
|
||
description: (document.getElementById('editKbDescription').value || '').trim() || null,
|
||
folder: (document.getElementById('editKbFolder').value || '').trim() || null,
|
||
// PUT requires explicit nulls to clear v26 fields when switching
|
||
// away from a strategy that used them — otherwise the merged-dict
|
||
// path keeps the stale value.
|
||
incremental_window_days: null,
|
||
max_history_days: null,
|
||
partition_by: null,
|
||
partition_granularity: null,
|
||
initial_load_chunk_days: null,
|
||
where_filters: null,
|
||
};
|
||
if (strategy === 'incremental' || strategy === 'partitioned') {
|
||
payload.incremental_window_days = _editIntOrNull('editKbIncrementalWindowDays');
|
||
payload.max_history_days = _editIntOrNull('editKbMaxHistoryDays');
|
||
}
|
||
if (strategy === 'partitioned') {
|
||
payload.partition_by = (document.getElementById('editKbPartitionBy').value || '').trim() || null;
|
||
payload.partition_granularity = document.getElementById('editKbPartitionGranularity').value || null;
|
||
payload.initial_load_chunk_days = _editIntOrNull('editKbInitialLoadChunkDays');
|
||
}
|
||
if (strategy !== 'incremental') {
|
||
payload.where_filters = _editParseFiltersJSONOrThrow('editKbWhereFilters');
|
||
}
|
||
return payload;
|
||
}
|
||
|
||
var common = {
|
||
query_mode: 'materialized',
|
||
primary_key: primaryKey,
|
||
sync_schedule: (document.getElementById('editKbSyncSchedule').value || '').trim() || null,
|
||
description: (document.getElementById('editKbDescription').value || '').trim() || null,
|
||
folder: (document.getElementById('editKbFolder').value || '').trim() || null,
|
||
};
|
||
if (mode === 'custom') {
|
||
return Object.assign({}, common, {
|
||
source_query: (document.getElementById('editKbSourceQuery').value || '').trim(),
|
||
});
|
||
}
|
||
return Object.assign({}, common, {
|
||
bucket: bucket,
|
||
source_table: sourceTable,
|
||
source_query: 'SELECT * FROM kbc."' + bucket + '"."' + sourceTable + '"',
|
||
});
|
||
}
|
||
|
||
function saveKeboolaTabEdit() {
|
||
var btn = document.getElementById('editKeboolaSubmitBtn');
|
||
var tableId = document.getElementById('editKbTableId').value;
|
||
if (!tableId) {
|
||
showToast('Missing table id', 'error');
|
||
return;
|
||
}
|
||
btn.disabled = true;
|
||
btn.textContent = 'Saving...';
|
||
var payload;
|
||
try {
|
||
payload = _buildKeboolaEditPayload();
|
||
} catch (e) {
|
||
showToast('' + e.message, 'error');
|
||
btn.disabled = false;
|
||
btn.textContent = 'Save Changes';
|
||
return;
|
||
}
|
||
fetch('/api/admin/registry/' + encodeURIComponent(tableId), {
|
||
method: 'PUT',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify(payload),
|
||
})
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) {
|
||
throw new Error(d.detail || d.error || 'Update failed');
|
||
});
|
||
return r.json();
|
||
})
|
||
.then(function() {
|
||
closeEditKeboolaModal();
|
||
showToast('Table updated', 'success');
|
||
loadRegistry();
|
||
})
|
||
.catch(function(err) {
|
||
showToast('' + err.message, 'error');
|
||
})
|
||
.finally(function() {
|
||
btn.disabled = false;
|
||
btn.textContent = 'Save Changes';
|
||
});
|
||
}
|
||
|
||
function prefillFromKeboolaTable(textareaId) {
|
||
// Edit-modal callers may pass an editKbBucket / editKbSourceTable
|
||
// pair instead of the Register modal's kbBucket / kbSourceTable.
|
||
// Detect the modal via the textarea id prefix.
|
||
var prefix = textareaId.indexOf('editKb') === 0 ? 'editKb' : 'kb';
|
||
var bucket = (document.getElementById(prefix + 'Bucket').value || '').trim();
|
||
var sourceTable = (document.getElementById(prefix + 'SourceTable').value || '').trim();
|
||
if (!bucket || !sourceTable) {
|
||
showToast('Fill bucket + source table first', 'error');
|
||
return;
|
||
}
|
||
var ta = document.getElementById(textareaId);
|
||
if (ta.value.trim()) {
|
||
if (!confirm('Replace existing SQL?')) return;
|
||
}
|
||
ta.value = 'SELECT *\nFROM kbc."' + bucket + '"."' + sourceTable + '"\nWHERE -- your filter here';
|
||
}
|
||
{% endif %}{# data_source_type == 'keboola' — discover/prefill JS #}
|
||
|
||
// C3: removed dead helper _buildKeboolaLegacyPayload — the Phase F
|
||
// _buildKeboolaPayload (above) replaced it.
|
||
|
||
function _getBqAccessMode() {
|
||
// Q1 radio. Default 'live' if nothing's checked yet (model-validator
|
||
// safety net for the initial render).
|
||
var el = document.querySelector('input[name="bqAccessMode"]:checked');
|
||
return el ? el.value : 'live';
|
||
}
|
||
|
||
function _getBqSyncMode() {
|
||
// Q2 radio (only meaningful when access mode is 'synced').
|
||
var el = document.querySelector('input[name="bqSyncMode"]:checked');
|
||
return el ? el.value : 'whole';
|
||
}
|
||
|
||
function _buildBigQueryPayload() {
|
||
// Two-question form maps to backend `query_mode`:
|
||
// live → query_mode='remote' (server auto-detects
|
||
// BASE TABLE vs VIEW)
|
||
// synced/whole → query_mode='materialized' (auto SELECT *)
|
||
// synced/custom → query_mode='materialized' (admin SQL)
|
||
// The UI never asks "Table vs View" — that's a server-side detail.
|
||
var accessMode = _getBqAccessMode();
|
||
var syncMode = _getBqSyncMode();
|
||
var viewName = document.getElementById('bqViewName').value.trim();
|
||
var description = document.getElementById('bqDescription').value.trim() || null;
|
||
var folder = document.getElementById('bqFolder').value.trim() || null;
|
||
var syncSchedule = document.getElementById('bqSyncSchedule').value.trim() || null;
|
||
|
||
if (accessMode === 'synced' && syncMode === 'custom') {
|
||
return {
|
||
name: viewName,
|
||
source_type: 'bigquery',
|
||
query_mode: 'materialized',
|
||
source_query: document.getElementById('bqSourceQuery').value.trim(),
|
||
profile_after_sync: false,
|
||
description: description,
|
||
folder: folder,
|
||
sync_schedule: syncSchedule,
|
||
};
|
||
}
|
||
|
||
var dataset = document.getElementById('bqDataset').value.trim();
|
||
var sourceTable = document.getElementById('bqSourceTable').value.trim();
|
||
|
||
if (accessMode === 'synced' && syncMode === 'whole') {
|
||
// Whole-table sync. We don't ship the project to the browser, so
|
||
// the SQL uses DuckDB three-part syntax against the materialize
|
||
// session's ATTACH alias. Native BQ dry-run can't parse this form
|
||
// (DuckDB identifier quoting), so the cost guardrail falls
|
||
// fail-open with a warning — operator who needs the cap to engage
|
||
// picks Custom query and writes backtick-quoted native BQ
|
||
// identifiers.
|
||
//
|
||
// Issue #266: also persist bucket+source_table on the row so the
|
||
// Edit modal can pre-fill those inputs on a subsequent open. Pre-
|
||
// #266 the register flow only sent source_query, leaving bucket=
|
||
// NULL in the registry; the Edit modal then loaded empty inputs
|
||
// and an admin saving with the visible empty fields would
|
||
// synthesize the broken `SELECT * FROM bq."".""` SQL.
|
||
return {
|
||
name: viewName || sourceTable,
|
||
source_type: 'bigquery',
|
||
query_mode: 'materialized',
|
||
bucket: dataset,
|
||
source_table: sourceTable,
|
||
source_query: 'SELECT * FROM bq."' + dataset + '"."' + sourceTable + '"',
|
||
profile_after_sync: false,
|
||
description: description,
|
||
folder: folder,
|
||
sync_schedule: syncSchedule,
|
||
};
|
||
}
|
||
|
||
// Live access — server auto-detects BASE TABLE vs VIEW at register
|
||
// time, so the UI doesn't make the operator pick.
|
||
return {
|
||
name: viewName || sourceTable,
|
||
source_type: 'bigquery',
|
||
bucket: dataset,
|
||
source_table: sourceTable,
|
||
query_mode: 'remote',
|
||
profile_after_sync: false,
|
||
description: description,
|
||
folder: folder,
|
||
sync_schedule: syncSchedule,
|
||
};
|
||
}
|
||
|
||
function onBqAccessModeChange() {
|
||
// Q1: toggle live ↔ synced. Q2 (sync mode) is meaningful only when
|
||
// synced; default it to 'whole' on first reveal so the form stays
|
||
// consistent without forcing the operator to click twice.
|
||
var accessMode = _getBqAccessMode();
|
||
var liveFields = document.querySelectorAll('.bq-access-live');
|
||
var syncedFields = document.querySelectorAll('.bq-access-synced');
|
||
liveFields.forEach(function(el) {
|
||
el.style.display = (accessMode === 'live') ? '' : 'none';
|
||
});
|
||
syncedFields.forEach(function(el) {
|
||
el.style.display = (accessMode === 'synced') ? '' : 'none';
|
||
});
|
||
// Q2 is fresh on first reveal; trigger its handler to apply the
|
||
// source-table vs source-custom visibility rules.
|
||
if (accessMode === 'synced') {
|
||
onBqSyncModeChange();
|
||
} else {
|
||
// Live mode: show source-table fields, hide custom-SQL textarea.
|
||
document.querySelectorAll('.bq-source-table').forEach(function(el) {
|
||
el.style.display = '';
|
||
});
|
||
document.querySelectorAll('.bq-source-custom').forEach(function(el) {
|
||
el.style.display = 'none';
|
||
});
|
||
}
|
||
}
|
||
|
||
function onBqSyncModeChange() {
|
||
// Q2: toggle whole-table ↔ custom-SQL. Whole reuses the
|
||
// dataset/source-table inputs (server-side SELECT *), Custom shows
|
||
// the SQL textarea. Only fires when access mode is already 'synced'.
|
||
var syncMode = _getBqSyncMode();
|
||
var tableFields = document.querySelectorAll('.bq-source-table');
|
||
var customFields = document.querySelectorAll('.bq-source-custom');
|
||
tableFields.forEach(function(el) {
|
||
el.style.display = (syncMode === 'whole') ? '' : 'none';
|
||
});
|
||
customFields.forEach(function(el) {
|
||
el.style.display = (syncMode === 'custom') ? '' : 'none';
|
||
});
|
||
}
|
||
|
||
// The discover / list-tables / prefill helpers are shared between the
|
||
// Register and Edit modals. The default datalist + input ids match the
|
||
// Register modal; Edit-modal callers pass their own ids so the
|
||
// autocomplete populates the right datalist and reads from the right
|
||
// dataset input.
|
||
|
||
function discoverBqDatasets(datalistId) {
|
||
// GET /api/admin/discover-tables (no `dataset`) → list datasets in
|
||
// the configured BQ project. Populate the named <datalist> so the
|
||
// dataset input shows them as autocomplete suggestions. Endpoint
|
||
// routes through BqAccess so config / auth errors come back as the
|
||
// standard {error, kind, details} shape.
|
||
var dlId = datalistId || 'bqDatasetList';
|
||
fetch('/api/admin/discover-tables')
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) {
|
||
var msg = (d && d.detail && d.detail.error) || (d && d.detail) || 'BQ discovery failed';
|
||
throw new Error(msg);
|
||
});
|
||
return r.json();
|
||
})
|
||
.then(function(data) {
|
||
var dl = document.getElementById(dlId);
|
||
dl.innerHTML = '';
|
||
(data.datasets || []).forEach(function(ds) {
|
||
var opt = document.createElement('option');
|
||
opt.value = ds.dataset_id;
|
||
opt.label = ds.full_id || ds.dataset_id;
|
||
dl.appendChild(opt);
|
||
});
|
||
showToast('Loaded ' + (data.count || 0) + ' datasets', 'success');
|
||
})
|
||
.catch(function(err) {
|
||
showToast(err.message || 'Discovery failed', 'error');
|
||
});
|
||
}
|
||
|
||
function discoverBqTables(datasetInputId, tablesDatalistId) {
|
||
// GET /api/admin/discover-tables?dataset=NAME → list tables + views
|
||
// in the dataset. Two-step (dataset → tables) avoids paying the
|
||
// per-dataset list_tables() cost up front on big projects.
|
||
var inId = datasetInputId || 'bqDataset';
|
||
var dlId = tablesDatalistId || 'bqTableList';
|
||
var dataset = document.getElementById(inId).value.trim();
|
||
if (!dataset) {
|
||
showToast('Fill Dataset first', 'error');
|
||
return;
|
||
}
|
||
fetch('/api/admin/discover-tables?dataset=' + encodeURIComponent(dataset))
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) {
|
||
var msg = (d && d.detail && d.detail.error) || (d && d.detail) || 'BQ table discovery failed';
|
||
throw new Error(msg);
|
||
});
|
||
return r.json();
|
||
})
|
||
.then(function(data) {
|
||
var dl = document.getElementById(dlId);
|
||
dl.innerHTML = '';
|
||
(data.tables || []).forEach(function(t) {
|
||
var opt = document.createElement('option');
|
||
opt.value = t.table_id;
|
||
// BQ entity types: TABLE, VIEW, MATERIALIZED_VIEW,
|
||
// EXTERNAL, SNAPSHOT — surface so the operator can
|
||
// pick the right entity option.
|
||
opt.label = t.table_id + (t.table_type ? ' (' + t.table_type + ')' : '');
|
||
dl.appendChild(opt);
|
||
});
|
||
showToast('Loaded ' + (data.count || 0) + ' tables in ' + dataset, 'success');
|
||
})
|
||
.catch(function(err) {
|
||
showToast(err.message || 'Table list failed', 'error');
|
||
});
|
||
}
|
||
|
||
function prefillFromTable(textareaId) {
|
||
// Convenience for the Custom SQL path: prompt for project.dataset.table
|
||
// and prefill `SELECT * FROM \`...\`` so the operator only edits the
|
||
// WHERE / projection. We can't know the project from the form (it's
|
||
// server-side config), so the prompt accepts both `dataset.table` and
|
||
// `project.dataset.table`. Empty / cancel → no change.
|
||
var taId = textareaId || 'bqSourceQuery';
|
||
var ta = document.getElementById(taId);
|
||
var existing = ta.value.trim();
|
||
if (existing && !confirm('SQL field is not empty — overwrite with prefill?')) {
|
||
return;
|
||
}
|
||
var ref = window.prompt(
|
||
'Enter the source table as `dataset.table` or `project.dataset.table` '
|
||
+ '(it will be wrapped in backticks):',
|
||
''
|
||
);
|
||
if (!ref) return;
|
||
ref = ref.trim().replace(/^`|`$/g, '');
|
||
var stub =
|
||
'SELECT *\n'
|
||
+ 'FROM `' + ref + '`\n'
|
||
+ 'WHERE -- TODO: filter or aggregate before the COPY runs (10 GiB cap)\n'
|
||
+ 'LIMIT 1000';
|
||
ta.value = stub;
|
||
}
|
||
|
||
// C3: removed dead entry points registerTable / _registerKeboolaTable
|
||
// that submitted the now-deleted #registerModal. The Keboola Register
|
||
// flow runs through #registerKeboolaModal → registerKeboolaTable().
|
||
|
||
function registerBqTable() {
|
||
// BQ tab modal entry point — runs the same two-step precheck →
|
||
// confirm flow against #registerBqSubmitBtn / #registerBqModal.
|
||
var btn = document.getElementById('registerBqSubmitBtn');
|
||
btn.disabled = true;
|
||
_registerBigQueryTable(btn);
|
||
}
|
||
|
||
function _registerBigQueryTable(btn) {
|
||
// Step 1 of two-step flow: precheck only. After precheck succeeds
|
||
// we surface row count / size / column count in the modal AND swap
|
||
// the primary button to "Register" — operator must explicitly
|
||
// click it to fire the actual register call. Pre-fix this chained
|
||
// precheck → register in a single promise, so the operator never
|
||
// got to review the summary before the row was committed (review
|
||
// IMPORTANT 4 in #119).
|
||
btn.textContent = 'Checking source...';
|
||
var payload = _buildBigQueryPayload();
|
||
var summary = document.getElementById('bqPrecheckSummary');
|
||
var summaryText = document.getElementById('bqPrecheckSummaryText');
|
||
|
||
fetch('/api/admin/register-table/precheck', {
|
||
method: 'POST',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify(payload)
|
||
})
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) { throw new Error(d.detail || 'Precheck failed'); });
|
||
return r.json();
|
||
})
|
||
.then(function(data) {
|
||
// Surface the source-side metrics so the operator can sanity-
|
||
// check the dataset/table before committing the row.
|
||
var t = data.table || {};
|
||
if (summary && summaryText) {
|
||
summary.style.display = '';
|
||
summaryText.textContent =
|
||
formatNumber(t.rows) + ' rows, ' +
|
||
formatSize(t.size_bytes) + ', ' +
|
||
(t.column_count != null ? t.column_count : (t.columns || []).length) + ' columns';
|
||
}
|
||
// Swap the primary button from "Register Table" (which fired
|
||
// the precheck) to "Register" (which fires the actual register
|
||
// call). Reusing the same modal means the operator can still
|
||
// hit Cancel or close without committing.
|
||
btn.disabled = false;
|
||
btn.textContent = 'Register';
|
||
btn.onclick = function() { _confirmRegisterBigQueryTable(btn, payload); };
|
||
})
|
||
.catch(function(err) {
|
||
showToast('Precheck failed: ' + err.message, 'error');
|
||
btn.disabled = false;
|
||
btn.textContent = 'Register Table';
|
||
});
|
||
}
|
||
|
||
function _confirmRegisterBigQueryTable(btn, payload) {
|
||
// Step 2 of the two-step BQ flow: actually POST /register-table.
|
||
// Reached only after the operator has reviewed the precheck
|
||
// summary in the modal and clicked the (relabeled) Register button.
|
||
btn.disabled = true;
|
||
btn.textContent = 'Registering...';
|
||
fetch('/api/admin/register-table', {
|
||
method: 'POST',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify(payload)
|
||
})
|
||
.then(function(r) {
|
||
// 200 (synchronous materialize) and 202 (BG materialize) are
|
||
// success; 500 with a `rebuild_failed` body is the new failure
|
||
// mode where the row is in the registry but the view wasn't
|
||
// created. Surface the error verbatim so the operator can fix
|
||
// it (typically a missing project in the overlay or auth
|
||
// missing bigquery.metadata.get).
|
||
return r.json().then(function(d) {
|
||
if (!r.ok) {
|
||
var msg = d.detail || d.message || 'Registration failed';
|
||
if (d.errors && d.errors.length) {
|
||
msg += ' (' + d.errors.length + ' error' + (d.errors.length > 1 ? 's' : '') + ')';
|
||
}
|
||
throw new Error(msg);
|
||
}
|
||
closeRegisterBqModal();
|
||
if (r.status === 202) {
|
||
showToast('Registration accepted, materializing in background', 'success');
|
||
} else {
|
||
showToast('Queryable as ' + (d.view_name || d.id || payload.name), 'success');
|
||
}
|
||
loadRegistry();
|
||
});
|
||
})
|
||
.catch(function(err) {
|
||
showToast('Registration failed: ' + err.message, 'error');
|
||
})
|
||
.finally(function() {
|
||
// Restore the modal's default button state — the operator may
|
||
// close + reopen the modal, and openRegisterModal also resets
|
||
// onclick back to registerBqTable.
|
||
btn.disabled = false;
|
||
btn.textContent = 'Register Table';
|
||
btn.onclick = registerBqTable;
|
||
});
|
||
}
|
||
|
||
// Legacy entry point — kept for backward-compat with any external
|
||
// wiring that may still call openBigQueryRegisterModal(). Phase E:
|
||
// routes through the new BQ-tab modal.
|
||
function openBigQueryRegisterModal() {
|
||
openRegisterModal('bigquery');
|
||
}
|
||
|
||
// ── Edit Modal ──────────────────────────────────────────────
|
||
|
||
// Original mode captured at openEditModal-time so the access-mode
|
||
// change handler can detect "operator just switched it" vs "loaded
|
||
// from registry" and only surface the destructive-edit warning on a
|
||
// real change.
|
||
let _editOriginalQueryMode = null;
|
||
|
||
function _getEditBqAccessMode() {
|
||
var el = document.querySelector('input[name="editBqAccessMode"]:checked');
|
||
return el ? el.value : 'live';
|
||
}
|
||
|
||
function _getEditBqSyncMode() {
|
||
var el = document.querySelector('input[name="editBqSyncMode"]:checked');
|
||
return el ? el.value : 'whole';
|
||
}
|
||
|
||
function _setEditBqRadio(name, value) {
|
||
var el = document.querySelector('input[name="' + name + '"][value="' + value + '"]');
|
||
if (el) el.checked = true;
|
||
}
|
||
|
||
function openEditModal(table) {
|
||
// C2: dispatch by source_type. BigQuery rows go to the relocated
|
||
// #editBqModal inside the BQ tab; Keboola rows go to the existing
|
||
// #editKeboolaModal inside the Keboola tab; anything else (jira,
|
||
// future sources) falls through to the legacy #editModal which now
|
||
// only carries Description/Folder + a Keboola-style Strategy/PK
|
||
// fallback.
|
||
var sourceType = (table.source_type || '').toLowerCase();
|
||
if (sourceType === 'bigquery') {
|
||
return _openEditBqModal(table);
|
||
}
|
||
if (sourceType === 'keboola') {
|
||
return openEditKeboolaModal(table);
|
||
}
|
||
return _openEditLegacyModal(table);
|
||
}
|
||
|
||
function _openEditBqModal(table) {
|
||
// Populate the BQ-tab Edit modal from a registry row. Mirror of
|
||
// populateEditModal's BQ branch from before C2 — same query_mode →
|
||
// radio mapping, same field bindings, but writes to the new
|
||
// #editBqModal scope.
|
||
currentEditTableId = table.id;
|
||
document.getElementById('editBqTableId').value = table.id || '';
|
||
var badge = document.getElementById('editBqSourceTypeBadge');
|
||
if (badge) badge.textContent = (table.source_type || 'bigquery');
|
||
|
||
var qmode = (table.query_mode || 'remote');
|
||
var sq = (table.source_query || '');
|
||
// Issue #266: register-time the whole-table branch omits
|
||
// bucket/source_table from the JSON payload (only source_query is
|
||
// sent), so existing whole-table materialized rows persist with
|
||
// bucket=NULL in the registry. When the Edit modal then reads
|
||
// table.bucket on those rows it gets null → empty input → admin
|
||
// saves a broken `SELECT * FROM bq."".""` SQL. Parse the source_query
|
||
// to recover the dataset+table for the pre-fill. Same regex shape
|
||
// we already use to detect whole-table mode below.
|
||
var SELECT_STAR_RE = /^\s*SELECT\s+\*\s+FROM\s+bq\."([^"]+)"\."([^"]+)"\s*$/i;
|
||
var isAutoSelectStar = SELECT_STAR_RE.test(sq);
|
||
if (qmode === 'materialized') {
|
||
_setEditBqRadio('editBqAccessMode', 'synced');
|
||
_setEditBqRadio('editBqSyncMode', isAutoSelectStar ? 'whole' : 'custom');
|
||
} else {
|
||
_setEditBqRadio('editBqAccessMode', 'live');
|
||
_setEditBqRadio('editBqSyncMode', 'whole');
|
||
}
|
||
|
||
// Pre-fill dataset/table inputs. For whole-table materialized rows
|
||
// without persisted bucket/source_table (pre-#266 register state),
|
||
// recover them from the SQL.
|
||
var preDataset = table.bucket || '';
|
||
var preSourceTable = table.source_table || '';
|
||
if (!preDataset && !preSourceTable && isAutoSelectStar) {
|
||
var m = sq.match(SELECT_STAR_RE);
|
||
if (m) { preDataset = m[1]; preSourceTable = m[2]; }
|
||
}
|
||
document.getElementById('editBqDataset').value = preDataset;
|
||
document.getElementById('editBqSourceTable').value = preSourceTable;
|
||
document.getElementById('editBqSourceQuery').value = sq;
|
||
document.getElementById('editBqSyncSchedule').value = table.sync_schedule || '';
|
||
document.getElementById('editBqDescription').value = table.description || '';
|
||
document.getElementById('editBqFolder').value = table.folder || '';
|
||
document.getElementById('editBqModeWarning').style.display = 'none';
|
||
_editOriginalQueryMode = qmode;
|
||
onEditBqAccessModeChange(); // fire to set field visibility
|
||
|
||
var btn = document.getElementById('editBqSubmitBtn');
|
||
btn.disabled = false;
|
||
btn.textContent = 'Save Changes';
|
||
document.getElementById('editBqModal').classList.add('active');
|
||
}
|
||
|
||
function closeEditBqModal() {
|
||
document.getElementById('editBqModal').classList.remove('active');
|
||
currentEditTableId = null;
|
||
_editOriginalQueryMode = null;
|
||
}
|
||
|
||
function _openEditLegacyModal(table) {
|
||
// Legacy fallback for source_types that aren't bigquery or keboola
|
||
// (e.g. jira). Carries the Strategy + Primary Key + Description +
|
||
// Folder fields only.
|
||
currentEditTableId = table.id;
|
||
document.getElementById('editTableId').value = table.id || '';
|
||
var sourceType = (table.source_type || '').toLowerCase();
|
||
document.getElementById('editSourceTypeBadge').textContent = sourceType || '—';
|
||
document.getElementById('editDescription').value = table.description || '';
|
||
document.getElementById('editFolder').value = table.folder || '';
|
||
document.getElementById('editStrategy').value = table.sync_strategy || 'full_refresh';
|
||
document.getElementById('editPrimaryKey').value = (table.primary_key || []).join(', ');
|
||
_editOriginalQueryMode = null;
|
||
document.getElementById('editSubmitBtn').disabled = false;
|
||
document.getElementById('editModal').classList.add('active');
|
||
}
|
||
|
||
function onEditBqAccessModeChange() {
|
||
var accessMode = _getEditBqAccessMode();
|
||
document.querySelectorAll('.bq-edit-access-synced').forEach(function(el) {
|
||
el.style.display = (accessMode === 'synced') ? '' : 'none';
|
||
});
|
||
|
||
if (accessMode === 'synced') {
|
||
onEditBqSyncModeChange();
|
||
} else {
|
||
// Live: dataset+table inputs visible, SQL textarea hidden.
|
||
document.querySelectorAll('.bq-edit-source-table').forEach(function(el) {
|
||
el.style.display = '';
|
||
});
|
||
document.querySelectorAll('.bq-edit-source-custom').forEach(function(el) {
|
||
el.style.display = 'none';
|
||
});
|
||
}
|
||
|
||
// Mode-switch warning: only fire when the operator actually flipped
|
||
// access mode from what was loaded — typo-fix edits stay quiet.
|
||
var newMode = (accessMode === 'synced') ? 'materialized' : 'remote';
|
||
var warn = document.getElementById('editBqModeWarning');
|
||
if (_editOriginalQueryMode && newMode !== _editOriginalQueryMode) {
|
||
var msg;
|
||
if (_editOriginalQueryMode === 'materialized' && newMode === 'remote') {
|
||
msg = '⚠ Switching Synced locally → Live from BigQuery will drop the materialized parquet on the next sync. Analysts who already pulled this table will start getting live BQ queries instead of a local copy; the sync schedule becomes ignored.';
|
||
} else {
|
||
msg = '⚠ Switching Live from BigQuery → Synced locally: the next scheduled sync runs a SELECT and writes a parquet. Analysts will start reading the local copy on their next `agnes pull`. Remember to set a sync schedule.';
|
||
}
|
||
warn.textContent = msg;
|
||
warn.style.display = '';
|
||
} else {
|
||
warn.style.display = 'none';
|
||
}
|
||
}
|
||
|
||
function onEditBqSyncModeChange() {
|
||
var syncMode = _getEditBqSyncMode();
|
||
document.querySelectorAll('.bq-edit-source-table').forEach(function(el) {
|
||
el.style.display = (syncMode === 'whole') ? '' : 'none';
|
||
});
|
||
document.querySelectorAll('.bq-edit-source-custom').forEach(function(el) {
|
||
el.style.display = (syncMode === 'custom') ? '' : 'none';
|
||
});
|
||
}
|
||
|
||
function closeEditModal() {
|
||
document.getElementById('editModal').classList.remove('active');
|
||
currentEditTableId = null;
|
||
_editOriginalQueryMode = null;
|
||
}
|
||
|
||
function saveTableEdit() {
|
||
// Legacy save handler — runs only for source_types that aren't
|
||
// bigquery (handled by saveBqTabEdit) or keboola (handled by
|
||
// saveKeboolaTabEdit). Currently the catch-all for jira-style rows.
|
||
if (!currentEditTableId) return;
|
||
|
||
var btn = document.getElementById('editSubmitBtn');
|
||
btn.disabled = true;
|
||
btn.textContent = 'Saving...';
|
||
|
||
var pk = document.getElementById('editPrimaryKey').value.trim();
|
||
var primaryKey = pk
|
||
? pk.split(',').map(function(s) { return s.trim(); }).filter(Boolean)
|
||
: [];
|
||
var payload = {
|
||
sync_strategy: document.getElementById('editStrategy').value,
|
||
primary_key: primaryKey,
|
||
description:
|
||
document.getElementById('editDescription').value.trim() || null,
|
||
folder: document.getElementById('editFolder').value.trim() || null,
|
||
};
|
||
|
||
fetch('/api/admin/registry/' + encodeURIComponent(currentEditTableId), {
|
||
method: 'PUT',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify(payload)
|
||
})
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) { throw new Error(d.detail || d.error || 'Update failed'); });
|
||
return r.json();
|
||
})
|
||
.then(function(data) {
|
||
closeEditModal();
|
||
showToast('Table updated successfully', 'success');
|
||
loadRegistry();
|
||
})
|
||
.catch(function(err) {
|
||
showToast('Update failed: ' + err.message, 'error');
|
||
})
|
||
.finally(function() {
|
||
btn.disabled = false;
|
||
btn.textContent = 'Save Changes';
|
||
});
|
||
}
|
||
|
||
function saveBqTabEdit() {
|
||
// Save handler for the relocated BQ Edit modal (#editBqModal).
|
||
// Mirrors the pre-C2 BQ branch of saveTableEdit() but reads from
|
||
// the BQ-modal-scoped fields (editBqDescription / editBqFolder).
|
||
if (!currentEditTableId) return;
|
||
|
||
var btn = document.getElementById('editBqSubmitBtn');
|
||
btn.disabled = true;
|
||
btn.textContent = 'Saving...';
|
||
|
||
// Two-question state machine — same as Register modal:
|
||
// live → query_mode='remote'
|
||
// synced/whole → materialized + auto SELECT *
|
||
// synced/custom → materialized + admin SQL
|
||
var accessMode = _getEditBqAccessMode();
|
||
var syncMode = _getEditBqSyncMode();
|
||
var dataset = document.getElementById('editBqDataset').value.trim();
|
||
var sourceTable = document.getElementById('editBqSourceTable').value.trim();
|
||
|
||
var payload = {
|
||
description: document.getElementById('editBqDescription').value.trim() || null,
|
||
folder: document.getElementById('editBqFolder').value.trim() || null,
|
||
source_type: 'bigquery',
|
||
sync_schedule:
|
||
document.getElementById('editBqSyncSchedule').value.trim() || null,
|
||
};
|
||
|
||
if (accessMode === 'synced' && syncMode === 'custom') {
|
||
payload.query_mode = 'materialized';
|
||
payload.source_query =
|
||
document.getElementById('editBqSourceQuery').value.trim();
|
||
// Issue #266: only clear bucket/source_table on a genuine mode
|
||
// flip (was non-materialized → now custom-SQL materialized). The
|
||
// original pre-#266 code nulled unconditionally on every save —
|
||
// so an admin editing only the description on an already-
|
||
// materialized custom-SQL row would silently wipe bucket/
|
||
// source_table. Destructive for rows that had them persisted
|
||
// (whole-table-then-switched-to-custom, or curl-set via API).
|
||
// On no-op saves (mode didn't change since modal load), omit
|
||
// the keys entirely so the PUT handler's `exclude_unset=True`
|
||
// preserves the existing values in the registry.
|
||
if (_editOriginalQueryMode !== 'materialized') {
|
||
payload.bucket = null;
|
||
payload.source_table = null;
|
||
}
|
||
} else if (accessMode === 'synced' && syncMode === 'whole') {
|
||
payload.query_mode = 'materialized';
|
||
payload.source_query =
|
||
'SELECT * FROM bq."' + dataset + '"."' + sourceTable + '"';
|
||
payload.bucket = dataset || null;
|
||
payload.source_table = sourceTable || null;
|
||
} else {
|
||
// Live.
|
||
payload.query_mode = 'remote';
|
||
payload.bucket = dataset || null;
|
||
payload.source_table = sourceTable || null;
|
||
payload.source_query = null;
|
||
}
|
||
|
||
fetch('/api/admin/registry/' + encodeURIComponent(currentEditTableId), {
|
||
method: 'PUT',
|
||
headers: { 'Content-Type': 'application/json' },
|
||
body: JSON.stringify(payload)
|
||
})
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) { throw new Error(d.detail || d.error || 'Update failed'); });
|
||
return r.json();
|
||
})
|
||
.then(function() {
|
||
closeEditBqModal();
|
||
showToast('Table updated successfully', 'success');
|
||
loadRegistry();
|
||
})
|
||
.catch(function(err) {
|
||
showToast('Update failed: ' + err.message, 'error');
|
||
})
|
||
.finally(function() {
|
||
btn.disabled = false;
|
||
btn.textContent = 'Save Changes';
|
||
});
|
||
}
|
||
|
||
// ── Registry ────────────────────────────────────────────────
|
||
|
||
function loadRegistry() {
|
||
fetch('/api/admin/registry')
|
||
.then(function(r) {
|
||
if (!r.ok) return r.json().then(function(d) { throw new Error(d.error || 'Load failed'); });
|
||
return r.json();
|
||
})
|
||
.then(function(data) {
|
||
registryData = data;
|
||
registryVersion = data.version;
|
||
var tables = data.tables || [];
|
||
var subtitle = document.getElementById('registrySubtitle');
|
||
if (subtitle) {
|
||
subtitle.textContent = tables.length + ' table' + (tables.length !== 1 ? 's' : '') + ' configured for sync';
|
||
}
|
||
renderRegistryListing(
|
||
document.getElementById('bqTableListing'),
|
||
tables.filter(function(t) { return t.source_type === 'bigquery'; })
|
||
);
|
||
renderRegistryListing(
|
||
document.getElementById('kbTableListing'),
|
||
tables.filter(function(t) { return t.source_type === 'keboola'; })
|
||
);
|
||
renderRegistryListing(
|
||
document.getElementById('jiraTableListing'),
|
||
tables.filter(function(t) { return t.source_type === 'jira'; })
|
||
);
|
||
renderRegistryListing(
|
||
document.getElementById('internalTableListing'),
|
||
tables.filter(function(t) { return t.source_type === 'internal'; })
|
||
);
|
||
})
|
||
.catch(function(err) {
|
||
var msg = '<div class="panel-body-empty" style="color: var(--error);">Failed to load registry: ' + escapeHtml(err.message) + '</div>';
|
||
[
|
||
document.getElementById('bqTableListing'),
|
||
document.getElementById('kbTableListing'),
|
||
document.getElementById('jiraTableListing'),
|
||
document.getElementById('internalTableListing'),
|
||
].forEach(function(target) { if (target) target.innerHTML = msg; });
|
||
});
|
||
}
|
||
|
||
function renderModeBadge(mode) {
|
||
var m = (mode || 'local').toLowerCase();
|
||
// `internal` is the v?? addition (agnes_* tables backed by
|
||
// system.duckdb). Allowlist it so the mode-badge picks up the
|
||
// dedicated CSS class instead of falling back to mode-local
|
||
// styling and lying about how the data is served.
|
||
var cls = 'mode-badge mode-' + (['local','remote','materialized','internal'].indexOf(m) >= 0 ? m : 'local');
|
||
return '<span class="' + cls + '">' + escapeHtml(m) + '</span>';
|
||
}
|
||
|
||
function renderRegistryListing(target, tables) {
|
||
if (!target) return;
|
||
if (tables.length === 0) {
|
||
target.innerHTML = '<div class="panel-body-empty">No tables registered yet.</div>';
|
||
return;
|
||
}
|
||
var html = '<table class="registry-table">';
|
||
html += '<thead><tr>';
|
||
html += '<th class="col-id">Table ID</th>';
|
||
html += '<th class="col-mode">Mode</th>';
|
||
html += '<th class="col-source">Source</th>';
|
||
html += '<th class="col-pk">Primary Key</th>';
|
||
html += '<th class="col-schedule">Schedule</th>';
|
||
html += '<th class="col-folder">Folder</th>';
|
||
html += '<th>Description</th>';
|
||
html += '<th class="col-registered">Registered</th>';
|
||
html += '<th class="col-status"></th>';
|
||
html += '<th class="col-actions">Actions</th>';
|
||
html += '</tr></thead><tbody>';
|
||
tables.forEach(function(table) {
|
||
html += '<tr>';
|
||
html += '<td class="col-id" title="' + escapeHtml(table.id) + '">' + escapeHtml(table.id) + '</td>';
|
||
html += '<td class="col-mode">' + renderModeBadge(table.query_mode) + '</td>';
|
||
|
||
// Source: bucket / source_table; em-dash when both empty (custom-SQL row).
|
||
var bucket = table.bucket || '';
|
||
var srcTable = table.source_table || '';
|
||
var sourceText = '';
|
||
if (bucket && srcTable) {
|
||
sourceText = bucket + ' / ' + srcTable;
|
||
} else if (bucket || srcTable) {
|
||
sourceText = bucket || srcTable;
|
||
}
|
||
var sourceCell = sourceText ? escapeHtml(sourceText) : '—';
|
||
html += '<td class="col-source" title="' + escapeHtml(sourceText) + '">' + sourceCell + '</td>';
|
||
|
||
html += '<td class="col-pk">' + escapeHtml((table.primary_key || []).join(', ') || '—') + '</td>';
|
||
html += '<td class="col-schedule">' + escapeHtml(table.sync_schedule || '—') + '</td>';
|
||
|
||
// Folder badge — '—' when null/empty.
|
||
if (table.folder) {
|
||
html += '<td class="col-folder"><span class="folder-badge">' + escapeHtml(table.folder) + '</span></td>';
|
||
} else {
|
||
html += '<td class="col-folder">—</td>';
|
||
}
|
||
|
||
var desc = unescapeShellQuoting(table.description || '');
|
||
html += '<td class="col-description" title="' + escapeHtml(desc) + '">' + escapeHtml(desc || '—') + '</td>';
|
||
|
||
// Registered: stacked email + date (first 10 chars of ISO timestamp).
|
||
var regBy = table.registered_by || '';
|
||
var regByDisplay = regBy;
|
||
if (regBy.length > 24 && regBy.indexOf('@') > 0) {
|
||
regByDisplay = regBy.split('@')[0];
|
||
}
|
||
var regAt = table.registered_at ? String(table.registered_at).slice(0, 10) : '';
|
||
html += '<td class="col-registered">';
|
||
html += '<div class="registered-by" title="' + escapeHtml(regBy) + '">' + (regByDisplay ? escapeHtml(regByDisplay) : '—') + '</div>';
|
||
html += '<div class="registered-at">' + escapeHtml(regAt || '') + '</div>';
|
||
html += '</td>';
|
||
|
||
// Status: warning icon when the last sync errored.
|
||
html += '<td class="col-status">';
|
||
if (table.last_sync_error) {
|
||
html += '<span title="' + escapeHtml(table.last_sync_error) + '">';
|
||
html += '<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" style="color: var(--error);"><path d="M10.29 3.86 1.82 18a2 2 0 0 0 1.71 3h16.94a2 2 0 0 0 1.71-3L13.71 3.86a2 2 0 0 0-3.42 0z"/><line x1="12" y1="9" x2="12" y2="13"/><line x1="12" y1="17" x2="12.01" y2="17"/></svg>';
|
||
html += '</span>';
|
||
}
|
||
html += '</td>';
|
||
|
||
html += '<td class="col-actions"><div style="display:flex; gap:4px; justify-content:flex-end;">';
|
||
// Internal source tables (agnes_*) are seeded on every app
|
||
// boot from connectors/internal/registry.py — Edit/Delete
|
||
// would either no-op or be reverted on next start. Hide both
|
||
// actions; Manage access stays because RBAC is still
|
||
// per-table (internal tables auto-grant; the link still
|
||
// takes the admin to the access page if they want to inspect).
|
||
//
|
||
// Issue #265: HTML attribute values do not recognize backslash
|
||
// escapes the way JavaScript string literals do — use HTML-
|
||
// entity escape (`'` for `'`, plus `"`, `<`, `>`, `&`)
|
||
// via escapeHtmlAttr, the right encoding for an HTML attribute.
|
||
var isInternal = (table.source_type === 'internal');
|
||
if (!isInternal) {
|
||
html += '<button class="btn-icon" title="Edit" onclick=\'openEditModal(' + escapeHtmlAttr(JSON.stringify(table)) + ')\'>';
|
||
html += '<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M11 4H4a2 2 0 0 0-2 2v14a2 2 0 0 0 2 2h14a2 2 0 0 0 2-2v-7"/><path d="M18.5 2.5a2.121 2.121 0 0 1 3 3L12 15l-4 1 1-4 9.5-9.5z"/></svg>';
|
||
html += '</button>';
|
||
}
|
||
// Manage access: deep-link to /admin/access pre-filtered to this table.
|
||
html += '<button class="btn-icon" title="Manage access" onclick="manageAccess('' + escapeHtmlAttr(table.id) + '')">';
|
||
html += '<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect x="3" y="11" width="18" height="11" rx="2" ry="2"/><path d="M7 11V7a5 5 0 0 1 10 0v4"/></svg>';
|
||
html += '</button>';
|
||
if (!isInternal) {
|
||
html += '<button class="btn-icon danger" title="Delete" onclick="deleteTable('' + escapeHtmlAttr(table.id) + '')">';
|
||
html += '<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><polyline points="3 6 5 6 21 6"/><path d="M19 6v14a2 2 0 0 1-2 2H7a2 2 0 0 1-2-2V6m3 0V4a2 2 0 0 1 2-2h4a2 2 0 0 1 2 2v2"/></svg>';
|
||
html += '</button>';
|
||
}
|
||
html += '</div></td></tr>';
|
||
});
|
||
html += '</tbody></table>';
|
||
target.innerHTML = html;
|
||
}
|
||
|
||
// Deep-link from a table row to /admin/access scoped to this table_id.
|
||
// The /admin/access page reads window.location.hash on bootstrap and,
|
||
// when it matches `table:<id>`, pre-fills the resource filter so the
|
||
// operator lands on the just-clicked table once they pick a group.
|
||
function manageAccess(tableId) {
|
||
window.location.href = '/admin/access#table:' + encodeURIComponent(tableId);
|
||
}
|
||
|
||
function deleteTable(tableId) {
|
||
if (!confirm('Are you sure you want to unregister "' + tableId + '"?\n\nThis will remove it from sync and clean up user subscriptions.')) {
|
||
return;
|
||
}
|
||
|
||
// DELETE doesn't need a body — the API has no `version` field.
|
||
fetch('/api/admin/registry/' + encodeURIComponent(tableId), {
|
||
method: 'DELETE',
|
||
})
|
||
.then(function(r) {
|
||
// 204 No Content has an empty body, so don't .json() it.
|
||
if (r.status === 204) return null;
|
||
if (!r.ok) return r.json().then(function(d) { throw new Error(d.detail || d.error || 'Delete failed'); });
|
||
return r.json();
|
||
})
|
||
.then(function() {
|
||
showToast('Table unregistered successfully', 'success');
|
||
loadRegistry();
|
||
})
|
||
.catch(function(err) {
|
||
showToast('Delete failed: ' + err.message, 'error');
|
||
});
|
||
}
|
||
|
||
// ── Modal close on overlay click ────────────────────────────
|
||
|
||
var _registerBqModalEl = document.getElementById('registerBqModal');
|
||
if (_registerBqModalEl) {
|
||
_registerBqModalEl.addEventListener('click', function(e) {
|
||
if (e.target === this) closeRegisterBqModal();
|
||
});
|
||
}
|
||
|
||
var _registerKeboolaModalEl = document.getElementById('registerKeboolaModal');
|
||
if (_registerKeboolaModalEl) {
|
||
_registerKeboolaModalEl.addEventListener('click', function(e) {
|
||
if (e.target === this) closeRegisterKeboolaModal();
|
||
});
|
||
}
|
||
|
||
var _editKeboolaModalEl = document.getElementById('editKeboolaModal');
|
||
if (_editKeboolaModalEl) {
|
||
_editKeboolaModalEl.addEventListener('click', function(e) {
|
||
if (e.target === this) closeEditKeboolaModal();
|
||
});
|
||
}
|
||
|
||
var _editBqModalEl = document.getElementById('editBqModal');
|
||
if (_editBqModalEl) {
|
||
_editBqModalEl.addEventListener('click', function(e) {
|
||
if (e.target === this) closeEditBqModal();
|
||
});
|
||
}
|
||
|
||
document.getElementById('editModal').addEventListener('click', function(e) {
|
||
if (e.target === this) closeEditModal();
|
||
});
|
||
|
||
// Close modals on Escape key
|
||
document.addEventListener('keydown', function(e) {
|
||
if (e.key === 'Escape') {
|
||
closeRegisterBqModal();
|
||
if (typeof closeRegisterKeboolaModal === 'function') closeRegisterKeboolaModal();
|
||
if (typeof closeEditKeboolaModal === 'function') closeEditKeboolaModal();
|
||
if (typeof closeEditBqModal === 'function') closeEditBqModal();
|
||
closeEditModal();
|
||
}
|
||
});
|
||
|
||
// ── Initialize ──────────────────────────────────────────────
|
||
|
||
loadRegistry();
|
||
|
||
// ── Cache warmup toolbar (issue #155 / #156) ────────────────
|
||
let cacheWarmupSource = null;
|
||
|
||
function _cacheWarmupClearPollFallback() {
|
||
if (window._cacheWarmupPollInterval) {
|
||
clearInterval(window._cacheWarmupPollInterval);
|
||
window._cacheWarmupPollInterval = null;
|
||
}
|
||
}
|
||
|
||
function cacheWarmupInit() {
|
||
cacheWarmupRefreshSnapshot();
|
||
cacheWarmupOpenStream();
|
||
}
|
||
|
||
function cacheWarmupRefreshSnapshot() {
|
||
fetch('/api/admin/cache-warmup/status')
|
||
.then(function(r) { return r.json(); })
|
||
.then(function(state) { cacheWarmupRender(state); })
|
||
.catch(function() { /* silent */ });
|
||
}
|
||
|
||
function cacheWarmupOpenStream() {
|
||
try {
|
||
cacheWarmupSource = new EventSource('/api/admin/cache-warmup/stream');
|
||
cacheWarmupSource.addEventListener('start', cacheWarmupOnStart);
|
||
cacheWarmupSource.addEventListener('row', cacheWarmupOnRow);
|
||
cacheWarmupSource.addEventListener('complete', cacheWarmupOnComplete);
|
||
cacheWarmupSource.addEventListener('snapshot', function(e) {
|
||
_cacheWarmupClearPollFallback();
|
||
cacheWarmupRender(JSON.parse(e.data));
|
||
});
|
||
cacheWarmupSource.onerror = function() {
|
||
if (cacheWarmupSource) {
|
||
cacheWarmupSource.close();
|
||
cacheWarmupSource = null;
|
||
}
|
||
// Continuous polling fallback. Try to re-open SSE every 30 s in
|
||
// case the proxy / network heals. Only one polling interval at a
|
||
// time (prevent stacking on repeated errors).
|
||
if (!window._cacheWarmupPollInterval) {
|
||
window._cacheWarmupPollInterval = setInterval(
|
||
cacheWarmupRefreshSnapshot, 3000
|
||
);
|
||
setTimeout(function tryReconnect() {
|
||
if (cacheWarmupSource) return; // already reconnected
|
||
try {
|
||
clearInterval(window._cacheWarmupPollInterval);
|
||
window._cacheWarmupPollInterval = null;
|
||
cacheWarmupOpenStream(); // recursive — onerror retries again
|
||
} catch (e) {
|
||
window._cacheWarmupPollInterval = setInterval(
|
||
cacheWarmupRefreshSnapshot, 3000
|
||
);
|
||
setTimeout(tryReconnect, 30000);
|
||
}
|
||
}, 30000);
|
||
}
|
||
};
|
||
} catch (e) {
|
||
setInterval(cacheWarmupRefreshSnapshot, 3000);
|
||
}
|
||
}
|
||
|
||
function cacheWarmupRender(state) {
|
||
var summary = document.getElementById('cacheWarmupSummary');
|
||
var bar = document.getElementById('cacheWarmupBar');
|
||
var btn = document.getElementById('cacheWarmupRunBtn');
|
||
if (!summary) return;
|
||
|
||
if (!state || state.state === 'never_run') {
|
||
summary.textContent = 'No cache warmup yet — click Re-warm all to start.';
|
||
bar.style.display = 'none';
|
||
btn.disabled = false;
|
||
return;
|
||
}
|
||
var inProgress = state.completed_at === null || state.completed_at === undefined;
|
||
var pct = state.total > 0 ? Math.round((state.completed * 100) / state.total) : 0;
|
||
summary.textContent = inProgress
|
||
? state.completed + ' / ' + state.total + ' fresh — running…'
|
||
: 'Last run: ' + state.completed + ' ok, ' + state.failed + ' errors';
|
||
bar.style.display = 'block';
|
||
bar.value = pct;
|
||
btn.disabled = inProgress;
|
||
|
||
if (state.rows) {
|
||
for (var tid in state.rows) {
|
||
cacheWarmupSetRowBadge(tid, state.rows[tid]);
|
||
}
|
||
}
|
||
}
|
||
|
||
function cacheWarmupOnStart(e) {
|
||
_cacheWarmupClearPollFallback();
|
||
var data = JSON.parse(e.data);
|
||
var log = document.getElementById('cacheWarmupLog');
|
||
log.textContent = '';
|
||
// Hide the inline hint paragraph as soon as a real log line lands
|
||
// — the empty <pre>'s placeholder context is no longer useful and
|
||
// the dark log block is what the operator wants to watch.
|
||
var hint = document.getElementById('cacheWarmupHint');
|
||
if (hint) hint.style.display = 'none';
|
||
// Auto-open the details so streamed events are visible immediately,
|
||
// even if the operator clicked Re-warm without expanding first.
|
||
var details = document.getElementById('cacheWarmupDetails');
|
||
if (details) details.open = true;
|
||
cacheWarmupAppendLog(
|
||
'[' + nowHHMMSS() + '] start trigger=' + data.trigger + ' total=' + data.total
|
||
);
|
||
cacheWarmupRefreshSnapshot();
|
||
}
|
||
|
||
function cacheWarmupOnRow(e) {
|
||
_cacheWarmupClearPollFallback();
|
||
var rs = JSON.parse(e.data);
|
||
cacheWarmupAppendLog(
|
||
'[' + nowHHMMSS() + '] ' + rs.status.padEnd(7) + ' ' + rs.table_id +
|
||
(rs.duration_ms ? ' (' + (rs.duration_ms / 1000).toFixed(1) + ' s)' : '') +
|
||
(rs.error ? ' ' + rs.error : '')
|
||
);
|
||
cacheWarmupSetRowBadge(rs.table_id, rs);
|
||
cacheWarmupRefreshSnapshot();
|
||
}
|
||
|
||
function cacheWarmupOnComplete(e) {
|
||
_cacheWarmupClearPollFallback();
|
||
var data = JSON.parse(e.data);
|
||
cacheWarmupAppendLog(
|
||
'[' + nowHHMMSS() + '] complete total=' + data.total +
|
||
' ok=' + data.completed + ' fail=' + data.failed
|
||
);
|
||
cacheWarmupRefreshSnapshot();
|
||
}
|
||
|
||
function cacheWarmupAppendLog(line) {
|
||
var log = document.getElementById('cacheWarmupLog');
|
||
if (!log) return;
|
||
log.textContent += line + '\n';
|
||
log.scrollTop = log.scrollHeight;
|
||
}
|
||
|
||
function cacheWarmupSetRowBadge(tableId, rs) {
|
||
document.querySelectorAll('tr').forEach(function(tr) {
|
||
var idCell = tr.querySelector('td.col-id');
|
||
if (!idCell || idCell.textContent.trim() !== tableId) return;
|
||
var statusCell = tr.querySelector('td.col-status');
|
||
if (!statusCell) return;
|
||
var color = {fresh: '#10B77F', warming: '#0073D1', pending: '#9CA3AF', error: '#EA580C'}[rs.status] || '#9CA3AF';
|
||
var label = rs.status === 'fresh' ? 'fresh' : rs.status;
|
||
// Build via DOM API so rs.error escapes safely into the title
|
||
// attribute (XSS guard — rs.error is server-derived, may contain
|
||
// quotes / angle brackets).
|
||
var span = document.createElement('span');
|
||
span.style.cssText =
|
||
'display:inline-block;padding:2px 6px;border-radius:3px;' +
|
||
'font-size:11px;background:' + color + ';color:white;';
|
||
if (rs.error) span.setAttribute('title', rs.error);
|
||
span.textContent = label;
|
||
statusCell.replaceChildren(span);
|
||
});
|
||
}
|
||
|
||
function nowHHMMSS() {
|
||
var d = new Date();
|
||
return d.toTimeString().slice(0, 8);
|
||
}
|
||
|
||
function cacheWarmupRun() {
|
||
var btn = document.getElementById('cacheWarmupRunBtn');
|
||
btn.disabled = true;
|
||
// Auto-open the log section so the operator sees streaming output
|
||
// instead of staring at a disabled button while ~24s pass per
|
||
// remote BQ row. The hint paragraph hides as soon as a real log
|
||
// line lands (see cacheWarmupOnStart).
|
||
var details = document.getElementById('cacheWarmupDetails');
|
||
if (details) details.open = true;
|
||
fetch('/api/admin/cache-warmup/run', {method: 'POST'})
|
||
.then(function(r) { return r.json(); })
|
||
.then(function() { /* SSE stream picks up the new run */ })
|
||
.catch(function() { btn.disabled = false; });
|
||
}
|
||
|
||
document.addEventListener('DOMContentLoaded', cacheWarmupInit);
|
||
|
||
</script>
|
||
{% include "_version_badge.html" %}
|
||
</body>
|
||
</html>
|