agnes-the-ai-analyst/app/web/templates/admin_server_config.html
minasarustamyan 69a1e22cf5
feat(initial-workspace): per-instance agnes init override (#292)
* feat(initial-workspace): per-instance agnes init override

Adds Initial Workspace Template — an admin-configurable per-instance
override for the agnes init analyst workspace. When configured, agnes
init downloads a server-rendered zip from a Git repo the admin registered
and extracts it into the analyst's workspace, fully bypassing Agnes-default
CLAUDE.md / settings.json / hooks / slash commands / AGNES_WORKSPACE.md.

Repo layout convention: only the contents of a top-level `workspace/`
subdirectory ship to analysts; admin docs (README, CI configs) at the
repo root stay in the repo and never reach an analyst. Sync rejects
repos without `workspace/` at root.

Server side:
- src/initial_workspace.py — clone (or fetch+reset), validate, build zip
  with strict path checks and reserved-path rejection
  (workspace/.claude/init-complete reserved by Agnes)
- app/api/initial_workspace.py — admin CRUD + sync endpoint + analyst-
  facing status/zip/applied endpoints; config persists to instance.yaml
  overlay, PAT to .env_overlay
- app/secrets.py — refactor: persist_overlay_token shared helper with
  threading.Lock for .env_overlay writes (closes pre-existing race
  between concurrent marketplaces saves)
- app/web/templates/admin_server_config.html — new "Initial Workspace
  Template" section + modal + Sync/Edit/Delete/Download buttons (matches
  existing cfg-section visual language)

CLI side:
- cli/lib/override.py — single source of truth for is_override_workspace
  sentinel detection
- cli/lib/initial_workspace.py — probe status, safe zip extraction with
  ../absolute/symlink rejection, typed-YES force confirmation
- cli/commands/init.py — override branch (skips Agnes-default workspace
  writes); extended sentinel with override:true, template_source,
  template_sha so future agnes self-upgrade does not auto-refresh hooks
- cli/lib/hooks.py + cli/lib/commands.py — short-circuit on override
  workspaces (install_claude_hooks, install_claude_commands,
  maybe_refresh_claude_hooks)

Audit-event strategy: server writes initial_workspace.fetch_started
inside GET /api/initial-workspace.zip (cannot be spoofed by PAT-holder);
CLI POST /applied writes initial_workspace.applied as best-effort
confirmation. Admin mutations log via the existing _audit pattern.

Tests: 27 server (clone/validate/zip + workspace-subdir convention +
concurrent persist_overlay_token + endpoint shapes + audit rows) + 29
CLI (override sentinel parse + probe fall-through + safe extraction +
YES strictness + hook guards + e2e mocked init).

Risk acceptance — documented in docs/initial-workspace-override.md +
CHANGELOG Internal section so AI reviewers understand the deviations
from defaults are intentional:
- maybe_refresh_claude_hooks deliberately no-ops on override workspaces
- --force on override does NOT back up CLAUDE.md (admin's repo is the
  source of truth)
- .claude/CLAUDE.local.md IS overwritten by override extraction when
  admin's repo ships one

* test+vendor-agnostic: drop Groupon tokens from #292 fixtures + extend admin-gate coverage

Two fixes from the takeover review on #292:

1. **Vendor-agnostic OSS rule**: Replace `Groupon` / `groupon/template`
   tokens in test fixtures with `Acme` / `acme/template` (8 sites in
   test_cli_init_override.py + 1 in test_initial_workspace_api.py).
   Per CLAUDE.md "Vendor-agnostic OSS — no customer-specific content"
   rule: customer-specific tokens don't belong in shipped artifacts,
   even in test fixtures. The pre-existing FoundryAI mentions in
   test_instance_config.py + test_setup_instructions.py are out of
   scope for this PR (didn't introduce them).

2. **Admin-gate coverage gap**: `test_admin_endpoints_require_admin`
   only covered GET /api/admin/initial-workspace + POST .../sync. The
   register-write (POST .../initial-workspace) and delete (DELETE
   .../initial-workspace) endpoints used the same `Depends(require_admin)`
   wiring but had no regression test. Loop now covers all 4 verbs so
   a future refactor that drops the dependency from one endpoint
   fails here instead of silently exposing the write/delete paths to
   any analyst with a PAT.

* release: 0.54.9 — Initial Workspace Template (per-instance agnes init override)

Last commit on the PR per CLAUDE.md hard rule. Patch bump (0.54.8 →
0.54.9) for Mina's Initial Workspace Template feature.

No DB migration (config lives in instance.yaml overlay). No
mandatory operator action — empty default keeps OSS-default
agnes init behavior. Operators wanting full template control link a
Git repo on /admin/server-config → "Initial Workspace Template".
See docs/initial-workspace-override.md for the full
responsibility-transfer contract.

---------

Co-authored-by: Minas Arustamyan <arustamyan.minas@gmail.com>
Co-authored-by: ZdenekSrotyr <zdenek.srotyr@keboola.com>
2026-05-13 20:35:01 +00:00

1458 lines
67 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

{% extends "base.html" %}
{% block title %}Server config — {{ config.INSTANCE_NAME }}{% endblock %}
{% block content %}
{# Server configuration editor — instance.yaml fields grouped by section.
Page-shell only: GET /api/admin/server-config feeds the form (with
secrets redacted), POST /api/admin/server-config saves a section. The
"danger-zone" sections (auth, server) get a confirmation dialog before
the request is sent. Saves trigger the restart banner — hot-reload is
out of scope for #91. #}
<style>
.container:has(.cfg-page) { max-width: none; padding: 24px 16px; }
.cfg-page { max-width: 1100px; margin: 0 auto; padding: 0; }
.cfg-toolbar {
display: flex; justify-content: space-between; align-items: center;
gap: 16px; margin-bottom: 16px; flex-wrap: wrap;
}
.cfg-title { margin: 0; font-size: 22px; font-weight: 600; }
.cfg-subtitle { color: var(--text-secondary, #6b7280); font-size: 13px; margin-top: 4px; }
.cfg-banner {
padding: 12px 16px; border-radius: 8px;
background: #fffbeb; border: 1px solid #fcd34d; color: #92400e;
font-size: 13px; margin-bottom: 16px; display: none;
}
.cfg-banner.is-visible { display: block; }
.cfg-banner.success { background: #ecfdf5; border-color: #34d399; color: #065f46; }
.cfg-banner.error { background: #fef2f2; border-color: #fca5a5; color: #991b1b; }
.cfg-section {
background: var(--surface, #fff);
border: 1px solid var(--border, #e5e7eb);
border-radius: 12px;
margin-bottom: 16px;
overflow: hidden;
}
.cfg-section.is-danger { border-color: #fca5a5; }
.cfg-section header {
padding: 14px 18px;
background: var(--border-light, #f9fafb);
border-bottom: 1px solid var(--border, #e5e7eb);
display: flex; align-items: center; justify-content: space-between;
gap: 12px;
}
.cfg-section.is-danger header { background: #fef2f2; }
.cfg-section h3 {
margin: 0; font-size: 15px; font-weight: 600;
}
.cfg-section h3 .danger-pill {
display: inline-block; margin-left: 10px;
padding: 2px 8px; border-radius: 999px;
background: #b91c1c; color: #fff;
font-size: 10px; font-weight: 700; text-transform: uppercase;
letter-spacing: 0.6px; vertical-align: 2px;
}
.cfg-section .section-help {
font-size: 12px; color: var(--text-secondary, #6b7280); margin-top: 4px;
}
.cfg-section .section-body { padding: 18px; }
.cfg-section .section-actions {
padding: 12px 18px;
background: var(--border-light, #fafafa);
border-top: 1px solid var(--border, #e5e7eb);
display: flex; justify-content: flex-end; gap: 8px;
}
.cfg-field { display: grid; grid-template-columns: 220px 1fr; gap: 12px; align-items: start; margin-bottom: 14px; }
.cfg-field:last-child { margin-bottom: 0; }
.cfg-field label { font-size: 13px; color: var(--text-primary, #111827); font-weight: 500; padding-top: 8px; }
.cfg-field .field-help { font-size: 11px; color: var(--text-secondary, #6b7280); margin-top: 4px; }
.cfg-field input[type="text"],
.cfg-field input[type="password"],
.cfg-field input[type="email"],
.cfg-field input[type="number"],
.cfg-field input[type="url"],
.cfg-field select,
.cfg-field textarea {
width: 100%; padding: 8px 12px;
border: 1px solid var(--border, #e5e7eb); border-radius: 8px;
font-size: 13px; box-sizing: border-box;
background: var(--surface, #fff); color: var(--text-primary, #111827);
font-family: inherit;
}
.cfg-field textarea { resize: vertical; min-height: 80px; font-family: var(--font-mono, ui-monospace, "SF Mono", Menlo, monospace); font-size: 12px; }
.cfg-field input:focus, .cfg-field select:focus, .cfg-field textarea:focus {
outline: 2px solid var(--primary, #6366f1); outline-offset: -1px;
}
.cfg-field input.is-secret { font-family: var(--font-mono, ui-monospace, monospace); }
.cfg-field .secret-pill {
display: inline-block; margin-left: 8px;
padding: 1px 6px; border-radius: 4px;
background: #f3f4f6; color: #6b7280;
font-size: 10px; text-transform: uppercase; letter-spacing: 0.5px;
}
.cfg-btn {
padding: 8px 16px; border-radius: 8px; font-size: 13px; font-weight: 500;
border: 1px solid var(--border, #e5e7eb); background: var(--surface, #fff);
cursor: pointer; transition: all 0.15s;
}
.cfg-btn:hover { background: var(--border-light, #f9fafb); }
.cfg-btn.primary { background: var(--primary, #6366f1); color: #fff; border-color: var(--primary, #6366f1); }
.cfg-btn.primary:hover { filter: brightness(1.05); }
.cfg-btn.danger { background: #dc2626; color: #fff; border-color: #dc2626; }
.cfg-btn.danger:hover { filter: brightness(1.05); }
.cfg-btn:disabled { opacity: 0.5; cursor: not-allowed; }
.cfg-loading { padding: 32px 16px; text-align: center; color: var(--text-secondary, #6b7280); font-size: 13px; }
/* Known-but-unset fields (sourced from the known_fields registry) — render
dashed and de-emphasised so the operator sees "this is a knob you can
turn" without confusing it with a populated value. */
.cfg-field.is-unset label { color: var(--text-secondary, #9ca3af); }
.cfg-field.is-unset input[type="text"],
.cfg-field.is-unset input[type="password"],
.cfg-field.is-unset input[type="number"],
.cfg-field.is-unset select,
.cfg-field.is-unset textarea {
border-style: dashed;
background: var(--background, #fafafa);
}
.cfg-field.is-unset .field-help { font-style: italic; }
.cfg-divider {
border: 0;
border-top: 1px dashed var(--border, #e5e7eb);
margin: 12px 0;
}
.cfg-divider-label {
display: block;
font-size: 11px;
color: var(--text-secondary, #9ca3af);
margin-bottom: 8px;
text-transform: uppercase;
letter-spacing: 0.05em;
}
/* Confirmation modal — danger-zone gate */
.modal-backdrop {
position: fixed; inset: 0; background: rgba(15, 23, 42, 0.55);
display: none; align-items: center; justify-content: center; z-index: 1000;
padding: 16px;
}
.modal-backdrop.is-open { display: flex; }
.modal-card {
background: var(--surface, #fff); border-radius: 12px;
padding: 24px; width: 100%; max-width: 520px;
box-shadow: 0 20px 60px rgba(0, 0, 0, 0.25);
}
.modal-card h3 { margin: 0 0 6px; font-size: 17px; font-weight: 600; }
.modal-card p.sub { margin: 0 0 12px; font-size: 13px; color: var(--text-secondary, #6b7280); }
.modal-card .diff-list {
background: var(--border-light, #f9fafb); border-radius: 8px;
padding: 10px 14px; font-family: var(--font-mono, ui-monospace, monospace);
font-size: 12px; max-height: 240px; overflow: auto; margin: 12px 0;
}
.modal-card .diff-row { padding: 4px 0; border-bottom: 1px dashed var(--border, #e5e7eb); }
.modal-card .diff-row:last-child { border-bottom: none; }
.modal-card .diff-row .path { color: #b91c1c; font-weight: 600; }
.modal-actions { display: flex; gap: 8px; justify-content: flex-end; margin-top: 18px; }
</style>
<div class="cfg-page" data-page="server-config">
<div class="cfg-toolbar">
<div>
<h2 class="cfg-title">Server configuration</h2>
<div class="cfg-subtitle">Edits land in <code>instance.yaml</code>. Save triggers an app restart (~10s downtime). Secret values are masked here — re-enter them to change.</div>
</div>
</div>
<div id="cfg-banner" class="cfg-banner" role="status" aria-live="polite"></div>
<div id="cfg-loading" class="cfg-loading">Loading current configuration…</div>
<div id="cfg-sections" hidden></div>
<!--
Initial Workspace Template — admin-configurable per-instance override
for `agnes init` analyst workspace. Lives on this page but is NOT part
of the generic instance.yaml form save (data routes through dedicated
/api/admin/initial-workspace endpoints because of PAT handling).
See docs/initial-workspace-override.md for the full responsibility-
transfer contract.
Visual shape matches the other .cfg-section blocks on this page
(header / section-body / section-actions) so the page reads as one
cohesive panel.
-->
<section class="cfg-section" id="iw-section">
<header>
<div>
<h3>Initial Workspace Template</h3>
<div class="section-help">
Optional. Replace the default <code>agnes init</code> workspace
skeleton with content from your own Git repo. When set, Agnes
ships <strong>none</strong> of its own files — your repo is
authoritative for CLAUDE.md, hooks, slash commands, settings,
and folder layout. See <code>docs/initial-workspace-override.md</code>
for the full responsibility-transfer contract.
</div>
</div>
</header>
<div class="section-body" id="iw-body">
<div id="iw-loading" class="cfg-loading">Loading…</div>
</div>
<div class="section-actions" id="iw-actions" hidden></div>
</section>
</div>
<!-- Initial Workspace Template — register / edit modal.
Form fields use a dedicated stacked layout (.iw-form-field) — NOT the
page-level .cfg-field grid (which is 220px label / 1fr value, designed
for the wide section body, not a 480px modal). Inside the modal,
stacking label-above-input is the standard for narrow forms. -->
<div class="modal-backdrop" id="iw-modal" role="dialog" aria-modal="true" aria-labelledby="iw-modal-title">
<style>
.iw-form-field { display: block; margin-bottom: 14px; }
.iw-form-field label {
display: block; font-size: 13px; font-weight: 500;
color: var(--text-primary, #111827); margin-bottom: 4px;
}
.iw-form-field label .iw-optional {
font-weight: 400; font-size: 11px; color: var(--text-secondary, #9ca3af);
margin-left: 4px;
}
.iw-form-field input {
width: 100%; box-sizing: border-box;
padding: 8px 10px; border-radius: 6px;
border: 1px solid var(--border, #e5e7eb);
background: var(--surface, #fff); font-size: 13px;
font-family: inherit;
}
.iw-form-field input:focus {
outline: none; border-color: var(--primary, #6366f1);
box-shadow: 0 0 0 2px rgba(99, 102, 241, 0.15);
}
.iw-form-field .field-help {
font-size: 11px; color: var(--text-secondary, #6b7280); margin-top: 4px;
}
</style>
<div class="modal-card" style="max-width: 480px;">
<h3 id="iw-modal-title">Link to Template Repository</h3>
<p class="sub" id="iw-modal-sub">Register a Git repo whose contents will replace the default <code>agnes init</code> workspace skeleton.</p>
<div class="iw-form-field">
<label for="iw-url">Repository URL (HTTPS)</label>
<input type="text" id="iw-url" placeholder="https://github.com/your-org/agnes-workspace-template" autocomplete="off">
<div class="field-help">Must be <code>https://</code>. Public repo or PAT-authed private.</div>
</div>
<div class="iw-form-field">
<label for="iw-branch">Branch <span class="iw-optional">(optional)</span></label>
<input type="text" id="iw-branch" placeholder="main" autocomplete="off">
<div class="field-help">Leave empty to track the remote's default branch.</div>
</div>
<div class="iw-form-field">
<label for="iw-token">GitHub PAT <span class="iw-optional">(optional)</span></label>
<input type="password" id="iw-token" placeholder="ghp_••• (leave blank to keep existing)" autocomplete="off">
<div class="field-help">
Required only for private repos. Stored at <code>.env_overlay</code>
(chmod 600), never in the YAML overlay. Leave blank to keep an
existing PAT; type a value to rotate.
</div>
</div>
<div class="modal-actions">
<button class="cfg-btn" data-close-modal="iw-modal">Cancel</button>
<button class="cfg-btn primary" id="iw-modal-save">Save</button>
</div>
</div>
</div>
<!-- Initial Workspace Template — sync result modal -->
<div class="modal-backdrop" id="iw-sync-modal" role="dialog" aria-modal="true" aria-labelledby="iw-sync-title">
<div class="modal-card" style="max-width: 520px;">
<h3 id="iw-sync-title">Sync result</h3>
<div id="iw-sync-body" class="diff-list" style="background: var(--border-light, #f9fafb);"></div>
<div class="modal-actions">
<button class="cfg-btn primary" data-close-modal="iw-sync-modal">Close</button>
</div>
</div>
</div>
<!-- Danger-zone confirmation modal -->
<div class="modal-backdrop" id="danger-modal" role="dialog" aria-modal="true" aria-labelledby="danger-title">
<div class="modal-card">
<h3 id="danger-title">Confirm danger-zone change</h3>
<p class="sub" id="danger-sub"></p>
<div class="diff-list" id="danger-diff"></div>
<p class="sub"><strong>Save anyway?</strong> An app restart is required for the change to take effect.</p>
<div class="modal-actions">
<button class="cfg-btn" data-close-modal="danger-modal">Cancel</button>
<button class="cfg-btn danger" id="danger-confirm-btn">Yes, save</button>
</div>
</div>
</div>
<script>
const CFG_API = "/api/admin/server-config";
// Secret-key heuristic — must match the server's _is_secret_key() patterns
// so the UI redacts the same fields the API would mask. Re-defined here
// instead of fetched so a render with the pre-loaded redacted payload
// still labels every secret field correctly even before the GET resolves.
const SECRET_PATTERNS = ["secret", "token", "password", "api_key"];
function isSecretKey(key) {
const k = (key || "").toLowerCase();
return SECRET_PATTERNS.some(p => k.includes(p));
}
// Section copy — kept short; the issue's Scope section explains the rest.
const SECTION_META = {
instance: { title: "Instance", help: "Branding shown in the header and emails." },
data_source: { title: "Data source", help: "Switch source type or update connection details. Optional BQ + Keboola knobs render below as structured fields with hints; expand each to edit." },
email: { title: "Email (SMTP)", help: "SMTP relay for magic-link login. Leave blank to disable." },
telegram: { title: "Telegram", help: "Bot credentials for notifications." },
jira: { title: "Jira", help: "Jira webhook + REST credentials." },
theme: { title: "Theme", help: "Brand colors and typography." },
server: { title: "Server", help: "Hostname and host. Changing these can break OAuth callbacks." },
auth: { title: "Authentication", help: "Allowed sign-in domain and Google OAuth keys. Misconfiguration can lock everyone out." },
ai: { title: "AI / LLM", help: "Provider + API key for the corporate-memory extractor. provider=anthropic|openai_compat; api_key uses ${ENV_VAR} so the secret stays in .env." },
openmetadata: { title: "OpenMetadata", help: "Optional REST catalog enrichment. Without it, the app runs without catalog cross-links." },
desktop: { title: "Desktop app", help: "JWT auth for the desktop client (rarely changed)." },
corporate_memory: {
title: "Corporate Memory",
help: "Optional governance for AI-extracted knowledge. When the section is unset, the system runs in legacy democratic-wiki mode with no admin review.",
},
materialize: {
title: "Materialize",
help: "Concurrency safety net for the materialize path. Controls the file-lock TTL used to detect and reclaim stale locks from hard-killed processes.",
},
guardrails: {
title: "Flea-market guardrails",
help: "Per-component content quality thresholds for store uploads. Lower the min-* knobs to relax the bar; raise to push submitters toward longer, more useful descriptions. The LLM tier (review_model + enabled) governs the second-stage substantive review.",
},
};
const DANGER_SECTIONS = new Set(["auth", "server"]);
// ── Banner ─────────────────────────────────────────────────────────────
function showBanner(msg, kind) {
const el = document.getElementById("cfg-banner");
el.textContent = msg;
el.className = "cfg-banner is-visible" + (kind ? " " + kind : "");
}
function hideBanner() {
document.getElementById("cfg-banner").className = "cfg-banner";
}
// ── Modal helpers ─────────────────────────────────────────────────────
function openModal(id) { document.getElementById(id).classList.add("is-open"); }
function closeModal(id) { document.getElementById(id).classList.remove("is-open"); }
document.querySelectorAll("[data-close-modal]").forEach(el =>
el.addEventListener("click", () => closeModal(el.dataset.closeModal)));
document.querySelectorAll(".modal-backdrop").forEach(el => {
el.addEventListener("click", e => { if (e.target === el) el.classList.remove("is-open"); });
});
// ── State ─────────────────────────────────────────────────────────────
// `original` keeps the redacted payload from GET — used for the diff
// preview in the danger-zone confirmation. Don't mutate it after load.
let original = {};
// ── Render ────────────────────────────────────────────────────────────
function escHtml(s) {
// textContent → innerHTML only escapes <, >, &. We splice the result
// into HTML attribute values like `value="${escHtml(v)}"`, where a
// raw " breaks out of the attribute and a raw ' breaks out when the
// attribute uses single quotes — both are stored-XSS vectors when
// config values come from a malicious admin. Escape both explicitly.
const d = document.createElement("div");
d.textContent = s == null ? "" : String(s);
return d.innerHTML.replace(/"/g, "&quot;").replace(/'/g, "&#39;");
}
// Encode a segment array as a JSON-string suitable for an HTML attribute.
// We store the path as JSON rather than dot-joined so that map keys (which
// are user-supplied data and can themselves contain '.', e.g.
// "user_verification.correction" in confidence.base) round-trip intact —
// splitting `data-key` on '.' would shred them into bogus extra segments.
function encodePath(segments) {
return escHtml(JSON.stringify(segments || []));
}
// Build a basic <input>/<select>/<textarea> for a leaf field. Returns the
// HTML for the input element only — the wrapping <div class="cfg-field">
// + label + hint is added by the caller.
//
// `pathSegments` is the array of registry path segments down to this leaf
// (e.g. ["bigquery", "billing_project"]). It's emitted as a JSON-encoded
// `data-path` attribute that the collector reads to rebuild the nested
// patch shape — bypassing the old dotted-string-splitting which would
// mis-parse map keys with embedded dots.
//
// `dottedKey` is kept for backward-compat / debugging; collectSection
// prefers data-path when present.
function renderLeafInput(fieldId, section, pathSegments, kind, value, opts, isUnset) {
const dottedKey = (pathSegments || []).join(".");
const dataPath = encodePath(pathSegments);
const leafKey = pathSegments && pathSegments.length ? pathSegments[pathSegments.length - 1] : "";
const isSecret = isSecretKey(String(leafKey)) || kind === "secret";
if (kind === "secret") {
const ph = isUnset
? "unset — type to set"
: (value === "<empty>" ? "unset — type to set" : "*** — type to overwrite");
return `<input id="${fieldId}" type="password" class="is-secret" data-section="${section}" data-key="${escHtml(dottedKey)}" data-path="${dataPath}" placeholder="${escHtml(ph)}" autocomplete="off">`;
}
if (kind === "int") {
const v = (value == null || value === "") ? "" : value;
return `<input id="${fieldId}" type="number" data-section="${section}" data-key="${escHtml(dottedKey)}" data-path="${dataPath}" value="${escHtml(v)}">`;
}
if (kind === "float") {
const v = (value == null || value === "") ? "" : value;
return `<input id="${fieldId}" type="number" step="any" data-section="${section}" data-key="${escHtml(dottedKey)}" data-path="${dataPath}" data-cast="float" value="${escHtml(v)}">`;
}
if (kind === "bool") {
const v = !!value;
return `<select id="${fieldId}" data-section="${section}" data-key="${escHtml(dottedKey)}" data-path="${dataPath}" data-cast="bool">
<option value="true" ${v ? "selected" : ""}>true</option>
<option value="false" ${!v ? "selected" : ""}>false</option>
</select>`;
}
if (kind === "select" && Array.isArray(opts && opts.spec && opts.spec.options)) {
const sel = value == null ? "" : String(value);
const options = opts.spec.options.map(o => {
const ov = String(o);
return `<option value="${escHtml(ov)}" ${sel === ov ? "selected" : ""}>${escHtml(ov)}</option>`;
}).join("");
return `<select id="${fieldId}" data-section="${section}" data-key="${escHtml(dottedKey)}" data-path="${dataPath}">${options}</select>`;
}
// Default: text. Use the registry's default when unset, else the value.
const v = isUnset
? (opts && opts.spec && opts.spec.default != null ? String(opts.spec.default) : "")
: (value == null ? "" : value);
// Issue #160 §4.7.5: `placeholder_from: ["a","b","c"]` walks the loaded
// `original` config dict and shows "(defaults to <resolved>)" greyed in
// the empty input. Used by data_source.bigquery.billing_project to
// surface the access.py:339-340 billing→data fallback in the UI.
let placeholderAttr = "";
if (isUnset && opts && opts.spec && Array.isArray(opts.spec.placeholder_from)) {
// `original` is the full GET /api/admin/server-config response shape:
// {sections: {data_source: ...}, editable_sections: [...], ...}.
// `placeholder_from` is a section-relative path (e.g. ["data_source",
// "bigquery", "project"]) so walk `original.sections` not `original`.
const resolved = opts.spec.placeholder_from.reduce(
(cur, k) => (cur && typeof cur === "object" ? cur[k] : undefined),
original && original.sections ? original.sections : original,
);
if (resolved !== undefined && resolved !== null && resolved !== "") {
placeholderAttr = ` placeholder="(defaults to ${escHtml(String(resolved))})"`;
}
}
return `<input id="${fieldId}" type="text" data-section="${section}" data-key="${escHtml(dottedKey)}" data-path="${dataPath}" value="${escHtml(v)}"${placeholderAttr}>`;
}
// Cast a string raw value to the JS type implied by an item_kind / value_kind.
// Used by the array-of-scalars + map-of-scalars renderers when reading user
// input back out into a structured patch.
function castScalar(raw, kind) {
if (raw === "" || raw == null) return null;
if (kind === "int") {
const n = Number(raw);
return Number.isFinite(n) ? Math.trunc(n) : null;
}
if (kind === "float") {
const n = Number(raw);
return Number.isFinite(n) ? n : null;
}
if (kind === "bool") {
return raw === "true" || raw === true;
}
return String(raw);
}
// Render an array of scalars (e.g. detection_types: ["correction", ...]).
// Produces a vertical stack of text inputs, one per item, plus an add/remove
// affordance per row and a trailing "+ Add" button. The container's
// data-array-collect path collects each row's value at save time.
function renderArrayField(section, pathSegments, label, value, spec, depth) {
spec = spec || {};
const indent = (depth || 0) * 24;
const itemKind = spec.item_kind || "string";
const items = Array.isArray(value) ? value
: (value === undefined && Array.isArray(spec.default) ? spec.default : []);
const dataPath = encodePath(pathSegments);
const dottedKey = (pathSegments || []).join(".");
const arrow = depth > 0 ? "↳ " : "";
const hintBlock = spec.hint
? `<div class="field-help">${escHtml(spec.hint)}</div>`
: "";
// `data-array-collect="1"` marks the wrapper so collectSection can pick
// it up as a single unit (otherwise the per-row inputs would each emit
// their own patch leaf and clobber each other).
const rows = items.map((item, idx) => `
<div class="array-row" data-array-row="${idx}" style="display: flex; gap: 6px; margin-bottom: 4px;">
<input type="text" class="array-item-input" data-array-item="${idx}" value="${escHtml(item == null ? "" : String(item))}" style="flex: 1;">
<button type="button" class="cfg-btn" data-array-remove="${idx}" title="Remove this item">×</button>
</div>`).join("");
return `
<div class="cfg-field nested-field" style="margin-left: ${indent}px;">
<label>${arrow}${escHtml(label)}</label>
<div>
<div class="array-container"
data-section="${section}"
data-key="${escHtml(dottedKey)}"
data-path="${dataPath}"
data-array-collect="1"
data-item-kind="${escHtml(itemKind)}">
<div class="array-rows">${rows}</div>
<button type="button" class="cfg-btn" data-array-add="1" data-item-kind="${escHtml(itemKind)}">+ Add item</button>
</div>
${hintBlock}
</div>
</div>`;
}
// Render a map of string → scalar/array/object (e.g. confidence.base:
// {"user_verification.correction": 0.9, ...}). Produces a vertical stack
// of <key-input>: <value-input> rows plus a "+ Add row" button. Map keys
// are user-supplied data and may contain dots — we never reuse them as
// path segments at collect time; instead they become the *final* path
// segment of each row, JSON-encoded so collectors don't split them.
function renderMapField(section, pathSegments, label, value, spec, depth) {
spec = spec || {};
const indent = (depth || 0) * 24;
const valueKind = spec.value_kind || "string";
const valueItemKind = spec.value_item_kind || "string"; // for value_kind="array"
// Use registry default only when the value is genuinely missing (undefined).
// An explicit empty {} from YAML must not get backfilled with the example default.
const obj = (value && typeof value === "object" && !Array.isArray(value))
? value
: (value === undefined && spec.default && typeof spec.default === "object" ? spec.default : {});
const dataPath = encodePath(pathSegments);
const dottedKey = (pathSegments || []).join(".");
const arrow = depth > 0 ? "↳ " : "";
const hintBlock = spec.hint
? `<div class="field-help">${escHtml(spec.hint)}</div>`
: "";
const renderRow = (k, v, idx) => {
if (valueKind === "array") {
// Map<string, array<scalar>> — value column is itself a comma-separated
// text input. Operator can edit the list inline; collectSection splits
// on commas. (Full nested array UI inside a map row would require more
// wiring; comma-list is the pragmatic compromise.)
const items = Array.isArray(v) ? v.join(", ") : "";
return `
<div class="map-row" data-map-row="${idx}" style="display: grid; grid-template-columns: minmax(160px, 1fr) 2fr auto; gap: 6px; margin-bottom: 4px;">
<input type="text" class="map-key-input" data-map-key="${idx}" value="${escHtml(String(k))}" placeholder="key">
<input type="text" class="map-value-input" data-map-value="${idx}" value="${escHtml(items)}" placeholder="comma,separated,values">
<button type="button" class="cfg-btn" data-map-remove="${idx}" title="Remove this row">×</button>
</div>`;
}
// Scalar value column.
const inputType = (valueKind === "int" || valueKind === "float") ? "number" : "text";
const stepAttr = valueKind === "float" ? ' step="any"' : "";
return `
<div class="map-row" data-map-row="${idx}" style="display: grid; grid-template-columns: minmax(160px, 1fr) 1fr auto; gap: 6px; margin-bottom: 4px;">
<input type="text" class="map-key-input" data-map-key="${idx}" value="${escHtml(String(k))}" placeholder="key">
<input type="${inputType}"${stepAttr} class="map-value-input" data-map-value="${idx}" value="${escHtml(v == null ? "" : String(v))}" placeholder="value">
<button type="button" class="cfg-btn" data-map-remove="${idx}" title="Remove this row">×</button>
</div>`;
};
const rows = Object.entries(obj).map(([k, v], idx) => renderRow(k, v, idx)).join("");
return `
<div class="cfg-field nested-field" style="margin-left: ${indent}px;">
<label>${arrow}${escHtml(label)}</label>
<div>
<div class="map-container"
data-section="${section}"
data-key="${escHtml(dottedKey)}"
data-path="${dataPath}"
data-map-collect="1"
data-value-kind="${escHtml(valueKind)}"
data-value-item-kind="${escHtml(valueItemKind)}">
<div class="map-rows">${rows}</div>
<button type="button" class="cfg-btn" data-map-add="1" data-value-kind="${escHtml(valueKind)}">+ Add row</button>
</div>
${hintBlock}
</div>
</div>`;
}
// Render a single nested subfield row recursively. Each leaf input carries
// `data-path` (JSON-encoded segment array) so collectSection can rebuild
// the nested patch shape without splitting on '.' — important for map keys
// that themselves contain dots (e.g. confidence.base keys like
// "user_verification.correction").
//
// Recursion: arbitrary depth supported. When a child spec has kind="object"
// with its own `fields` map, we recurse with the indent bumped up. The depth
// bound is implicit (browser stack); registries with ridiculous depth would
// blow up, but the entries we ship max out around 4 levels (corporate_memory
// in subagent 3) which is comfortably within budget.
//
// `pathSegments` — array of registry path segments down to this field (e.g.
// ["bigquery", "billing_project"]). Used both for the rendered data-path
// attribute and to derive the legacy dotted key for back-compat.
function renderNestedField(section, pathSegments, label, value, spec, depth) {
spec = spec || {};
const segs = Array.isArray(pathSegments) ? pathSegments : [pathSegments];
const dottedKey = segs.join(".");
const indent = (depth || 0) * 24;
const kind = spec.kind || (
Array.isArray(value) ? "array"
: typeof value === "number" ? "int"
: typeof value === "boolean" ? "bool"
: (value && typeof value === "object") ? "object"
: "string"
);
const isSecret = isSecretKey(label) || kind === "secret";
const isUnset = (value === undefined);
const fieldId = `f_${section}_${dottedKey.replace(/\W/g, "_")}`;
const wrapperClass = "cfg-field nested-field" + (isUnset ? " is-unset" : "");
const arrow = depth > 0 ? "↳ " : "";
const secretPill = isSecret ? '<span class="secret-pill">secret</span>' : "";
const hintBlock = spec.hint
? `<div class="field-help">${escHtml(spec.hint)}</div>`
: "";
// Array-of-scalars: dedicated stack-of-inputs renderer.
if (kind === "array" && spec.item_kind && spec.item_kind !== "object") {
return renderArrayField(section, segs, label, value, spec, depth);
}
// Map<string, …>: dedicated key/value-row renderer. Handles map of scalars,
// map of arrays, and (with a JSON-textarea fallback) map of complex objects.
if (kind === "map") {
if (spec.value_kind === "object" && spec.value_fields && Object.keys(spec.value_fields).length > 0) {
// TODO: structured editor for "map of objects with declared subfields"
// (e.g. confidence.modifiers — Map<string, Map<string, float>>).
// Falls through to the JSON-textarea fallback below for now.
} else {
return renderMapField(section, segs, label, value, spec, depth);
}
}
// Registry-declared object with explicit fields → recurse for each child
// as a structured form; emit a header row for the parent.
if (kind === "object" && spec.fields && typeof spec.fields === "object") {
const childValue = (value && typeof value === "object" && !Array.isArray(value)) ? value : {};
const knownChildKeys = Object.keys(spec.fields);
const knownSet = new Set(knownChildKeys);
const populatedChildKeys = Object.keys(childValue).filter(k => knownSet.has(k));
const unsetChildKeys = knownChildKeys.filter(k => !(k in childValue));
// YAML-only keys that aren't in the registry — preserve via a small JSON
// expander so admins who hand-edited an unusual key in the YAML don't
// lose it on round-trip. Keys are still editable as a single JSON blob
// (deliberately less prominent than registry-known leaves).
const fallbackKeys = Object.keys(childValue).filter(k => !knownSet.has(k));
const fallbackBlob = fallbackKeys.length
? Object.fromEntries(fallbackKeys.map(k => [k, childValue[k]]))
: null;
const renderChild = (k) => renderNestedField(
section,
segs.concat([k]),
k,
(k in childValue) ? childValue[k] : undefined,
spec.fields[k] || {},
(depth || 0) + 1,
);
const populatedHtml = populatedChildKeys.sort().map(renderChild).join("");
const unsetHtml = unsetChildKeys.sort().map(renderChild).join("");
const fallbackHtml = fallbackBlob
? (() => {
const fbId = `f_${section}_${dottedKey.replace(/\W/g, "_")}_fallback`;
const fbPath = encodePath(segs.concat(["__other__"]));
// The fallback uses the same path convention with a literal
// "__other__" leaf so the collector emits it under the parent
// in collectSection. Cast=json so the textarea content
// round-trips as an object.
const indentInner = ((depth || 0) + 1) * 24;
return `
<div class="cfg-field" style="margin-left: ${indentInner}px;">
<label for="${fbId}">↳ Other (YAML-only) keys</label>
<div>
<textarea id="${fbId}" data-section="${section}" data-key="${escHtml(dottedKey + ".__other__")}" data-path="${fbPath}" data-cast="json">${escHtml(JSON.stringify(fallbackBlob, null, 2))}</textarea>
<div class="field-help">Keys present in YAML but not in the registry. Edit as a JSON object — keys at this layer survive round-trip.</div>
</div>
</div>`;
})()
: "";
return `
<div class="cfg-field nested-field nested-parent" style="margin-left: ${indent}px;">
<label>${arrow}${escHtml(label)}</label>
<div>${hintBlock || `<div class="field-help">Nested structured fields below.</div>`}</div>
</div>
${populatedHtml}${unsetHtml}${fallbackHtml}`;
}
// Leaf field (string / int / float / bool / secret / select / array,
// OR an object without explicit `fields`, OR a map with complex values
// — the last two fall back to JSON).
let inp;
if (kind === "object" || kind === "map" || kind === "array") {
// No explicit structured renderer for this shape — JSON-textarea
// fallback so a YAML-populated subtree still round-trips even
// without finer-grained schema.
const blobValue = isUnset ? "" : JSON.stringify(value || (kind === "array" ? [] : {}), null, 2);
const dataPath = encodePath(segs);
inp = `<textarea id="${fieldId}" data-section="${section}" data-key="${escHtml(dottedKey)}" data-path="${dataPath}" data-cast="json" placeholder="${isUnset ? 'unset — paste JSON to populate' : ''}">${escHtml(blobValue)}</textarea>`;
} else {
inp = renderLeafInput(fieldId, section, segs, kind, value, { spec }, isUnset);
}
return `
<div class="${wrapperClass}" style="margin-left: ${indent}px;">
<label for="${fieldId}">${arrow}${escHtml(label)}${secretPill}</label>
<div>${inp}${hintBlock}</div>
</div>`;
}
function renderField(section, key, value, opts) {
// opts: { isUnset: bool, hint: string, kind: string, spec: {…} }
// - isUnset: render the field as a dashed placeholder (.is-unset) so the
// operator can tell at a glance that the value is sourced from the
// known_fields registry rather than the live YAML.
// - hint: one-line operator-facing help (rendered as .field-help).
// - kind: registry-declared input kind. Overrides the typeof-value
// heuristic for known-but-unset entries (we have no value to inspect).
// - spec: the raw registry entry — when kind="object" + spec.fields is
// declared, we render a fully-editable structured form (every leaf is
// a real input with a dotted-path data-key so collectSection rebuilds
// the nested patch). When spec.fields is absent / the object isn't in
// the registry, we fall back to the JSON-textarea path.
opts = opts || {};
const isUnset = !!opts.isUnset;
const valueForKind = isUnset ? undefined : value;
// Registry-declared structured object → delegate to the recursive
// nested-form renderer. Replaces the old read-only preview path.
if (opts.kind === "object" && opts.spec && opts.spec.fields && typeof opts.spec.fields === "object") {
return renderNestedField(section, [key], key, valueForKind, opts.spec, 0);
}
// Pass through ALL spec fields (item_kind, key_kind, value_kind, fields,
// value_fields, default, options, hint) so the top-level entry point can
// render arrays, maps, and primitive leaves correctly.
return renderNestedField(section, [key], key, valueForKind, opts.spec || {
kind: opts.kind,
hint: opts.hint,
}, 0);
}
function renderSection(section, payload, knownForSection) {
// knownForSection: registry slice for this section, e.g.
// { bigquery: { kind: "object", hint: "...", fields: { billing_project: {...} } } }
// Keys present in `payload` render as populated; keys present in
// `knownForSection` but absent from `payload` render as dashed
// placeholders (.is-unset).
const meta = SECTION_META[section] || { title: section, help: "" };
const isDanger = DANGER_SECTIONS.has(section);
const danger = isDanger ? '<span class="danger-pill">danger</span>' : "";
const populatedKeys = Object.keys(payload || {}).sort();
const known = knownForSection || {};
const populatedSet = new Set(populatedKeys);
const knownUnsetKeys = Object.keys(known).filter(k => !populatedSet.has(k)).sort();
const populatedHtml = populatedKeys.map(k => {
const spec = known[k] || {};
return renderField(section, k, payload[k], {
isUnset: false,
hint: spec.hint || "",
kind: spec.kind, // may be undefined; renderField falls back to typeof inference
spec,
});
}).join("");
const unsetHtml = knownUnsetKeys.map(k => {
const spec = known[k] || {};
return renderField(section, k, undefined, {
isUnset: true,
hint: spec.hint || "",
kind: spec.kind || "string",
spec,
});
}).join("");
// Visual divider between populated and known-but-unset rows so the
// operator sees at a glance which knobs they're already using vs which
// ones the registry exposes for them.
const divider = (populatedHtml && unsetHtml)
? `<hr class="cfg-divider"><span class="cfg-divider-label">Available but unset</span>`
: (unsetHtml ? `<span class="cfg-divider-label">Available but unset</span>` : "");
const fieldsHtml = (populatedHtml || unsetHtml)
? (populatedHtml + divider + unsetHtml)
: `<div class="section-help">No fields populated yet — type below to add common keys, or edit the YAML directly via the API.</div>`;
// For empty sections (no populated *and* no known-but-unset), give the
// operator a textarea so they can paste a YAML/JSON blob to bootstrap
// the section. We persist it via the JSON cast so non-trivial structures
// still merge correctly.
const bootstrap = (populatedKeys.length === 0 && knownUnsetKeys.length === 0)
? `<div class="cfg-field">
<label for="bootstrap_${section}">JSON patch</label>
<div>
<textarea id="bootstrap_${section}" data-section="${section}" data-key="__bootstrap__" data-cast="json" placeholder='{"name": "Acme Analyst", ...}'></textarea>
<div class="field-help">Paste a JSON object to populate this section.</div>
</div>
</div>`
: "";
return `
<section class="cfg-section ${isDanger ? "is-danger" : ""}" data-section="${section}">
<header>
<div>
<h3>${escHtml(meta.title)}${danger}</h3>
<div class="section-help">${escHtml(meta.help)}</div>
</div>
</header>
<div class="section-body">
${fieldsHtml}
${bootstrap}
</div>
<div class="section-actions">
<button class="cfg-btn primary" data-action="save-section" data-section="${section}">Save ${escHtml(meta.title.toLowerCase())}</button>
${section === "data_source" ? `
<button class="cfg-btn" data-action="test-bigquery" type="button">Test BigQuery connection</button>
<span class="bq-test-result" data-section="${section}" hidden style="margin-left: 1ex;"></span>
` : ""}
</div>
</section>`;
}
function renderAll(data) {
const wrap = document.getElementById("cfg-sections");
const sections = data.editable_sections || Object.keys(data.sections || {});
const known = data.known_fields || {};
wrap.innerHTML = sections.map(s => renderSection(s, data.sections[s] || {}, known[s] || {})).join("");
document.getElementById("cfg-loading").style.display = "none";
wrap.hidden = false;
wrap.querySelectorAll('[data-action="save-section"]').forEach(btn =>
btn.addEventListener("click", () => onSaveSection(btn.dataset.section)));
// Issue #160 §4.9: Test BigQuery connection — admin probe to verify the
// saved data_source.bigquery config is reachable WITHOUT having to
// ssh to the VM or wait for an analyst's failed query.
wrap.querySelectorAll('[data-action="test-bigquery"]').forEach(btn =>
btn.addEventListener("click", () => onTestBigQuery(btn)));
// Wire array-of-scalars + map-of-scalars add/remove buttons via event
// delegation on the wrapper. Re-attaching after every renderAll() is
// fine because we replace innerHTML wholesale on each load.
wrap.addEventListener("click", (e) => {
const target = e.target;
if (!(target instanceof Element)) return;
// Add an array row.
if (target.dataset.arrayAdd) {
const container = target.closest('[data-array-collect="1"]');
if (!container) return;
const rows = container.querySelector('.array-rows');
const idx = rows.querySelectorAll('[data-array-row]').length;
const div = document.createElement("div");
div.className = "array-row";
div.dataset.arrayRow = String(idx);
div.style.display = "flex";
div.style.gap = "6px";
div.style.marginBottom = "4px";
div.innerHTML = `<input type="text" class="array-item-input" data-array-item="${idx}" value="" style="flex: 1;">
<button type="button" class="cfg-btn" data-array-remove="${idx}" title="Remove this item">×</button>`;
rows.appendChild(div);
const inp = div.querySelector('input');
if (inp) inp.focus();
return;
}
// Remove an array row.
if (target.dataset.arrayRemove != null) {
const row = target.closest('[data-array-row]');
if (row) row.remove();
return;
}
// Add a map row.
if (target.dataset.mapAdd) {
const container = target.closest('[data-map-collect="1"]');
if (!container) return;
const valueKind = container.dataset.valueKind || "string";
const rows = container.querySelector('.map-rows');
const idx = rows.querySelectorAll('[data-map-row]').length;
const div = document.createElement("div");
div.className = "map-row";
div.dataset.mapRow = String(idx);
div.style.display = "grid";
div.style.gridTemplateColumns = valueKind === "array"
? "minmax(160px, 1fr) 2fr auto"
: "minmax(160px, 1fr) 1fr auto";
div.style.gap = "6px";
div.style.marginBottom = "4px";
const valuePlaceholder = valueKind === "array" ? "comma,separated,values" : "value";
const inputType = (valueKind === "int" || valueKind === "float") ? "number" : "text";
const stepAttr = valueKind === "float" ? ' step="any"' : "";
div.innerHTML = `<input type="text" class="map-key-input" data-map-key="${idx}" value="" placeholder="key">
<input type="${inputType}"${stepAttr} class="map-value-input" data-map-value="${idx}" value="" placeholder="${valuePlaceholder}">
<button type="button" class="cfg-btn" data-map-remove="${idx}" title="Remove this row">×</button>`;
rows.appendChild(div);
const inp = div.querySelector('input');
if (inp) inp.focus();
return;
}
// Remove a map row.
if (target.dataset.mapRemove != null) {
const row = target.closest('[data-map-row]');
if (row) row.remove();
return;
}
});
}
// Recursively strip secret-keyed leaves whose value is the redaction sentinel
// (`***` or `<empty>`) so a JSON-textarea round-trip can't overwrite real
// overlay secrets with the placeholder shown in the form. The GET handler
// redacts secret-keyed children inside nested objects (token_env contains
// "token", so it gets masked alongside actual credentials), and the textarea
// renders the masked JSON verbatim — without this scrub a no-op save of e.g.
// `data_source.keboola` would persist `token_env: "***"` on top of the real
// value `"KEBOOLA_STORAGE_TOKEN"` and silently break the next sync.
function scrubRedactedSecrets(value) {
if (Array.isArray(value)) return value.map(scrubRedactedSecrets);
if (value && typeof value === "object") {
const out = {};
for (const [k, v] of Object.entries(value)) {
if (isSecretKey(k) && (v === "***" || v === "<empty>")) continue;
out[k] = scrubRedactedSecrets(v);
}
return out;
}
return value;
}
// Resolve the registry-path segments for a leaf input. We prefer the
// JSON-encoded `data-path` attribute (introduced for array/map renderers
// where data keys can themselves contain dots) and fall back to splitting
// the legacy `data-key` on '.' for older inputs.
//
// The "__other__" segment is the YAML-fallback expander — its parsed
// content is merged into the parent dict (not nested under the literal
// segment). See `setNested` for that special case.
function resolvePath(el) {
const raw = el.dataset && el.dataset.path;
if (raw) {
try {
const arr = JSON.parse(raw);
if (Array.isArray(arr)) return arr.map(s => String(s));
} catch (_) {
// fall through to dotted-key parsing
}
}
const dotKey = el.dataset && el.dataset.key;
if (!dotKey) return [];
return dotKey.split(".");
}
// Legacy alias kept for tests asserting on the helper name.
function splitDotted(dotKey) {
if (!dotKey) return [];
return dotKey.split(".");
}
// Set value at a nested path inside `out`, creating intermediate dicts as
// needed. The "__other__" segment is special-cased: its dict value gets
// merged into the parent rather than stored under the literal segment.
function setNested(out, segments, value) {
if (!segments.length) return;
let node = out;
for (let i = 0; i < segments.length - 1; i++) {
const seg = segments[i];
if (typeof node[seg] !== "object" || node[seg] === null || Array.isArray(node[seg])) {
node[seg] = {};
}
node = node[seg];
}
const last = segments[segments.length - 1];
if (last === "__other__") {
// Fallback expander: merge the JSON object into the parent. Skip if the
// user cleared the textarea or the value isn't an object.
if (value && typeof value === "object" && !Array.isArray(value)) {
Object.assign(node, value);
}
return;
}
node[last] = value;
}
// Collect the value of an array-of-scalars container (data-array-collect="1")
// — concatenates each non-empty row's input cast to the declared item_kind.
function collectArrayContainer(container) {
const itemKind = container.dataset.itemKind || "string";
const inputs = container.querySelectorAll('input[data-array-item]');
const out = [];
for (const inp of inputs) {
const raw = inp.value;
if (raw === "" || raw == null) continue; // drop blank rows
const cast = castScalar(raw, itemKind);
if (cast === null) continue;
out.push(cast);
}
return out;
}
// Collect the value of a map-of-scalars container (data-map-collect="1")
// — pairs each row's key-input + value-input, casting the value to the
// declared value_kind. Map keys keep their literal string form (we never
// split them on '.' — that's the whole point of the data-path/JSON encoding).
function collectMapContainer(container) {
const valueKind = container.dataset.valueKind || "string";
const valueItemKind = container.dataset.valueItemKind || "string";
const rows = container.querySelectorAll('[data-map-row]');
const out = {};
for (const row of rows) {
const keyInput = row.querySelector('[data-map-key]');
const valInput = row.querySelector('[data-map-value]');
if (!keyInput) continue;
const key = keyInput.value;
if (!key) continue; // skip incomplete rows
let value;
if (valueKind === "array") {
// Comma-separated list → array of scalars cast to value_item_kind.
const raw = valInput ? valInput.value : "";
value = raw.split(",").map(s => s.trim()).filter(s => s.length > 0)
.map(s => castScalar(s, valueItemKind))
.filter(v => v !== null);
} else {
const raw = valInput ? valInput.value : "";
value = castScalar(raw, valueKind);
if (value === null && raw === "") continue; // drop empty values
}
out[key] = value;
}
return out;
}
// ── Collect form values for one section ───────────────────────────────
function collectSection(section) {
const sectionRoot = document.querySelector(`section.cfg-section[data-section="${section}"]`)
|| document;
const patch = {};
// Track ancestor paths covered by an array/map container so we don't
// double-collect their inner inputs as individual leaves.
const handledRoots = new Set();
// 1) Array containers — collect each as a single leaf.
const arrayContainers = sectionRoot.querySelectorAll('[data-array-collect="1"]');
for (const c of arrayContainers) {
if (c.dataset.section && c.dataset.section !== section) continue;
const segments = resolvePath(c);
if (!segments.length) continue;
handledRoots.add(c);
const arr = collectArrayContainer(c);
setNested(patch, segments, arr);
}
// 2) Map containers — collect each as a single dict leaf.
const mapContainers = sectionRoot.querySelectorAll('[data-map-collect="1"]');
for (const c of mapContainers) {
if (c.dataset.section && c.dataset.section !== section) continue;
const segments = resolvePath(c);
if (!segments.length) continue;
handledRoots.add(c);
const obj = collectMapContainer(c);
setNested(patch, segments, obj);
}
// 3) Plain leaf inputs (everything outside an array/map container).
const inputs = document.querySelectorAll(`[data-section="${section}"]`);
for (const el of inputs) {
if (el.dataset.action) continue; // skip buttons
// Skip inner inputs that belong to an array/map container we already
// collected as a single unit.
if (el.closest('[data-array-collect="1"]') || el.closest('[data-map-collect="1"]')) {
// …unless the element IS itself the container (the container also
// carries data-section). In that case it was already handled above.
continue;
}
const dotKey = el.dataset.key;
if (!dotKey && !el.dataset.path) continue;
let raw = el.value;
// Skip empty secret fields — operator left them blank to preserve the
// existing value. Sending "" would overwrite the secret with empty.
if (el.classList.contains("is-secret") && raw === "") continue;
let value;
if (dotKey === "__bootstrap__") {
// Bootstrap textarea — parse the entire blob and merge it as the
// section patch. Skip empty input entirely. Scrub redacted sentinels
// out of the parsed object so a round-trip can't overwrite real
// secrets with `"***"`.
if (!raw.trim()) continue;
try { Object.assign(patch, scrubRedactedSecrets(JSON.parse(raw))); }
catch (e) { throw new Error(`Bootstrap JSON for "${section}" is not valid JSON: ${e.message}`); }
continue;
}
if (el.dataset.cast === "bool") {
value = raw === "true";
} else if (el.dataset.cast === "float") {
value = raw === "" ? null : Number(raw);
} else if (el.dataset.cast === "json") {
if (!raw.trim()) {
// Empty JSON textarea: skip entirely so a blank fallback expander
// doesn't wipe its parent. The deep-merge on the server preserves
// whatever's already on disk for this slot.
continue;
}
try { value = scrubRedactedSecrets(JSON.parse(raw)); }
catch (e) { throw new Error(`Field ${section}.${dotKey} is not valid JSON: ${e.message}`); }
} else if (el.type === "number") {
value = raw === "" ? null : Number(raw);
} else {
value = raw;
}
// If the operator left a secret-keyed scalar at the redaction sentinel
// — e.g. typed nothing into a `token_env` text input that already shows
// `"***"` — drop it rather than persisting the placeholder.
const segments = resolvePath(el);
const leafKey = segments[segments.length - 1] || "";
if (isSecretKey(leafKey) && (value === "***" || value === "<empty>")) continue;
setNested(patch, segments, value);
}
return patch;
}
// ── BigQuery test connection (#160 §4.9) ───────────────────────────────
async function onTestBigQuery(btn) {
const resultEl = btn.parentElement.querySelector(".bq-test-result");
resultEl.hidden = false;
resultEl.textContent = "Testing…";
resultEl.style.color = "";
btn.disabled = true;
try {
const r = await fetch("/api/admin/bigquery/test-connection", {
method: "POST",
credentials: "include",
});
if (r.ok) {
const body = await r.json();
resultEl.textContent = `✓ ok (${body.elapsed_ms} ms; billing=${body.billing_project}, data=${body.data_project})`;
resultEl.style.color = "#2a8c4a";
} else {
let body;
try { body = await r.json(); } catch (_) { body = await r.text(); }
const detail = body && typeof body === "object" ? body.detail : body;
const kind = detail && typeof detail === "object" ? (detail.kind || "error") : "error";
const hint = detail && typeof detail === "object" ? (detail.hint || detail.message || "") : String(detail);
resultEl.textContent = `${kind}${hint ? " — " + hint : ""}`;
resultEl.style.color = "#c0392b";
}
} catch (e) {
resultEl.textContent = `✗ network error — ${e.message}`;
resultEl.style.color = "#c0392b";
} finally {
btn.disabled = false;
}
}
// ── Save flow ─────────────────────────────────────────────────────────
async function onSaveSection(section) {
hideBanner();
let patch;
try { patch = collectSection(section); }
catch (e) { showBanner(e.message, "error"); return; }
if (Object.keys(patch).length === 0) {
showBanner(`No changes to save in "${section}".`);
return;
}
const isDanger = DANGER_SECTIONS.has(section);
if (isDanger) {
const confirmed = await confirmDanger(section, patch);
if (!confirmed) return;
}
await postPatch(section, patch, isDanger);
}
function diffPreview(section, patch) {
// Compare patch fields against the redacted original snapshot. Shows the
// operator exactly which keys they're about to change before they
// confirm a danger-zone save.
const before = (original.sections && original.sections[section]) || {};
const rows = [];
for (const [k, v] of Object.entries(patch)) {
const b = before[k];
if (JSON.stringify(b) !== JSON.stringify(v)) {
rows.push({ path: `${section}.${k}`, before: b, after: v });
}
}
return rows;
}
function confirmDanger(section, patch) {
return new Promise(resolve => {
const rows = diffPreview(section, patch);
const sub = `You're about to change the <strong>${escHtml(section)}</strong> section. ` +
`This is flagged as danger-zone — a typo here can lock you out or break OAuth callbacks.`;
document.getElementById("danger-sub").innerHTML = sub;
document.getElementById("danger-diff").innerHTML = rows.length
? rows.map(r => `<div class="diff-row"><span class="path">${escHtml(r.path)}</span> &mdash; ${escHtml(JSON.stringify(r.before))} &rarr; <strong>${escHtml(JSON.stringify(r.after))}</strong></div>`).join("")
: `<em>No visible diff (secret fields are masked in this preview).</em>`;
const btn = document.getElementById("danger-confirm-btn");
const modalEl = document.getElementById("danger-modal");
const cancelBtns = document.querySelectorAll('#danger-modal [data-close-modal]');
const onOk = () => { closeModal("danger-modal"); cleanup(); resolve(true); };
const onCancel = () => { cleanup(); resolve(false); };
// Backdrop click visually closes via the global handler at the top of the
// file, but that handler doesn't know about the Promise — without this
// listener the await would hang and stack listeners on the next save.
const onBackdrop = (e) => { if (e.target === modalEl) { cleanup(); resolve(false); } };
function cleanup() {
btn.removeEventListener("click", onOk);
modalEl.removeEventListener("click", onBackdrop);
cancelBtns.forEach(b => b.removeEventListener("click", onCancel));
}
btn.addEventListener("click", onOk, { once: true });
modalEl.addEventListener("click", onBackdrop);
cancelBtns.forEach(b => b.addEventListener("click", onCancel, { once: true }));
openModal("danger-modal");
});
}
async function postPatch(section, patch, confirmDanger) {
try {
const r = await fetch(CFG_API, {
method: "POST", credentials: "include",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ sections: { [section]: patch }, confirm_danger: confirmDanger }),
});
const data = await r.json().catch(() => ({}));
if (!r.ok) {
showBanner(`Save failed: ${data.detail || r.statusText}`, "error");
return;
}
showBanner(
`Saved "${section}" (${data.diff_count} field(s) changed). Restart the app for the change to take effect.`,
"success",
);
// Re-fetch so the form reflects the new (still-redacted) state.
await loadConfig();
} catch (e) {
showBanner(`Save failed: ${e.message}`, "error");
}
}
// ── Init ──────────────────────────────────────────────────────────────
async function loadConfig() {
try {
const r = await fetch(CFG_API, { credentials: "include" });
if (!r.ok) throw new Error("HTTP " + r.status);
original = await r.json();
renderAll(original);
} catch (e) {
document.getElementById("cfg-loading").textContent = "Failed to load config: " + e.message;
}
}
loadConfig();
// ════════════════════════════════════════════════════════════════════════
// Initial Workspace Template — dedicated lifecycle (NOT part of generic
// instance.yaml form save). Data routes through /api/admin/initial-workspace
// because of PAT routing to .env_overlay.
// ════════════════════════════════════════════════════════════════════════
const IW_API = "/api/admin/initial-workspace";
async function iwLoad() {
const body = document.getElementById("iw-body");
const actions = document.getElementById("iw-actions");
body.innerHTML = '<div class="cfg-loading">Loading…</div>';
actions.hidden = true;
actions.innerHTML = "";
try {
const r = await fetch(IW_API, { credentials: "include" });
if (!r.ok) throw new Error("HTTP " + r.status);
const data = await r.json();
iwRender(data);
} catch (e) {
body.innerHTML = `<div class="cfg-loading">Failed to load: ${escHtml(e.message)}</div>`;
}
}
function iwRender(data) {
const body = document.getElementById("iw-body");
const actions = document.getElementById("iw-actions");
if (!data.configured) {
// Empty state — mirror the bootstrap-textarea pattern used by other
// sections when they have no data yet: friendly explanation in the
// body, primary action in the section-actions footer.
body.innerHTML = `
<div class="section-help" style="font-size: 13px;">
No template repository linked. Click <strong>Link to Template Repository</strong>
to register one. The repo's contents will replace the default
<code>agnes init</code> workspace skeleton for every analyst on this instance.
</div>
`;
actions.innerHTML = `<button class="cfg-btn primary" id="iw-register-btn">Link to Template Repository</button>`;
actions.hidden = false;
document.getElementById("iw-register-btn").addEventListener("click", () => {
iwOpenModal(/* editing */ false, null);
});
return;
}
// Configured — render label/value pairs using the same .cfg-field grid
// layout the other sections use, so the panel reads as part of the page.
const syncedAt = data.last_synced_at
? new Date(data.last_synced_at).toLocaleString()
: "never";
const sha = data.last_commit_sha
? `<code>${escHtml(data.last_commit_sha.slice(0, 10))}</code>`
: '<span style="color: var(--text-secondary, #9ca3af);">never synced</span>';
const tokenLine = data.has_token
? '<span class="secret-pill" style="background:#dcfce7;color:#166534;border-color:#86efac;">PAT set</span>'
: '<span class="secret-pill" style="background:#f3f4f6;color:#6b7280;">no PAT</span>';
const lastError = data.last_error
? `<div class="cfg-field"><label>Last sync error</label><div><div class="cfg-banner error is-visible" style="margin:0;">${escHtml(data.last_error)}</div></div></div>`
: "";
body.innerHTML = `
<div class="cfg-field">
<label>Repository URL</label>
<div><code>${escHtml(data.url)}</code></div>
</div>
<div class="cfg-field">
<label>Branch</label>
<div>${data.branch ? `<code>${escHtml(data.branch)}</code>` : '<span style="color: var(--text-secondary, #9ca3af);">(remote default)</span>'}</div>
</div>
<div class="cfg-field">
<label>GitHub PAT</label>
<div>${tokenLine}</div>
</div>
<div class="cfg-field">
<label>Last sync</label>
<div>${escHtml(syncedAt)} · commit ${sha} · ${data.file_count} file(s)</div>
</div>
${lastError}
`;
// Download button uses the same analyst-facing endpoint so what the
// admin downloads is byte-identical to what `agnes init` extracts on
// an analyst's laptop. Disabled (rendered as a faded button) when not
// synced — endpoint would return 503. Browser session cookie carries
// auth (get_current_user accepts cookie + Bearer).
const downloadBtn = data.last_commit_sha
? `<a class="cfg-btn" id="iw-download-btn" href="/api/initial-workspace.zip" download="initial-workspace.zip">Download zip</a>`
: `<button class="cfg-btn" disabled title="No synced commit yet — click Sync now first">Download zip</button>`;
actions.innerHTML = `
<button class="cfg-btn primary" id="iw-sync-btn">Sync now</button>
${downloadBtn}
<button class="cfg-btn" id="iw-edit-btn">Edit</button>
<button class="cfg-btn danger" id="iw-delete-btn">Delete</button>
`;
actions.hidden = false;
document.getElementById("iw-sync-btn").addEventListener("click", iwSync);
document.getElementById("iw-edit-btn").addEventListener("click", () => {
iwOpenModal(/* editing */ true, data);
});
document.getElementById("iw-delete-btn").addEventListener("click", iwDelete);
}
function iwOpenModal(editing, data) {
document.getElementById("iw-modal-title").textContent =
editing ? "Edit Template Repository" : "Link to Template Repository";
document.getElementById("iw-url").value = (editing && data) ? (data.url || "") : "";
document.getElementById("iw-branch").value = (editing && data) ? (data.branch || "") : "";
document.getElementById("iw-token").value = ""; // Never prefill PAT
openModal("iw-modal");
}
async function iwSave() {
const url = document.getElementById("iw-url").value.trim();
const branch = document.getElementById("iw-branch").value.trim();
const token = document.getElementById("iw-token").value;
if (!url) {
showBanner("Repository URL is required.", "error");
return;
}
const body = { url };
if (branch) body.branch = branch;
// Only include token when admin typed one — empty string means "leave existing alone"
if (token) body.token = token;
try {
const r = await fetch(IW_API, {
method: "POST", credentials: "include",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(body),
});
const data = await r.json().catch(() => ({}));
if (!r.ok) {
const detail = (data && data.detail) ? (typeof data.detail === "string" ? data.detail : JSON.stringify(data.detail)) : r.statusText;
showBanner("Save failed: " + detail, "error");
return;
}
closeModal("iw-modal");
showBanner("Initial Workspace Template saved. Click 'Sync now' to fetch the repo.", "success");
iwLoad();
} catch (e) {
showBanner("Save failed: " + e.message, "error");
}
}
async function iwSync() {
const btn = document.getElementById("iw-sync-btn");
btn.disabled = true;
btn.textContent = "Syncing…";
try {
const r = await fetch(IW_API + "/sync", {
method: "POST", credentials: "include",
});
const data = await r.json().catch(() => ({}));
if (!r.ok) {
const detail = (data && data.detail) ? data.detail : { kind: "unknown", message: r.statusText };
const msg = (typeof detail === "string")
? detail
: (detail.message || detail.kind || "Unknown error");
const kind = (typeof detail === "object") ? (detail.kind || "") : "";
document.getElementById("iw-sync-body").innerHTML =
`<div class="diff-row" style="color: #b91c1c;"><strong>Sync failed${kind ? " (" + escHtml(kind) + ")" : ""}:</strong><br>${escHtml(msg)}</div>`;
openModal("iw-sync-modal");
return;
}
document.getElementById("iw-sync-body").innerHTML =
`<div class="diff-row"><strong>Action:</strong> ${escHtml(data.action)}</div>
<div class="diff-row"><strong>Commit:</strong> <code>${escHtml(data.commit_sha)}</code></div>
<div class="diff-row"><strong>Files:</strong> ${data.file_count}</div>
<div class="diff-row"><strong>Path:</strong> <code>${escHtml(data.path)}</code></div>`;
openModal("iw-sync-modal");
} catch (e) {
document.getElementById("iw-sync-body").innerHTML =
`<div class="diff-row" style="color: #b91c1c;"><strong>Sync failed:</strong><br>${escHtml(e.message)}</div>`;
openModal("iw-sync-modal");
} finally {
btn.disabled = false;
btn.textContent = "Sync now";
iwLoad();
}
}
async function iwDelete() {
if (!confirm("Remove Initial Workspace Template? This restores the default `agnes init` flow. The on-disk working copy is also wiped.")) {
return;
}
try {
const r = await fetch(IW_API + "?purge=true", {
method: "DELETE", credentials: "include",
});
if (!r.ok) {
const data = await r.json().catch(() => ({}));
showBanner("Delete failed: " + (data.detail || r.statusText), "error");
return;
}
showBanner("Initial Workspace Template removed.", "success");
iwLoad();
} catch (e) {
showBanner("Delete failed: " + e.message, "error");
}
}
document.getElementById("iw-modal-save").addEventListener("click", iwSave);
iwLoad();
</script>
{% endblock %}