agnes-the-ai-analyst/app/web/templates/_quarantine_banner.html
Vojtech a694a30a5e
fix(store): surface review failures + harden publish gate (#316)
* fix(store): surface review failures + harden publish gate

Four independent fixes to the flea-market submission pipeline, all surfaced
by an admin upload that landed at status='approved' without an LLM review.

1. LLM truncation no longer pins submissions in review_error.
   - Raised MAX_RESPONSE_TOKENS 2500 → 6000 in llm_review.py
   - Added one-shot retry-with-doubled-budget in anthropic_provider.py
     (capped at 4× initial)

2. Flea detail page surfaces the latest submission's failure verdict even
   when a previously-approved version is still serving (deferred-promotion
   path). The _quarantine_banner gate widened from `visibility != approved`
   to also fire on `blocked_inline / blocked_llm / review_error`, with copy
   that distinguishes the v2+ edit case ("Latest edit failed review —
   previously approved version (vN) keeps serving") from the initial-upload
   quarantine wording.

3. Restore button + endpoint no longer allow restoring a version that was
   never approved. Added StoreEntitiesRepository.get_with_version_approvals
   joining store_submissions, gated the UI button on submission_status in
   ('approved', None), rendered status pills for non-restorable rows, and
   added a 400 version_not_approved guard in POST /restore.

4. **BREAKING (operator-facing)**: publish gate is now fail-CLOSED on
   misconfig. The previous get_guardrails_enabled() silently fell back to
   "disabled, auto-approve everything" when guardrails.enabled=true in YAML
   but no ANTHROPIC_API_KEY was in env. Split into:
     - get_guardrails_enabled()              (intent — YAML)
     - get_guardrails_llm_provider_ready()   (readiness — env)
   Three-state matrix:
     enabled=false                       → auto-approve (unchanged)
     enabled=true + ready=true           → normal pipeline (unchanged)
     enabled=true + ready=false (NEW)    → submissions hold at pending_llm
                                           awaiting admin retry or override
                                           (was: silent auto-approve)
   Admin "Retry review" eligibility broadened to include pending_llm.
   Boot-time WARNING banner surfaces the misconfig in app/main.py.
   docs/STORE_GUARDRAILS.md updated with the three-state matrix.
   Operators relying on the auto-fallback for local-dev no-LLM setups must
   now explicitly set `guardrails.enabled: false` in instance.yaml.

Tests: 4623 passed. Added TestPublishGateFailClosed (4 tests) and
TestRestoreVersion::test_restore_rejects_* (3 tests). conftest.py adds an
autouse fixture defaulting guardrails OFF so legacy tests don't need to
know about the new toggle.

* fix(store): admin override promotes v2+ edits to current

The override handler at app/api/admin.py:3708 only flipped submission
status → 'overridden' and entity visibility → 'approved'. Under the v37+
deferred-promotion model that's insufficient for v2+ edits / restores:
the new bundle sits in versions/v<N>/plugin/ and the entity row stays at
the prior approved version_no + hash + on-disk live bundle. Installers
kept getting the OLD bytes the admin had just intended to replace.

Mirror the runner.run_llm_review auto-approval branch: look up the
submission's version_hash in entity.version_history, and if its `n`
differs from entity.version_no, promote_version + _swap_live_to_version.
Initial v1 overrides are unaffected — the loop finds n=1 == version_no
and skips promotion.

Tests:
- test_override_v2_edit_promotes_to_current: stage v1 approved + v2
  blocked_llm; override the v2 sub; assert entity.version_no=2,
  entity.version flips off the v1 hash, and the live plugin/ dir
  mirrors versions/v2/plugin/.
- test_override_v1_initial_upload_no_promote: regression guard so the
  promote loop doesn't accidentally bump a v1 override.

Audit log gains a promoted_to_version_no field on the override action.

* fix(store): retry/rescan review staged bundle; override forward-only

Two adversarial-review findings from a Codex pass on the publish-gate
work.

C1. Admin retry + rescan were passing live `plugin/` to the LLM. For a
v2+ submission held at `pending_llm` / `blocked_llm` / `review_error`,
live still holds the prior approved version's bytes — so the LLM
reviewed the WRONG bytes, and the runner's hash-match promotion in
`run_llm_review` would then advance the entity to staged bytes that
were never actually reviewed. Resolve the staged
`<entity>/versions/v<N>/plugin/` from the submission's
`version_history` entry, with a fall-back to live for legacy pre-v37
rows that never seeded a versions/ dir. Helpers
`_submission_plugin_dir` and `_version_no_for_submission` added to
`app/api/store.py` so override / retry / rescan share one path.

H1. Override's promote loop used `target != current`, which would
silently demote the live bundle when admin overrode a stale v2
submission while v3 was already approved + live. Changed to
`target > current` so override flips status + visibility on the row
regardless, but on-disk promotion only fires forward. Same `>`
defensive guard applied in `runner.run_llm_review` so a late LLM
verdict racing with a newer approval can't demote either.

Tests:
- TestAdminRetryReviewsStagedBundle::test_retry_v2_blocked_passes_staged_dir_not_live
- TestAdminRetryReviewsStagedBundle::test_rescan_v2_blocked_passes_staged_dir_not_live
- TestOverrideForwardOnly::test_override_stale_v2_does_not_demote_when_v3_current

* review polish: CHANGELOG drift, override eligibility, defensive copy

Three small additions on top of the retry/rescan staged-bundle fix:

1. CHANGELOG: the PR's bullets had drifted into the released
   [0.54.17] section during rebase (context-match landed them next
   to already-released content). Moved them up to [Unreleased] where
   they belong; [0.54.17] now holds only what was actually released
   (refresh-marketplace ls-remote, /me/activity hero, CI sharding +
   workflow polish).

2. app/api/admin.py: admin override eligibility now accepts
   pending_llm alongside blocked_inline + blocked_llm + review_error.
   Closes a UX gap from the new fail-CLOSED behavior: under
   enabled-but-not-ready, a known-good submission would otherwise
   sit indefinitely until the admin set credentials AND clicked
   Retry. Override already routes through version_history (and is
   now forward-only on promote), so it stays safe for v2+ deferred-
   promotion submissions.

3. src/repositories/store_entities.py: get_with_version_approvals
   defensively copies each version_history entry before annotating
   with submission_status. self.get() re-parses JSON each call today
   so this is belt-and-suspenders against any future caching layer
   leaking the annotated key into a subsequent plain get() call.

Tests: 112 passed (focused on test_store_entity_versions +
test_admin_store_submissions, covering the retry/rescan staged-
bundle fix the author shipped + this polish).

---------

Co-authored-by: ZdenekSrotyr <zdenek.srotyr@keboola.com>
2026-05-15 15:52:07 +02:00

329 lines
13 KiB
HTML

{# Shared quarantine banner partial.
Surfaces submission status (under review / quarantined / hidden /
override-applied) to the entity owner + admins. Self-guarded so it's
safe to {% include %} from any detail page — renders nothing when
the entity is approved or the viewer isn't owner/admin.
Required scope:
entity — store_entities row (must carry visibility_status,
visibility_status; entity.id surfaces in admin
detail link)
quarantine_sub — latest store_submissions row for entity, or None
is_owner — bool, viewer == entity.owner_user_id
is_admin — bool, viewer is in Admin group
Mirror of the version that previously lived in store_detail.html.
Wording stays consistent with the per-status messaging the user
approved earlier — only the rendering location changed.
#}
{# Gate widened for failure surfacing: under deferred promotion
(v37+), a v2+ edit can leave the entity approved at the prior
version while the latest submission landed in `review_error` /
`blocked_llm` / `blocked_inline`. The original
`visibility_status != 'approved'` gate silently hid those
failures from the owner. Render the banner whenever EITHER the
entity itself is non-approved OR the latest submission carries
a *failure* verdict the owner needs to see. Pending edits keep
the original behavior — Edit button locks instead, no banner. #}
{% if (is_owner or is_admin) and (
entity.visibility_status != 'approved'
or (quarantine_sub and quarantine_sub.status in [
'blocked_inline', 'blocked_llm', 'review_error',
])
) %}
<style>
.vis-banner {
margin: 12px 0 16px 0;
padding: 14px 18px;
border-radius: 10px;
font-size: 14px;
border: 1px solid;
}
.vis-banner.pending { background: #fef3c7; color: #92400e; border-color: #fde68a; }
.vis-banner.blocked { background: #fee2e2; color: #991b1b; border-color: #fecaca; }
.vis-banner.hidden { background: #e5e7eb; color: #374151; border-color: #d1d5db; }
.vis-banner h3 { margin: 0 0 6px 0; font-size: 15px; font-weight: 600; }
.vis-banner ul { margin: 6px 0 0 0; padding-left: 20px; font-size: 13px; }
.vis-banner code { background: rgba(0,0,0,0.06); padding: 1px 6px; border-radius: 4px; font-size: 12px; }
.vis-banner .actions { margin-top: 10px; }
.vis-banner .actions a {
display: inline-block; padding: 5px 12px; border-radius: 6px;
background: rgba(0,0,0,0.08); color: inherit; text-decoration: none;
font-size: 12px; font-weight: 500;
}
</style>
{% set sub = quarantine_sub %}
{% set st = sub.status if sub else entity.visibility_status %}
{% set bcls = 'pending' if st in ['pending_inline','pending_llm','pending']
else ('blocked' if st in ['blocked_inline','blocked_llm','review_error']
else 'hidden') %}
<div class="vis-banner {{ bcls }}">
{% if st == 'pending_llm' or st == 'pending_inline' or st == 'pending' %}
{% set _is_edit_review = entity.version_no and entity.version_no > 1 %}
{% if _is_edit_review %}
<h3>⟳ Version {{ entity.version_no }} under review</h3>
<div>
Your edit is being checked. The previously approved version
(v{{ entity.version_no - 1 }}) keeps serving to existing
installers until v{{ entity.version_no }} passes review. The
page refreshes automatically when the verdict lands.
</div>
{% else %}
<h3>⟳ Under review</h3>
<div>
Your submission is being checked. It is hidden from the public
Store and from anyone else's view until all checks pass. Page
refreshes automatically when the verdict lands — usually a few
seconds.
</div>
{% endif %}
{% elif st == 'blocked_inline' %}
{% set _is_edit_review = entity.visibility_status == 'approved' %}
{% if _is_edit_review %}
<h3>⚠ Latest edit failed automated checks</h3>
<div>
Your latest edit failed at least one automated check. The
previously approved version (v{{ entity.version_no }}) keeps
serving to existing installers. Fix the issues below and
re-upload, or wait for an admin to resolve the quarantine.
</div>
{% else %}
<h3>⚠ Quarantined — automated checks failed</h3>
<div>
Your submission failed at least one automated check and has been
quarantined. It is hidden from the public Store and from every
other user; nobody can install it. Fix the issues below and
re-upload to retry, or wait for an admin to resolve the
quarantine.
</div>
{% endif %}
{% if sub and sub.inline_checks %}
{% set ic = sub.inline_checks %}
{% if ic.manifest and ic.manifest.issues %}
<ul>
{% for issue in ic.manifest.issues %}<li>manifest: <code>{{ issue }}</code></li>{% endfor %}
</ul>
{% endif %}
{% if ic.static_security and ic.static_security.findings %}
<ul>
{% for f in ic.static_security.findings[:6] %}
<li>security: <code>{{ f.file }}:{{ f.line }}</code> — {{ f.reason }}</li>
{% endfor %}
{% if ic.static_security.findings|length > 6 %}
<li><em>… and {{ ic.static_security.findings|length - 6 }} more</em></li>
{% endif %}
</ul>
{% endif %}
{% if ic.content and ic.content.issues %}
{% include "_content_findings.html" with context %}
{% endif %}
{% endif %}
{% elif st == 'blocked_llm' %}
{% set _is_edit_review = entity.visibility_status == 'approved' %}
{% if _is_edit_review %}
<h3>⚠ Latest edit failed review</h3>
<div>
The reviewer flagged your latest edit for security risk and/or
weak component descriptions. The previously approved version
(v{{ entity.version_no }}) keeps serving to existing installers.
Address the findings below and re-upload, or wait for an admin
to resolve the quarantine.
</div>
{% else %}
<h3>⚠ Quarantined — review flagged issues</h3>
<div>
The reviewer flagged this submission for security risk and/or
weak component descriptions. It is hidden from the public Store
and from every other user; nobody can install it. Address the
findings below and re-upload, or wait for an admin to resolve
the quarantine.
</div>
{% endif %}
{% if sub and sub.llm_findings %}
{% if sub.llm_findings.summary %}
<div style="margin-top: 6px;"><em>{{ sub.llm_findings.summary }}</em></div>
{% endif %}
{% if sub.llm_findings.findings %}
<div style="margin-top: 8px; font-weight: 600;">Security findings</div>
<ul>
{% for f in sub.llm_findings.findings[:6] %}
<li>[{{ f.severity }}] <code>{{ f.file }}</code> — {{ f.explanation }}</li>
{% endfor %}
</ul>
{% endif %}
{% if sub.llm_findings.content_quality and sub.llm_findings.content_quality.issues %}
<div style="margin-top: 10px; font-weight: 600;">Description quality — reviewer suggestions</div>
<ul>
{% for issue in sub.llm_findings.content_quality.issues[:8] %}
<li>
<code>{{ issue.file }}</code> — {{ issue.issue }}
{% if issue.hint %}
<div style="margin: 4px 0 0 0; font-size: 12px; opacity: 0.85;">
<strong>Rewrite:</strong> {{ issue.hint }}
</div>
{% endif %}
</li>
{% endfor %}
</ul>
{% endif %}
{% endif %}
{% elif st == 'review_error' %}
{% set _is_edit_review = entity.visibility_status == 'approved' %}
{% if _is_edit_review %}
<h3>⚠ Latest edit failed review</h3>
<div>
The security reviewer couldn't complete its check on your latest
edit. The previously approved version (v{{ entity.version_no }})
keeps serving to existing installers. No action needed from you —
an admin will retry.
</div>
{% else %}
<h3>⚠ Under review — security check errored</h3>
<div>
The security reviewer couldn't complete its check. The submission
stays hidden until an admin retries. No action needed from you.
</div>
{% endif %}
{% if sub and sub.llm_findings and sub.llm_findings.error %}
<div style="margin-top: 6px; font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; word-break: break-word;">
Error: {{ sub.llm_findings.error }}
</div>
{% endif %}
{# Surface any inline-check findings that were captured before the
LLM step errored — gives the submitter something concrete to
look at instead of a bare "errored" message. #}
{% if sub and sub.inline_checks %}
{% set ic = sub.inline_checks %}
{% if ic.static_security and ic.static_security.findings %}
<ul>
{% for f in ic.static_security.findings[:6] %}
<li>security: <code>{{ f.file }}:{{ f.line }}</code> — {{ f.reason }}</li>
{% endfor %}
</ul>
{% endif %}
{% endif %}
{% elif st == 'overridden' %}
<h3>✓ Admin override applied</h3>
<div>This submission was force-published by an admin.</div>
{% if sub and sub.override_reason %}
<div style="margin-top: 6px; font-size: 13px;">
<em>Override reason:</em> {{ sub.override_reason }}
</div>
{% endif %}
{% else %}
{# Fallback for hidden / unexpected lifecycle states. Surface
whatever verdict context the submission row carries so an
admin doesn't see a bare "Hidden" with no actionable detail. #}
<h3>Hidden</h3>
<div>
This entity is not visible in the public Store
(<code>visibility_status = "{{ entity.visibility_status }}"</code>).
</div>
{% if sub and sub.inline_checks %}
{% set ic = sub.inline_checks %}
{% if ic.manifest and ic.manifest.issues %}
<ul>
{% for issue in ic.manifest.issues %}<li>manifest: <code>{{ issue }}</code></li>{% endfor %}
</ul>
{% endif %}
{% if ic.static_security and ic.static_security.findings %}
<ul>
{% for f in ic.static_security.findings[:6] %}
<li>security: <code>{{ f.file }}:{{ f.line }}</code> — {{ f.reason }}</li>
{% endfor %}
</ul>
{% endif %}
{% endif %}
{% if sub and sub.llm_findings %}
{% if sub.llm_findings.summary %}
<div style="margin-top: 6px;"><em>{{ sub.llm_findings.summary }}</em></div>
{% endif %}
{% if sub.llm_findings.findings %}
<ul>
{% for f in sub.llm_findings.findings[:6] %}
<li>[{{ f.severity }}] <code>{{ f.file }}</code> — {{ f.explanation }}</li>
{% endfor %}
</ul>
{% endif %}
{% endif %}
{% endif %}
{# How-to-fix panel — render once below the per-tier findings whenever
content-quality issues exist on either tier. Same guidance regardless
of whether the inline mechanical check or the LLM substantive check
rejected the submission. #}
{% if sub and ((sub.inline_checks and sub.inline_checks.content and sub.inline_checks.content.issues)
or (sub.llm_findings and sub.llm_findings.content_quality and sub.llm_findings.content_quality.issues)) %}
{% include "_content_howto_fix.html" with context %}
{% endif %}
{% if is_admin and sub %}
<div class="actions">
<a href="/admin/store/submissions/{{ sub.id }}">Open submission detail →</a>
</div>
{% endif %}
</div>
{# Auto-refresh while the verdict is pending. Banner copy promises
"page refreshes automatically when the verdict lands" — this is
what does it. Polls the owner-accessible flea detail endpoint and
reloads when EITHER visibility flips off 'pending' OR the
submission verdict flips off 'pending_inline' / 'pending_llm'.
Both signals are needed because `blocked_llm` keeps the entity at
`visibility_status='pending'` (admin can override → publish), so
visibility alone doesn't fire. Only emits the script while the
verdict itself is still pending; terminal states render the
final banner copy and don't need to reload. #}
{% if quarantine_sub and quarantine_sub.status in ['pending_inline', 'pending_llm'] %}
<script>
(function () {
const entityId = {{ entity.id|tojson }};
const initialSubStatus = {{ quarantine_sub.status|tojson }};
const initialVisibility = {{ entity.visibility_status|tojson }};
let attempts = 0;
async function tick() {
attempts++;
try {
const r = await fetch(`/api/marketplace/flea/${entityId}/detail`, {
credentials: 'same-origin',
headers: {'Accept': 'application/json'},
});
if (r.ok) {
const data = await r.json();
const subFlipped = data.submission_status
&& data.submission_status !== initialSubStatus
&& data.submission_status !== 'pending_inline'
&& data.submission_status !== 'pending_llm';
const visFlipped = data.visibility_status
&& data.visibility_status !== initialVisibility;
if (subFlipped || visFlipped) {
window.location.reload();
return;
}
} else if (r.status === 404) {
// Entity might have been archived/deleted — reload so the
// page refetches and renders the new state (or a 404).
window.location.reload();
return;
}
} catch (e) { /* network blip; keep polling */ }
// First 30 attempts at 3s = 90s of fast polling, then back off
// to 10s. Same cadence as admin detail polling so an LLM review
// on Sonnet/Opus has room to land.
const next = attempts < 30 ? 3000 : 10000;
setTimeout(tick, next);
}
setTimeout(tick, 3000);
})();
</script>
{% endif %}
{% endif %}