agnes-the-ai-analyst/app/api/my_stack.py
Vojtech d6ad08f107
Flea-market upload guardrails + soft delete + JOIN-based admin queue (#233)
* feat(store): flea-market upload guardrails + soft delete + JOIN-based admin queue

Adds an end-to-end guardrails pipeline for store uploads (manifest +
static-security + LLM review), persists blocked bundles for forensics,
introduces soft-delete (Archive) semantics, consolidates the legacy
/store/{id} surface into /marketplace/flea/{id}, and reworks the admin
queue so lifecycle filters read live entity visibility via LEFT JOIN
rather than a denormalized submission column.

Schema v29 → v35:
  * v29 store_submissions table + store_entities.visibility_status
  * v30 file_size, bundle_sha256, bundle_purged_at on submissions
  * v31 reshape store_submissions (drop legacy unique on entity_id)
  * v32 store_entities.archived_at/by + 'archived' visibility value
  * v33 drop store_submissions.retry_count (unused)
  * v34 ensure idx_store_submissions_entity exists post column-drop
  * v35 broaden visibility_status enum + JOIN architecture cutover

Pipeline (src/store_guardrails/):
  * Inline checks: manifest_check, static_scan, quality_check
  * LLM review configurable haiku|sonnet|opus (default haiku)
  * BackgroundTasks-driven async path with structured-output JSON
  * Per-submitter daily quota (default 50)
  * 30-day TTL purge job (POST /api/admin/run-blocked-purge)
  * Bundle SHA256 + size persisted; sha256 survives purge for forensics

Visibility model:
  * pending | approved | hidden | archived
  * _enforce_visibility returns 404 (no leak) for non-owner non-admin
  * Owner sees own non-approved entries via include_owner_id widening
  * Install refused with 409 entity_not_approved when not approved

Soft-delete (DELETE /api/store/entities/{id}):
  * Default = soft (visibility_status='archived'); existing installs
    keep getting served the bundle so users don't lose the plugin
  * ?hard=true admin-only: drops bundle + cascades user_store_installs
  * Hard-delete preserves entity_id on submission as tombstone so
    audit_log linkage survives for the activity timeline

Admin queue lifecycle (the JOIN refactor):
  * Verdict (store_submissions.status) is immutable forensic record
  * Lifecycle (store_entities.visibility_status) is live state
  * /admin/store/submissions Archived chip translates to
    `e.visibility_status='archived'` via LEFT JOIN — any path that
    flips visibility surfaces in the queue immediately
  * Detail page renders Status (verdict) and Entity lifecycle side by
    side so admins see "approved at review, now archived" at a glance

URL consolidation:
  * /store/{id} deleted (no redirect, stale bookmarks 404)
  * /marketplace/flea/{id} is the canonical detail surface
  * Three in-tree callers (upload-success, my-stack card, store
    listing card) updated to point at the new URL
  * Quarantine banner extracted to _quarantine_banner.html partial,
    self-guarded, included from both flea detail templates
  * Banner JS auto-refreshes when the verdict lands by polling
    /api/marketplace/flea/{id}/detail (visibility_status +
    submission_status — the latter is needed because blocked_llm
    keeps the entity at visibility_status='pending')

Audit log resource format:
  * runner.py emits prefixed `store_submission:{id}` (post-fix)
  * Detail-page timeline query handles three patterns: prefixed
    submission, helper-emitted `store_entity:{sub_id}`, and bare-id
    legacy rows — all surface in the activity timeline

UX fixes:
  * Owner sees Under review / Quarantined / Hidden banner with status
  * Install button gray-disabled (not blue) when non-approved
  * Owner cannot delete quarantined entries (403); admin can
  * Admin queue: filter chips, sortable columns, paging, page-size
  * Auto-refresh queue every 5s while pending rows are visible
  * Store upload page file picker no longer opens twice (label →
    input default action collided with explicit JS handler)

Tests: 168 passed across the guardrails suites (admin submissions,
store API, inline / LLM / purge guardrails, store repositories,
marketplace filter, schema version). New regression coverage
includes: archive surfaces via JOIN even when API path is bypassed;
deleted submission renders activity timeline (tombstone); flea
detail surfaces submission_status only for owner/admin; detail page
renders Entity lifecycle row; audit log resource format covers both
helper and runner paths.

* fix(store-guardrails): PR #233 follow-up — prompt injection, atomic PUT, BG race, schema, reaper, sort whitelist

Addresses 9 of the 23 findings from the PR #233 review (spec at
docs/superpowers/specs/2026-05-09-pr233-guardrails-fixes-spec.md).
Merge-gate items #1-#6 plus high-value mediums #7, #9-#12, #23.
Architectural items (#8 enum split, #14 factory) and pure
maintainability (#15-#22) deferred to follow-ups.

Security:
* #1 prompt injection — SYSTEM_PROMPT now passed via the SDK's
  dedicated system= parameter; bundle wrapped in <bundle>...</bundle>
  sentinels declared data-only by the system prompt; literal
  sentinel strings in user content are escaped so an adversarial
  README can't forge a close tag.
* #6 static scan honesty — module docstring + admin copy + docs
  declare static scan as signal not gate; .md/.txt/.rst/.html/.json/
  .yaml/.yml/.toml skipped to avoid false positives on prose.
  AST mode for Python deferred (separate flag, FP comparison work).

Correctness:
* #2 PUT atomicity — bundles bake into plugin.staging-<rand>/
  alongside live, atomic-rename on success; failed checks leave
  live tree byte-for-byte intact.
* #3 BG-task race — set_visibility_if_pending guards verdict flips
  to the (pending, hidden) review window; admin archives during
  review survive; skipped flips audit-logged.
* #4 v35 NOT NULL/DEFAULT — schema v35→v36 re-applies them on
  store_entities.visibility_status. CHECK constraint enforced
  application-side (DuckDB ADD CHECK on existing column unsupported).
* #7 stuck-review reaper — reap_stuck_llm_reviews flips pending_llm
  rows older than guardrails.stuck_review_grace_seconds (default
  1800) to review_error. Scheduler runs every 15 min via new
  /api/admin/run-reap-stuck-reviews. Set knob to 0 to disable.
* #9 quota counter — count_blocked_for_submitter_since now counts
  blocked_inline + blocked_llm + review_error so a submitter
  triggering only LLM-blocked verdicts is bounded.
* #10 missing risk_level — surfaces as review_error with
  error='missing_risk_level' instead of silently defaulting to
  'medium' (which looked like a model-decided block).
* #11 archived_at clear — set_visibility nulls archived_at +
  archived_by when transitioning out of 'archived' so a future
  read doesn't show stale archive forensics on an approved row.

Maintainability:
* #12 FSM doc comment — accurate insert/transition/lifecycle
  description in src/db.py near store_submissions schema.
* #23 sort-key whitelist — admin queue rejects unknown sort keys
  with 400 invalid_sort_key; substring-replace footgun removed.

Deferred (separate PRs):
* #5 quota race — proper fix requires asyncio.Lock spanning the
  full pipeline; threading.Lock blocks event loop, DuckDB MVCC
  doesn't help. API-level slowapi bounds worst case for now.
* #6 part 3 (AST static scan), #8 (enum split), #13 (import
  bundle docs), #14 (factory consolidation), #15-#22 (maint).

Tests:
* New: tests/test_store_guardrails_prompt_injection.py (corpus +
  trust-boundary invariants), tests/test_store_put_atomic.py,
  tests/test_store_guardrails_reaper.py.
* Extended: test_store_guardrails_llm.py (system param, missing
  risk_level, BG race), test_admin_store_submissions.py (quota
  counter widening, sort whitelist 400), test_store_repositories.py
  (un-archive metadata clear), test_db_schema_version.py (v36).
* Full suite: 3738 passed; 17 pre-existing baseline failures
  unchanged (db migration tests, cli binary rename, catalog export,
  user mgmt v5 backfill — confirmed by stash + rerun on clean tree).
2026-05-09 17:32:53 +04:00

217 lines
7.2 KiB
Python

"""Per-user composition view of the served Claude Code marketplace.
Provides:
* ``GET /api/my-stack`` — combined view
* ``PUT /api/my-stack/curated/{marketplace_id}/{plugin}`` — toggle opt-out
Backs the ``/my-ai-stack`` web page. Both endpoints touch the same caches as
the Store endpoints (ETag invalidation) so any change here propagates to
``/marketplace.zip`` + ``/marketplace.git/`` on the next request.
"""
from __future__ import annotations
import logging
from datetime import datetime
from pathlib import Path
from typing import Any, List, Optional
import duckdb
from fastapi import APIRouter, Depends, HTTPException
from pydantic import BaseModel
from app.auth.dependencies import _get_db, get_current_user
from src.marketplace_filter import resolve_allowed_plugins
from src.repositories.audit import AuditRepository
from src.repositories.store_entities import StoreEntitiesRepository
from src.repositories.user_curated_subscriptions import (
UserCuratedSubscriptionsRepository,
)
from src.repositories.user_store_installs import UserStoreInstallsRepository
from src.store_naming import suffixed_name
logger = logging.getLogger(__name__)
router = APIRouter(prefix="/api/my-stack", tags=["my-stack"])
class CuratedPlugin(BaseModel):
marketplace_id: str
marketplace_slug: str
plugin_name: str
manifest_name: str
description: Optional[str] = None
version: Optional[str] = None
enabled: bool
class StoreInstallEntry(BaseModel):
entity_id: str
type: str
name: str
description: Optional[str] = None
category: Optional[str] = None
version: str
owner_user_id: str
owner_username: str
invocation_name: str
install_count: int
photo_url: Optional[str] = None
installed_at: Optional[str] = None
# v35: surface visibility so my_ai_stack.html can render an
# "Archived by owner" badge on cards whose owner soft-deleted the
# entity. Bundle still serves to existing installs (per
# UserStoreInstallsRepository.list_for_user filter).
visibility_status: Optional[str] = None
class MyStackResponse(BaseModel):
curated: List[CuratedPlugin]
store: List[StoreInstallEntry]
class ToggleRequest(BaseModel):
enabled: bool
class OkResponse(BaseModel):
ok: bool = True
def _to_iso(value: Any) -> Optional[str]:
if value is None:
return None
if isinstance(value, datetime):
return value.isoformat()
return str(value)
def _audit(
conn: duckdb.DuckDBPyConnection,
actor_id: str,
action: str,
target: str,
params: Optional[dict] = None,
) -> None:
try:
AuditRepository(conn).log(
user_id=actor_id, action=action, resource=target, params=params
)
except Exception:
pass
@router.get("", response_model=MyStackResponse)
async def get_my_stack(
user: dict = Depends(get_current_user),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
"""Combined view of admin-curated plugins (with current opt-out state)
and Store entities the caller has installed.
"""
granted = resolve_allowed_plugins(conn, user)
# Model B (v28+): explicit subscriptions decide what's enabled.
# `enabled` mirrors the legacy "not opted_out" UX so the existing toggle
# remains semantically intuitive in /my-ai-stack.
subs = UserCuratedSubscriptionsRepository(conn).subscribed_set(user["id"])
curated: List[CuratedPlugin] = []
for p in granted:
is_subscribed = (p["marketplace_id"], p["original_name"]) in subs
curated.append(
CuratedPlugin(
marketplace_id=p["marketplace_id"],
marketplace_slug=p["marketplace_slug"],
plugin_name=p["original_name"],
manifest_name=p["manifest_name"],
description=p["raw"].get("description"),
version=p.get("version"),
enabled=is_subscribed,
)
)
installs = UserStoreInstallsRepository(conn).list_for_user(user["id"])
store_items: List[StoreInstallEntry] = []
from src.store_naming import strip_archive_suffix
for row in installs:
photo_url = (
f"/api/store/entities/{row['id']}/photo" if row.get("photo_path") else None
)
# Display name strips the archive-rename suffix so the user
# sees their installed plugin's original label even after the
# owner archived (and renamed) it. The served ``invocation_name``
# carries the renamed slug since that's what Claude Code's
# `/plugin` lookup will resolve to after the next sync — this
# is the consumer-side rename described in the rename-on-
# archive plan; the My AI Stack card surfaces it via the
# "Archived by owner" badge already.
raw_name = row["name"]
display_name = strip_archive_suffix(raw_name)
store_items.append(
StoreInstallEntry(
entity_id=row["id"],
type=row["type"],
name=display_name,
description=row.get("description"),
category=row.get("category"),
version=row["version"],
owner_user_id=row["owner_user_id"],
owner_username=row["owner_username"],
invocation_name=suffixed_name(raw_name, row["owner_username"]),
install_count=int(row.get("install_count") or 0),
photo_url=photo_url,
installed_at=_to_iso(row.get("installed_at")),
visibility_status=row.get("visibility_status") or "approved",
)
)
return MyStackResponse(curated=curated, store=store_items)
@router.put(
"/curated/{marketplace_id}/{plugin_name}",
response_model=OkResponse,
)
async def toggle_curated(
marketplace_id: str,
plugin_name: str,
body: ToggleRequest,
user: dict = Depends(get_current_user),
conn: duckdb.DuckDBPyConnection = Depends(_get_db),
):
"""Toggle subscribe/unsubscribe for a single admin-granted plugin.
UI thinks in terms of *enabled* (default off in Model B). v28+ the
repository stores *subscribed* rows (presence = enabled in served set);
``enabled=true`` writes a row, ``enabled=false`` removes it.
"""
# Sanity: caller must actually have the plugin granted (otherwise the
# toggle is meaningless and would just leak rows for ungranted plugins).
granted = resolve_allowed_plugins(conn, user)
has_grant = any(
p["marketplace_id"] == marketplace_id and p["original_name"] == plugin_name
for p in granted
)
if not has_grant:
raise HTTPException(status_code=404, detail="grant_not_found")
repo = UserCuratedSubscriptionsRepository(conn)
if body.enabled:
repo.subscribe(user["id"], marketplace_id, plugin_name)
else:
repo.unsubscribe(user["id"], marketplace_id, plugin_name)
_audit(
conn,
user["id"],
"my_stack.curated.toggle",
f"plugin:{marketplace_id}/{plugin_name}",
{"enabled": body.enabled},
)
try:
from app.marketplace_server import packager
packager.invalidate_etag_cache()
except Exception:
logger.exception("failed to invalidate marketplace etag cache")
return OkResponse()