agnes-the-ai-analyst/app/web/templates/profile_sessions.html
minasarustamyan e26236fdc1
Extract session-pipeline framework + UsageProcessor skeleton (#232)
* Extract session pipeline framework, refactor verification, add UsageProcessor skeleton

Pluggable framework under services/session_pipeline/ (contract + lib + per-processor
runner) so multiple processors can read /data/user_sessions/<key>/*.jsonl on their
own cadence with full failure isolation. Verification flow becomes the first plugin;
a no-op UsageProcessor reserves the second slot pending a separate brainstorm on
extraction logic + storage shape.

Schema v28→v29: rename session_extraction_state → session_processor_state with
composite PK (processor_name, session_file). Existing rows copied over with
processor_name='verification'; legacy table dropped. Migration is idempotent and
no-ops the copy step on fresh installs that came up at the new schema.

Endpoint: /api/admin/run-verification-detector replaced by parametrized
/api/admin/run-session-processor?processor=<name>. Audit action format follows.
Scheduler JOBS: verification-detector entry split into session-processor:verification
+ session-processor:usage. SCHEDULER_VERIFICATION_DETECTOR_INTERVAL retained for
operator compatibility (drives both cadence and health-check grace window);
SCHEDULER_USAGE_PROCESSOR_INTERVAL added.

* Address PR #232 review: scan dead branch + per-processor lock

- `SessionProcessorStateRepository.scan_unprocessed_for` dead else: both
  branches surfaced every jsonl, the SELECT was unused, runner MD5-rehashed
  every stable session per tick. Replaced with an mtime precheck — stable
  sessions (mtime <= processed_at) are filtered at scan; modified files
  still surface for the runner's authoritative `file_hash` invalidation.
  Naive-local comparison matches the existing health-check idiom (DuckDB
  TIMESTAMP strips tz on storage).

- Per-processor advisory lock around `_run_processor` in
  `/api/admin/run-session-processor`. Scheduler tick + manual admin POST
  could otherwise both run, both call create_evidence on overlapping
  detections, and accumulate duplicate verification_evidence rows (the
  dedup short-circuit only covers create+contradiction, not evidence per
  ADR Decision 3). Non-blocking acquire → 409 Conflict on concurrent
  invocation; release in finally so a runner exception doesn't wedge the
  processor.

Tests: two new scan unit tests (mtime filter + post-mark mtime bump), 409
endpoint test, lock-released-on-exception test. Two existing tests updated
for the new "filtered at scan" stat shape (previously asserted skipped == 1,
now scanned == 0).

* Address PR #232 review #2: parallel scheduler tick + last_run on terminal state

Two pre-existing scaffold bugs in services/scheduler/__main__.py amplified
by adding more session-pipeline jobs:

1. Serial for-loop over jobs with synchronous httpx.post(timeout=900) — a
   10-minute verification run blocked every other job (data-refresh,
   health-check, usage, corporate-memory) for the whole window. The PR's
   stated isolation guarantee held inside the runner but broke at the
   scheduler dispatch layer.

2. last_run advanced only when _call_api returned True. Permanent-failure
   jobs hot-looped on every tick (30s) instead of cadence (15min).

Fix: ThreadPoolExecutor.submit per due job + per-job in_flight set so a
long-running job can't be re-launched on subsequent ticks. last_run
advances unconditionally in finally; errors still surface via _call_api
logging + audit_log on the receiving side.

_run_job extracted to module-level for unit testing. New tests:
- TestRunJobBookkeeping: advances on success / failure / unhandled raise
- TestRunLoopParallelism: in_flight protection prevents duplicate
  launches across ticks for a single slow job

---------

Co-authored-by: Minas Arustamyan <arustamyan.minas@gmail.com>
2026-05-08 19:47:46 +02:00

128 lines
4.9 KiB
HTML

{% extends "base.html" %}
{% block title %}My sessions — {{ config.INSTANCE_NAME }}{% endblock %}
{% block content %}
<style>
.container:has(.sess-page) { max-width: none; padding: 24px 16px; }
.sess-page { max-width: 1200px; margin: 0 auto; padding: 0; }
.sess-title { margin: 0 0 8px 0; font-size: 22px; font-weight: 600; }
.sess-help { color: var(--text-secondary, #6b7280); font-size: 13px; margin-bottom: 20px; line-height: 1.55; }
.sess-help code { background: var(--border-light, #f3f4f6); padding: 1px 6px; border-radius: 4px; font-size: 12px; }
.sess-table-wrap {
background: var(--surface, #fff);
border: 1px solid var(--border, #e5e7eb);
border-radius: 12px;
overflow-x: auto;
}
.sess-table { width: 100%; border-collapse: collapse; font-size: 13px; }
.sess-table thead th {
text-align: left; padding: 12px 16px;
background: var(--border-light, #f9fafb);
border-bottom: 1px solid var(--border, #e5e7eb);
font-weight: 600; color: var(--text-secondary, #6b7280);
font-size: 11px; text-transform: uppercase; letter-spacing: 0.4px;
white-space: nowrap;
}
.sess-table tbody td {
padding: 10px 16px;
border-bottom: 1px solid var(--border-light, #f3f4f6);
vertical-align: middle;
}
.sess-table tbody tr:last-child td { border-bottom: none; }
.sess-table tbody tr:hover { background: var(--border-light, #fafafa); }
.sess-table .name { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: 12px; }
.sess-table .ts { white-space: nowrap; color: var(--text-secondary, #6b7280); font-variant-numeric: tabular-nums; }
.sess-table .num { text-align: right; font-variant-numeric: tabular-nums; }
.badge {
display: inline-block; padding: 2px 8px; border-radius: 999px;
font-size: 11px; font-weight: 500;
}
.badge-pending { background: #fef3c7; color: #92400e; }
.badge-processed { background: #dbeafe; color: #1e40af; }
.badge-extracted { background: #d1fae5; color: #065f46; }
.dl-link {
display: inline-block;
padding: 4px 10px;
border: 1px solid var(--border, #e5e7eb);
border-radius: 6px;
color: inherit; text-decoration: none;
font-size: 12px; font-weight: 500;
background: var(--surface, #fff);
transition: background 0.15s, border-color 0.15s;
}
.dl-link:hover {
background: var(--border-light, #f9fafb);
border-color: var(--primary, #6366f1);
color: var(--primary, #4338ca);
}
.empty {
padding: 40px 16px; text-align: center;
color: var(--text-secondary, #6b7280); font-size: 13px;
}
</style>
<div class="sess-page">
<h1 class="sess-title">My sessions</h1>
<p class="sess-help">
Sessions you uploaded via <code>agnes push</code> from your Claude Code workspace, with
extraction status from the verification processor's rows in <code>session_processor_state</code>.
<br>
<strong>Items extracted = 0</strong> means the verification detector ran successfully
but the LLM didn't find anything worth tracking in that session — that's expected for
sessions that are mostly tool calls or coding without confident factual claims.
<br>
<em>Pending</em> means the file is on disk but the scheduler hasn't processed it yet
(next verification-detector tick: every 15 min by default).
</p>
<div class="sess-table-wrap">
{% if sessions %}
<table class="sess-table">
<thead>
<tr>
<th>Session file</th>
<th>Uploaded</th>
<th class="num">Size</th>
<th>Status</th>
<th>Processed</th>
<th class="num">Items extracted</th>
<th></th>
</tr>
</thead>
<tbody>
{% for s in sessions %}
<tr>
<td class="name">{{ s.name }}</td>
<td class="ts">{{ s.uploaded_at.strftime("%Y-%m-%d %H:%M:%S UTC") }}</td>
<td class="num">{{ s.size_kb }} kB</td>
<td>
{% if not s.is_processed %}
<span class="badge badge-pending">pending</span>
{% elif s.items_extracted and s.items_extracted > 0 %}
<span class="badge badge-extracted">extracted</span>
{% else %}
<span class="badge badge-processed">processed</span>
{% endif %}
</td>
<td class="ts">
{% if s.processed_at %}{{ s.processed_at.strftime("%Y-%m-%d %H:%M:%S UTC") }}{% endif %}
</td>
<td class="num">
{% if s.items_extracted is not none %}{{ s.items_extracted }}{% endif %}
</td>
<td>
<a class="dl-link" href="/profile/sessions/{{ s.name }}" download>Download</a>
</td>
</tr>
{% endfor %}
</tbody>
</table>
{% else %}
<div class="empty">
No session files yet. Run <code>agnes push</code> from a Claude Code workspace
where <code>agnes init</code> installed the SessionEnd hook.
</div>
{% endif %}
</div>
</div>
{% endblock %}