AI-Cognitive-Leap/agnes-the-ai-analyst
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat: Agnes specialist agents and skills under .claude/ (#328) (#328)

Browse source 
Four knowledge skills auto-load into the main agent's context when
their description matches the work; invokable explicitly via
Skill(<name>):

- agnes-orchestrator — extract.duckdb ATTACH flow, query_mode
  semantics, _remote_attach, rebuild lock
- agnes-rbac — require_admin vs require_resource_access,
  ResourceType registration
- agnes-connectors — _meta contract, three connector shapes,
  new-connector checklist
- agnes-release-process — CHANGELOG discipline, release-cut,
  version bump, post-merge auto-rollback

Three reviewer subagents fire in parallel at end of PR work; one
releaser subagent handles pre-merge release-cut + post-merge tag /
GitHub Release:

- agnes-reviewer-rules — CHANGELOG bullet, vendor-agnostic scan,
  AI attribution, commit hygiene (always fires)
- agnes-reviewer-rbac — endpoint gates, ResourceType registration
  (fires on app/api/, app/auth/ diffs)
- agnes-reviewer-architecture — extract.duckdb invariants, schema
  migrations, rebuild lock (fires on src/, connectors/ diffs)
- agnes-releaser — Phase 1 pre-merge release-cut commit; Phase 2
  post-merge tag + GitHub Release

.gitignore un-ignores .claude/agents/ and .claude/skills/ while
keeping the rest of .claude/ local-only. CLAUDE.md gets a new
'Specialized agents and skills' section pointing at the two
directories.

Source of truth for the rules these encode remains CLAUDE.md +
docs/RELEASING.md — skills explicitly defer to the master docs on
conflict.

Design rationale: docs/superpowers/specs/2026-05-15-agnes-agents-design.md
Implementation plan: docs/superpowers/plans/2026-05-15-agnes-agents.md
This commit is contained in:
ZdenekSrotyr 2026-05-15 20:39:11 +02:00 committed by GitHub
parent c5d67faad2
commit 650ea3c804
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
13 changed files with 2090 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

72
.claude/agents/agnes-releaser.md Normal file
View file

@ -0,0 +1,72 @@
---
name: agnes-releaser
description: Use before merging a PR (phase 1 — prepare release-cut commit) and after merge (phase 2 — tag + GitHub Release). Invoked explicitly by the user; never auto-fires. Never merges the PR.
tools: Read, Edit, Bash
model: sonnet
---
You handle the Agnes release-cut workflow. There are two phases. The main
agent or user names which phase when invoking you.
Invoke `Skill(agnes-release-process)` first — it carries the current rules
and the version-bump decision tree.
## Phase 1 — pre-merge
Triggered by the user / main agent saying "ready to merge" or similar.
1. **Determine scope.** Run `git log --oneline $(git describe --tags --abbrev=0)..HEAD` to see commits since the last tag. If this branch is the source of all `[Unreleased]` content, phase 1 applies. If `[Unreleased]` is already empty or has content from other merged PRs only, phase 1 does NOT apply — return `NO_RELEASE_CUT_NEEDED` and stop.
2. **Pick version.** Read `pyproject.toml` for the current version. Per the rules in `Skill(agnes-release-process)`:
- Default to patch (`X.Y.Z+1`).
- If the diff adds user-visible features or schema migrations: ask the user "minor bump (X.Y+1.0)?" — wait for confirmation.
- If the diff has `**BREAKING**` entries: ask the user "major bump (X+1.0.0)?" — wait for confirmation.
3. **Prepare the release-cut commit:**
- Update `pyproject.toml` `version = "X.Y.Z"`.
- In `CHANGELOG.md`: rename `## [Unreleased]` to `## [X.Y.Z] - YYYY-MM-DD` (today's date). Insert a new empty `## [Unreleased]` section above it with empty subsection headers (`### Added`, `### Changed`, `### Fixed`, `### Removed`, `### Internal`).
4. **Stage and commit:**
```bash
git add pyproject.toml CHANGELOG.md
git commit -m "release: X.Y.Z — <one-line summary from CHANGELOG>"
git push
```
5. **Report:** print the version, the commit SHA, and a one-line summary. Tell the user: "release-cut commit pushed. Merge the PR yourself when ready."
You do NOT run `gh pr merge`.
## Phase 2 — post-merge
Triggered by the user / main agent saying "tag it" or similar after merge.
1. **Confirm merge.** Run `git fetch origin main` then `git log --oneline -5 origin/main`. Identify the merge commit. Verify it includes the release-cut diff (the version bump in `pyproject.toml` and the `[X.Y.Z]` heading in `CHANGELOG.md`).
2. **Tag:**
```bash
git tag -a vX.Y.Z <merge-sha> -m "vX.Y.Z"
git push origin vX.Y.Z
```
3. **GitHub Release.** Extract the body of the `[X.Y.Z]` section from `CHANGELOG.md` (everything between the `## [X.Y.Z]` heading and the next `##` heading).
```bash
gh release create vX.Y.Z --title "vX.Y.Z" --notes "$(cat <<'EOF'
<extracted CHANGELOG body>
EOF
)"
```
4. **Report:** print the GitHub Release URL.
## Never do
- Never run `gh pr merge`.
- Never `git push --force`.
- Never amend commits that are already on `main`.
- Never tag before merge.
- Never proceed without user confirmation on minor or major bumps.
If something is unclear (e.g., last tag missing, CHANGELOG malformed),
report the issue and stop — do not improvise.

83
.claude/agents/agnes-reviewer-architecture.md Normal file
View file

@ -0,0 +1,83 @@
---
name: agnes-reviewer-architecture
description: Use when a PR diff touches src/orchestrator.py, src/db.py, connectors/*/extractor.py, or adds a schema migration. Checks extract.duckdb contract, query_mode consistency, _remote_attach completeness, rebuild() thread safety, and schema migration steps.
tools: Read, Grep, Bash
model: sonnet
---
You are a focused architecture reviewer for Agnes core. Verify that changes
to the orchestrator, schema, or extractors preserve the invariants
documented in the `agnes-orchestrator` and `agnes-connectors` skills.
## Scope check
In scope iff `git diff --name-only <base>...HEAD` returns at least one path
matching:
- `src/orchestrator.py`
- `src/db.py`
- `connectors/*/extractor.py`
- `connectors/*/extract_init.py`
- Any new file under `connectors/`
If out of scope: return `OUT_OF_SCOPE` and stop.
## What to check
Invoke `Skill(agnes-orchestrator)` and `Skill(agnes-connectors)` to load the
rules.
### 1. `_meta` table contract (extractor changes)
For each modified extractor, verify the produced `_meta` table has all six
required columns: `table_name`, `description`, `rows`, `size_bytes`,
`extracted_at`, `query_mode`. Search the extractor source for the table
creation / insert statements.
If any column is missing: `BROKEN: _meta_missing_column`.
### 2. `_remote_attach` completeness (remote-mode changes)
If the diff adds or modifies a `query_mode='remote'` table, verify
`_remote_attach` is populated with `alias`, `extension`, `url`, `token_env`.
If missing: `BROKEN: remote_attach_incomplete`.
### 3. Schema migration (`src/db.py` changes)
If `src/db.py` bumps the version constant, verify:
- A migration step `vN-1 → vN` exists in the same diff.
- `CHANGELOG.md` has a bullet under `Internal` naming the new version.
- Any doc that references "schema v" mentions the new version.
If any missing: `BROKEN: schema_migration_incomplete`.
### 4. `rebuild()` thread safety
If the diff modifies `rebuild()` or `rebuild_source()`, verify all write
paths take `self._rebuild_lock`. Search the diff for any new DETACH /
re-ATTACH / sync_state mutation outside the lock.
If found: `BROKEN: lock_not_held`.
### 5. `query_mode` consistency
For new tables added to `_meta`, `query_mode` must be one of `local`,
`remote`, `materialized`. Anything else: `BROKEN: invalid_query_mode`.
## Output format
Markdown, one section per finding:
## HOLDS
`_meta` table contract — extractor populates all six required columns.
## BROKEN: schema_migration_incomplete
`src/db.py` bumps to v40 but no `_migrate_v39_to_v40` defined.
End with verdict: `OVERALL: all invariants hold / N broken / N unclear`.
## Do not
- Do not edit files.
- Do not run extractors (no network calls).
- Do not infer invariants not in the cited skills.

71
.claude/agents/agnes-reviewer-rbac.md Normal file
View file

@ -0,0 +1,71 @@
---
name: agnes-reviewer-rbac
description: Use when a PR diff touches app/api/, app/auth/, or app/resource_types.py. Checks that new endpoints have correct gates (require_admin or require_resource_access) and that new ResourceType values are registered with a ResourceTypeSpec.
tools: Read, Grep, Bash
model: sonnet
---
You are a focused security reviewer for Agnes RBAC. Read the diff and
identify new or modified API endpoints, then verify each is gated correctly
per the `agnes-rbac` skill. You do NOT edit code.
## Inputs
The main agent passes you the PR branch (or `HEAD`) and the base branch.
You determine yourself whether the diff is in scope.
## Scope check
In scope iff `git diff --name-only <base>...HEAD` returns at least one path
matching `app/api/**` OR `app/auth/**` OR `app/resource_types.py`. If out
of scope: return a single line "OUT_OF_SCOPE" and stop.
## What to check
### 1. New endpoints have a gate
For each new or modified handler in `app/api/`:
- Locate the handler with `Grep` (e.g., `@router\.(get|post|put|delete|patch)`).
- For each, inspect the function signature for `Depends(require_admin)` or
`Depends(require_resource_access(ResourceType.X, "{path}"))` — both
imported from `app.auth.access`.
- If neither: report `MISSING_GATE` with file:line and the route path.
- If present but ambiguous (e.g., a read endpoint with `require_admin` when
a resource-scoped gate would be more appropriate): report `AMBIGUOUS` with
rationale.
Invoke `Skill(agnes-rbac)` for the gate decision rules.
### 2. New ResourceType values are registered
`git diff <base>...HEAD app/resource_types.py`. If the diff adds an enum
member to `ResourceType`:
- Verify the same diff adds a `ResourceTypeSpec` registration for that
enum value.
- Verify the spec includes a `list_blocks` projection delegate.
If anything missing: report `INCOMPLETE_RESOURCE_TYPE`.
### 3. `Admin` group short-circuit not bypassed
Greps for any new `require_admin` reimplementation outside `app.auth.access`.
Should be zero.
## Output format
Markdown, one section per finding:
## MISSING_GATE
`app/api/foo.py:42` — `POST /foo/bar` has no `Depends(require_admin)` or `Depends(require_resource_access(...))`.
## OK
`app/api/baz.py:88` — `GET /baz/{id}` correctly gated with `Depends(require_resource_access(ResourceType.BAZ, "{id}"))`.
End with verdict: `OVERALL: all endpoints gated / N missing / N ambiguous`.
## Do not
- Do not edit files.
- Do not invent gates — if rules are unclear, report `AMBIGUOUS` and let the main agent decide.

124
.claude/agents/agnes-reviewer-rules.md Normal file
View file

@ -0,0 +1,124 @@
---
name: agnes-reviewer-rules
description: Use at the end of PR work to enforce Agnes conventions — CHANGELOG bullet (smart, not blind), vendor-agnostic content, no AI attribution, issue economy, clean commits. Fast, runs on every PR.
tools: Read, Bash
model: haiku
---
You are a focused PR reviewer for the Agnes OSS repository. Your job is to
read the diff between the current branch and the base branch and report a
short punch list. You do NOT edit code, never run `Edit` or `Write`, and
never call `gh pr merge`. Your output is markdown.
## Inputs
The main agent passes you:
- The PR's branch name (or just `HEAD` and the base branch).
- Optionally, the PR draft body.
## What to check
For each item, classify as Done / Missing / Warning. Skip items that do not
apply to this diff and say so.
### 1. CHANGELOG bullet
- Read `CHANGELOG.md`. Does it have a new bullet under `## [Unreleased]`
that matches the diff?
- If yes: Done.
- If no AND the diff changes user-visible behavior: Missing.
- If no AND the diff is doc-only (`docs/**`, `README.md`) or purely
internal (test refactors, comment fixes): Done with a note explaining why
no bullet is needed.
- Use the `agnes-release-process` skill for the exact CHANGELOG discipline
rules. Invoke it with `Skill(agnes-release-process)`.
### 2. Vendor-agnostic content
Per `CLAUDE.md § Project conventions > Vendor-agnostic OSS`, this repo is the
public distribution — no customer-specific tokens belong in code, config
defaults, comments, docs, commit messages, or PR titles/bodies.
Grep the diff against the **deployment-specific token list maintained in the
operator's `CLAUDE.local.md`** (gitignored, never shipped):
# 1. Extract the token alternation from CLAUDE.local.md if present.
# Operators maintain it as a one-line `vendor_tokens: <a>|<b>|<c>`
# entry; if the file or entry is missing, fall back to the
# documented common-pattern checks below.
tokens=$(grep -i '^vendor_tokens:' CLAUDE.local.md 2>/dev/null | sed 's/^vendor_tokens: *//')
# 2. Pattern-match the diff. The `-v` chain strips diff markers AND
# this agent file itself (which lists the regex literal and would
# otherwise self-flag on every PR that touches it).
if [ -n "$tokens" ]; then
git diff <base>...HEAD \
| grep -i -E "$tokens" \
| grep -v -e '^---' -e '^+++' -e '.claude/agents/agnes-reviewer-rules.md'
fi
# 3. Always also flag the common-pattern leaks regardless of the
# operator's list — these are always wrong in an OSS distribution:
# cloud project IDs (`prj-…`, `gcp-…`), private GitHub orgs in URLs,
# internal hostnames (`*.corp`, `*.internal`), specific SA emails.
git diff <base>...HEAD \
| grep -i -E 'prj-[a-z0-9-]+|gcp-[a-z0-9-]+|\.internal[: /]|\.corp[: /]|[a-z0-9._-]+\.iam\.gserviceaccount\.com' \
| grep -v -e '^---' -e '^+++'
Expected matches: zero (outside `docs/archive/` and `CHANGELOG.md` historical
entries). Report any other match as Warning with the file:line. The OSS repo
must NOT carry the literal token list — that's why this agent reads it from
`CLAUDE.local.md` instead of inlining customer names here.
### 3. AI attribution
Check commit messages and PR body:
git log --format='%B' <base>..HEAD | grep -i -E 'co-authored-by: claude|generated with claude|claude code'
Expected: zero matches. Any match is Missing — main agent must remove them
before opening the PR.
### 4. Issue economy
Read the PR body and any new `TODO`/`FIXME` comments in the diff. Red flag:
filing a follow-up issue for something that is either fixable in this PR
(≤30 min, ≤1 file) or moot on current `main`. If found, report as Warning
with a recommendation: fix now, close as moot, or leave a `TODO` in the
touching diff.
### 5. Commit hygiene
`git log --oneline <base>..HEAD`. Red flags:
- Commit message includes AI attribution (already covered above).
- WIP / fixup / squash markers left in messages.
- Commits that should have been amended (e.g., "typo", "lint fix" of an
immediately preceding commit).
Report each as Warning, naming the SHA.
### 6. Release-cut implication
Invoke `Skill(agnes-release-process)`. Then: would this PR land the only
`[Unreleased]` content since the last tag? If yes, the release-cut commit
must be the last commit on this PR (version bump + CHANGELOG rename + new
empty `[Unreleased]`). Report as Done if present, Missing if not.
## Output format
Markdown, one section per check, three-line max per finding:
## CHANGELOG bullet — Done
Bullet under `## [Unreleased] > Added`: "Add foo to bar."
## Vendor-agnostic content — Warning
`docs/operator/runbook.md:42` mentions `<cloud-project-id>` (matched the `prj-…` pattern). Replace with `<project>` placeholder or move to the operator's private infra repo.
End with a one-line verdict: `OVERALL: ready / needs fixes`.
## Do not
- Do not edit files.
- Do not run tests.
- Do not call `gh pr merge` or push branches.
- Do not invent rules that are not in `CLAUDE.md` or the `agnes-release-process` skill.

72
.claude/skills/agnes-connectors.md Normal file
View file

@ -0,0 +1,72 @@
---
name: agnes-connectors
description: Rules for the extract.duckdb contract every data source must produce — the _meta table, the _remote_attach mechanism for remote-mode tables, parquet layout, and the pattern for adding a new connector. Use when adding a new data source or modifying an existing extractor in connectors/.
---
# Agnes connectors — the extract.duckdb contract
Every data source produces the same output:
/data/extracts/{source_name}/
├── extract.duckdb ← _meta table + views
└── data/ ← parquet files (local sources only)
See `CLAUDE.md § Architecture: extract.duckdb Contract` and
`docs/architecture.md`.
## Required `_meta` table
Every `extract.duckdb` MUST contain a `_meta` table with these columns:
| column | type | meaning |
|---|---|---|
| `table_name` | VARCHAR | name used in views |
| `description` | VARCHAR | human-readable description |
| `rows` | BIGINT | row count at extraction time |
| `size_bytes` | BIGINT | parquet size for local mode, 0 for remote |
| `extracted_at` | TIMESTAMP | extraction time |
| `query_mode` | VARCHAR | one of `local`, `remote`, `materialized` |
If `_meta` is missing or malformed, `SyncOrchestrator.rebuild()` skips the
source with an error logged. Tests for new connectors MUST assert `_meta` is
well-formed.
## Four connector shapes
- **Batch pull** (Keboola, `query_mode='local'`) — DuckDB extension downloads
data to parquet, scheduled. Extractor in
`connectors/<name>/extractor.py`.
- **Remote attach** (BigQuery, `query_mode='remote'`) — DuckDB BQ extension,
no download. Queries hit the upstream at query time. Requires `_remote_attach`.
- **Materialized SQL** (`query_mode='materialized'`) — scheduler runs
admin-registered SQL through DuckDB and writes the result to a parquet under
`/data/extracts/<source>/data/`. Distributed via the same manifest +
`agnes pull` flow as `local`. BigQuery cost guardrail:
`data_source.bigquery.max_bytes_per_materialize` (default 10 GiB; `0` disables).
- **Real-time push** (Jira) — webhooks update parquets incrementally; the
webhook handler triggers `rebuild_source('jira')`.
## `_remote_attach` table (remote mode only)
For each remote-mode table in `_meta`, the extractor writes a row in
`_remote_attach` with `alias`, `extension`, `url`, `token_env`. See the
`agnes-orchestrator` skill for how the orchestrator consumes it.
## Adding a new connector — checklist
1. Create `connectors/<name>/extractor.py` that emits `extract.duckdb` (+
`data/*.parquet` if local) into `/data/extracts/<name>/`.
2. Populate `_meta` with one row per table.
3. If any table is `query_mode='remote'`, populate `_remote_attach`.
4. Register the connector type in the catalog (search for existing
`source_type` values to follow the pattern).
5. Add a fixture-based test that runs the extractor against a fixture
upstream and asserts `_meta` is complete.
6. CHANGELOG bullet under `Added` per `agnes-release-process`.
## Stable infrastructure — do NOT modify
`connectors/jira/file_lock.py`. (`connectors/jira/transform.py` was
previously off-limits but as of 0.54.19 is no longer; it remains
sensitive — touch only with end-to-end understanding of the
JSON-overlay / parquet-rewrite pipeline.)

75
.claude/skills/agnes-orchestrator.md Normal file
View file

@ -0,0 +1,75 @@
---
name: agnes-orchestrator
description: Rules for the SyncOrchestrator, the extract.duckdb ATTACH flow, query_mode semantics (local / remote / materialized), and when to call rebuild() vs rebuild_source(). Use when editing src/orchestrator.py, src/db.py, or anything that produces extract.duckdb in connectors/.
---
# Agnes orchestrator
Source of truth for orchestrator invariants. See `CLAUDE.md § Architecture`
and `docs/architecture.md` for the canonical description.
## ATTACH flow
`SyncOrchestrator.rebuild()` scans `/data/extracts/*/extract.duckdb`,
ATTACHes each into the master `analytics.duckdb`, creates views like
`<source>."<bucket>"."<table>"`, and updates `sync_state`.
Per-source rebuild is `rebuild_source(name)` — used after Jira webhooks where
only one source changed. Full `rebuild()` is the fallback when scope is
unclear.
## Thread safety
All write paths take `self._rebuild_lock` (a `threading.Lock`). New write
paths — anything that DETACHes / re-ATTACHes / updates `sync_state` — MUST
hold the lock. Read paths must not hold it.
## query_mode
Every table has a `query_mode` in its `_meta` row:
- `local` — batch-pulled to parquet, queried locally. Parquets live under
`/data/extracts/<source>/data/`. Synced via `agnes pull`.
- `remote` — queried against the upstream (e.g., BigQuery) at query time.
No parquet on disk. Requires a `_remote_attach` row in `extract.duckdb`.
- `materialized` — admin-registered SQL run by the scheduler. Result lands as
a parquet under `/data/extracts/<source>/data/`. Distributed like `local`.
## `_remote_attach` mechanism
For `query_mode='remote'` tables, the extractor writes a `_remote_attach`
table in `extract.duckdb` with columns:
| column | meaning |
|---|---|
| `alias` | name used in the ATTACH statement |
| `extension` | DuckDB extension to install + load |
| `url` | upstream connection URL |
| `token_env` | env var holding the auth token (`''` if extension-specific auth, e.g., BigQuery's GCE metadata server) |
At query time the orchestrator installs/loads the extension, resolves the
token, creates a session-scoped SECRET when required, and ATTACHes the
source so views like `kbc."bucket"."table"` resolve.
## Master DB locations
- System DB: `${DATA_DIR}/state/system.duckdb` (sync_state, table_registry, users, RBAC).
- Analytics DB: `${DATA_DIR}/analytics/server.duckdb` (master views).
## Schema migrations
`src/db.py` auto-migrates from `v1 → vN` on startup. Per-version notes live
in `CHANGELOG.md`. Adding a schema version means:
1. Bumping the version constant in `src/db.py`.
2. Adding the `vN-1 → vN` migration step.
3. Adding a CHANGELOG bullet that names the version.
4. Updating documentation that references the schema version (search for
"schema v" in `docs/` + `CLAUDE.md`).
## Files NOT to modify
`connectors/jira/file_lock.py` and `services/ws_gateway/` — stable
infrastructure. (`connectors/jira/transform.py` was previously off-limits
but as of 0.54.19 is no longer; it remains sensitive — touch only with
end-to-end understanding of the JSON-overlay / parquet-rewrite pipeline.)

64
.claude/skills/agnes-rbac.md Normal file
View file

@ -0,0 +1,64 @@
---
name: agnes-rbac
description: Rules for endpoint gating (require_admin vs require_resource_access), ResourceType registration, and the user_groups model. Use when adding or changing endpoints in app/api/, touching app/auth/, or introducing a new resource type.
---
# Agnes access control
Two-layer model with no role hierarchy. See `CLAUDE.md § Access control` and
`docs/RBAC.md`.
## Tables
- `user_groups` — named groups. `Admin` (god-mode short-circuit on every
authorization check) and `Everyone` (auto-membership) are seeded as
`is_system=TRUE`.
- `user_group_members` — `(user_id, group_id, source)`. `source` segregates
writers so Google's nightly sync does not clobber admin-added members.
- `resource_grants` — `(group, resource_type, resource_id)` triples for any
entity-scoped grant.
## Gate decision
For every new endpoint, pick one:
- `Depends(require_admin)` — app-level mutations (anything that changes shared
state without a per-entity scope: registering tables, creating users,
managing groups, server config).
- `Depends(require_resource_access(ResourceType.X, "{path}"))` — entity-scoped
reads or mutations. The path expression extracts the `resource_id` from the
request.
Both imports live in `app.auth.access`.
## Adding a new ResourceType
1. Extend the `ResourceType` `StrEnum` in `app/resource_types.py` with the
new value.
2. Register a `ResourceTypeSpec` for it in the same file, including a
`list_blocks` projection delegate that returns the rows visible to a
given caller.
3. **No DB migration needed** — `resource_grants` is generic.
4. Gate the endpoints that consume the new type with
`require_resource_access(ResourceType.NEW, "{path}")`.
## Admin layer is the source of truth for auto-sync
For `agnes pull`: `query_mode IN ('local', 'materialized')` plus a
`resource_grants` row for one of the analyst's groups → table appears in
their manifest. There is no per-user sync config.
## Auth providers
Auth providers live in `app/auth/`:
- **Google OAuth** — sign-in via Google. Workspace group memberships are
pulled at sign-in (see `docs/auth-groups.md` for GCP setup checklist + the
`security` label gotcha).
- **Email magic link** — itsdangerous token.
- **Desktop JWT** — for the CLI / API.
## Admin UI and CLI
- Admin UI: `/admin/access`.
- CLI: `agnes admin group …` and `agnes admin grant …`.

86
.claude/skills/agnes-release-process.md Normal file
View file

@ -0,0 +1,86 @@
---
name: agnes-release-process
description: Rules for opening a PR, the CHANGELOG bullet, the release-cut commit, and the post-merge tag + GitHub Release. Use before opening a PR, before merge, when handling a release-cut, and when picking a version bump.
---
# Agnes release process
Source of truth for the rules in `CLAUDE.md § Release process` and
`docs/RELEASING.md`. This skill is invoked by the main agent during planning,
by `agnes-reviewer-rules` during review, and by `agnes-releaser` during the
release-cut. When the rules below conflict with the master documents above,
the master documents win — update this skill.
## When this skill applies
- Opening a PR
- Reviewing a PR (release-cut implications)
- Cutting a release (version bump, CHANGELOG rename)
- Post-merge tagging + GitHub Release
## CHANGELOG discipline
Every PR that changes **user-visible behavior** MUST add a bullet under
`## [Unreleased]` in `CHANGELOG.md`, grouped under Added / Changed / Fixed /
Removed / Internal. Breaking changes are prefixed `**BREAKING**`.
Doc-only PRs (`docs/**`, README) typically do not need a bullet. Apply
judgment based on the diff — if the docs change describes new behavior that
should have shipped with a code change, the *code* PR carries the bullet.
The CHANGELOG entry is part of the PR that introduces the change — never a
follow-up PR.
## Release-cut belongs in the PR
If a PR lands the only `[Unreleased]` content since the last release, the
release-cut is the **last commit on that PR**:
1. Bump `pyproject.toml` (`version = "X.Y.Z"`).
2. Rename `## [Unreleased]` to `## [X.Y.Z] - YYYY-MM-DD`.
3. Add a new empty `## [Unreleased]` above it.
The release-cut is never a standalone follow-up PR.
## Version bump decision
- **Patch** (X.Y.Z+1): default for bug fixes, internal refactors, doc tweaks,
small features that do not change documented behavior.
- **Minor** (X.Y+1.0): new user-visible features, new APIs, schema migrations
that are backwards-compatible. **Ask the user before picking minor.**
- **Major** (X+1.0.0): breaking changes, removed APIs, incompatible schema
changes. Requires explicit user confirmation.
## Post-merge sequence
After the PR with the release-cut is merged to `main`:
1. `git tag vX.Y.Z <merge-sha>`
2. `git push origin vX.Y.Z`
3. `gh release create vX.Y.Z --title "vX.Y.Z" --notes "<CHANGELOG body for [X.Y.Z]>"`
Never tag or release before merge.
## Post-merge auto-rollback
On every `main` push, GitHub Actions `release.yml` builds the `:stable`
image and a `smoke-test` job pulls it and runs a docker-compose stack.
If the smoke test fails:
- `rollback-on-smoke-fail` calls `rollback.yml`, which re-points `:stable`
to the previous known-good build.
- A tracking issue labeled `bug` is opened with the failing image, the
commit SHA, the deprecated tag, and the rollback target.
Success signal after merge: `smoke-test` green AND `rollback-on-smoke-fail`
skipped. If rollback fires, the merge shipped a broken image to GHCR —
investigate the tracking issue before any further push.
Manual rollback, forced target, and weekly tag-pruning operator commands
live in `docs/RELEASING.md`.
## Tests before push
Run `.venv/bin/pytest tests/ --tb=short -n auto -q` before every push.
Failures in code you touched: fix before pushing. Failures unrelated:
confirm they reproduce on a clean branch, note in the PR body, do not block.

4
.gitignore vendored
View file

@ -1,5 +1,7 @@
# Claude Code
.claude/
.claude/*
!.claude/agents/
!.claude/skills/
CLAUDE.local.md
# Local dev data (copied from server for testing)

17
CHANGELOG.md
View file

@ -10,6 +10,23 @@ CalVer image tags (`stable-YYYY.MM.N`, `dev-YYYY.MM.N`) are produced for every C
## [Unreleased]
### Internal
- **Repo-committed Claude Code agents + skills under `.claude/`.**
Four knowledge skills (`agnes-orchestrator`, `agnes-rbac`,
`agnes-connectors`, `agnes-release-process`) auto-load into the main
agent's context when their description matches the work or are
invokable explicitly via `Skill(<name>)`. Four specialist subagents
(`agnes-reviewer-rules`, `agnes-reviewer-rbac`,
`agnes-reviewer-architecture`, `agnes-releaser`) wire into the
Agent tool — reviewers fire in parallel at the end of PR work;
the releaser handles pre-merge release-cut + post-merge tag /
GitHub Release. `.gitignore` un-ignores `.claude/agents/` and
`.claude/skills/` while keeping the rest of `.claude/` local-only.
Source of truth for the rules these encode remains `CLAUDE.md` +
`docs/RELEASING.md`. Design rationale +
implementation plan: `docs/superpowers/specs/2026-05-15-agnes-agents-design.md`
and `docs/superpowers/plans/2026-05-15-agnes-agents.md`.
## [0.54.20] — 2026-05-15
### Added

9
CLAUDE.md
View file

@ -311,6 +311,15 @@ Full recipe, deploy workflows, manual rollback runbook, weekly tag-housekeeping,
- **Run the full test suite before every push** — `.venv/bin/pytest tests/ --tb=short -n auto -q` (this is what CI runs). Failures in code you touched: fix before pushing. Failures unrelated to your diff: confirm with `git stash` they reproduce on a clean branch, note them in the PR body, don't block on them.
- **Watch the post-merge `release.yml` run.** On `main` pushes a `smoke-test` job pulls the just-built `:stable` image and runs a docker-compose stack; if it fails, the `rollback-on-smoke-fail` job calls the reusable `rollback.yml` workflow which re-points `:stable` to the previous known-good build and opens a tracking issue labeled `bug`. Success signal after merge = `smoke-test` green + `rollback-on-smoke-fail` skipped. If the rollback fires, the merge shipped a broken image to GHCR — investigate the tracking issue before any further push (the issue body has the failing image, commit SHA, deprecated tag, and rollback target). Manual rollback / forced target / weekly tag-pruning operator commands are in [`docs/RELEASING.md`](docs/RELEASING.md).
## Specialized agents and skills
Two committed locations carry Agnes-specific Claude Code behavior:
- `.claude/skills/agnes-*.md` — knowledge skills (`agnes-orchestrator`, `agnes-rbac`, `agnes-connectors`, `agnes-release-process`). Loaded into the main agent's context when their description matches the work, or invoked explicitly via `Skill(<name>)`. Read these before editing the corresponding part of the codebase.
- `.claude/agents/agnes-*.md` — specialist subagents (`agnes-reviewer-rules`, `agnes-reviewer-rbac`, `agnes-reviewer-architecture`, `agnes-releaser`). Spawned via the Agent tool at the end of PR work (reviewers, in parallel) or explicitly before/after merge (releaser).
Design rationale: `docs/superpowers/specs/2026-05-15-agnes-agents-design.md`.
## Project conventions
### Vendor-agnostic OSS — no customer-specific content

1101
docs/superpowers/plans/2026-05-15-agnes-agents.md Normal file
View file

File diff suppressed because it is too large Load diff

313
docs/superpowers/specs/2026-05-15-agnes-agents-design.md Normal file
View file

@ -0,0 +1,313 @@
# Agnes specialized agents — design
**Status:** approved (brainstorm), not yet implemented
**Date:** 2026-05-15
**Author:** zsrotyr
## Problem
Working on Agnes through Claude Code, three classes of friction recur:
1. **Mental model drift.** Claude (and humans) forget how Agnes hangs together — the `extract.duckdb` contract, RBAC layering (`require_admin` vs `require_resource_access`), `query_mode` semantics (local / remote / materialized), the orchestrator's `rebuild()` flow. Edits to one part silently break invariants of another, caught only at code review.
2. **Convention enforcement at review.** Several CLAUDE.md rules (CHANGELOG bullet, vendor-agnostic OSS, no AI attribution, issue economy, RBAC gates on new endpoints) are easy to forget. Manual review catches most but not all.
3. **Release-cut workflow.** The non-negotiable rules from `docs/RELEASING.md` and `CLAUDE.md § Release process` — release-cut belongs in the PR that earned it, last commit on the PR, post-merge tag + GitHub Release — evolve and are repetitive enough that doing them by hand each time is error-prone.
The brainstorm explored variants A ("one big architect"), B ("three specialist subagents covering all three concerns"), and C ("knowledge as skills, enforcement/workflow as subagents"). C was selected because the mental model is *not delegatable* — when the main agent writes code it needs the context in its own window, not in a subagent's.
## Approach
Four layers, each with a distinct mechanism and lifecycle:
```
Layer 1: KNOWLEDGE SKILLS (.claude/skills/agnes-*.md)
- Auto-trigger by description + explicit invocation by main agent
- Load into MAIN agent's context
- Purpose: main agent knows how Agnes works while writing code
Layer 2: REVIEWER SUBAGENTS (.claude/agents/agnes-reviewer-*.md)
- Spawned via Agent tool at end of PR work
- Own context window; return a punch list
- Three specialists, fired in parallel by main agent
Layer 3: RELEASER SUBAGENT (.claude/agents/agnes-releaser.md)
- Spawned via Agent tool pre-merge (phase 1) and post-merge (phase 2)
- Own context window; produces release-cut commit + tag + GitHub Release
Layer 4: PERSONAL (~/.claude/agents/<customer>-deploy.md)
- Outside this repo (operator's home dir, never committed)
- Customer-specific context (customer VMs, gcloud / cloud accounts,
private infra repos, deploy targets)
```
Skills (Layer 1) share context with the main agent — when Claude edits
`src/orchestrator.py`, the orchestrator skill is loaded so rules stay "in head"
across delegation. Subagents (Layers 2 and 3) isolate context — review/release
read many files but only a punch list returns to the main conversation.
Cross-layer sharing happens through skills. The `agnes-release-process` skill is
read by the main agent (planning), by `agnes-reviewer-rules` (checking), and by
`agnes-releaser` (executing). Single source of truth.
## Components
### Layer 1 — Knowledge skills (`.claude/skills/`)
Four skills. Each file is ~80–120 lines. Files are kept focused; if one grows
past ~150 lines that is a signal to split. Skills do not duplicate `CLAUDE.md`
content verbatim — they reference (`see CLAUDE.md § Access control`) so master
rules have one location.
#### `agnes-orchestrator`
- **Description (triggers auto-spawn):** Use when editing `src/orchestrator.py`,
`src/db.py`, or anything that produces `extract.duckdb` in `connectors/*/`.
Rules for ATTACH flow, `query_mode` semantics, and when `rebuild()` is
required.
- **Body:** master view lifecycle in `analytics.duckdb`; thread-safety via
`_rebuild_lock`; `rebuild_source(name)` vs full `rebuild()` decision;
`_remote_attach` reattach flow at query time (extension install + token
resolution via `token_env` or extension-specific auth path).
#### `agnes-rbac`
- **Description:** Use when adding or changing an endpoint in `app/api/`,
touching `app/auth/`, or introducing a new resource type. Enforces gate
pattern (`require_admin` vs `require_resource_access`) and `ResourceType`
registration.
- **Body:** decision tree for picking a gate (app-level mutation vs
entity-scoped); how to add a new `ResourceType` (StrEnum value +
`ResourceTypeSpec` with `list_blocks` delegate in `app/resource_types.py`, no
DB migration); when a grant is needed even for reads; god-mode short-circuit
for the `Admin` group.
#### `agnes-connectors`
- **Description:** Use when adding a new data source or modifying an existing
extractor in `connectors/`. Enforces the `extract.duckdb` contract — `_meta`
table, `query_mode` column, parquet layout.
- **Body:** required `_meta` columns (`table_name`, `description`, `rows`,
`size_bytes`, `extracted_at`, `query_mode`); when a connector is batch-pull
vs remote-attach vs real-time push; how to expose `_remote_attach` for remote
mode; where the extractor writes (`/data/extracts/{source}/`).
#### `agnes-release-process`
- **Description:** Use before opening a PR, before merge, or when handling a
release-cut. Rules for CHANGELOG bullet, when the release-cut commit belongs
in the PR, version bump decision (patch is default; ask before minor).
- **Body:** CHANGELOG discipline (Added / Changed / Fixed / Removed / Internal
grouping, `**BREAKING**` prefix); release-cut decision tree (when it is the
last commit on the PR); post-merge sequence (tag `vX.Y.Z` on merge commit +
`gh release create`); patch / minor / major guidance.
### Layer 2 — Reviewer subagents (`.claude/agents/`)
Each subagent has a standard frontmatter (`name`, `description`, `tools`,
`model`). All are read-only (no `Edit` / `Write`) — they return punch lists, not
code changes.
#### `agnes-reviewer-rules`
- **When fired:** every PR, at end of work, before opening the PR.
- **Tools:** `Read`, `Bash` (restricted to `git diff`, `git log`, `grep`).
- **Model:** Haiku — fast, mostly text work.
- **Input from main agent:** PR branch name (or current HEAD), optionally PR
draft body.
- **Checks:**
- CHANGELOG.md has a new bullet under `[Unreleased]` *iff* the PR changes
user-visible behavior. Smart, not blind — doc-only PRs typically do not
need a bullet; judgment applied based on the diff.
- No customer-specific tokens in the diff or PR body (deployment names,
internal hostnames, cloud project IDs, internal SA emails). The
`agnes-reviewer-rules` subagent loads the operator's token list from
`CLAUDE.local.md` (`vendor_tokens:` entry) rather than inlining
customer names in the public OSS repo.
- Commits do not contain `Co-Authored-By: Claude` or any AI attribution; PR
body is the same.
- Issue-economy red flags — filing follow-up issues rather than fix-it-now or
close-it-as-moot.
- Commit messages are clean and concise per project convention.
- Consults `agnes-release-process` skill for the release-cut implication of
this PR.
- **Output:** Done / Missing / Warning punch list.
#### `agnes-reviewer-rbac`
- **When fired:** when the diff touches `app/api/`, `app/auth/`, or
`app/resource_types.py`.
- **Tools:** `Read`, `Grep`, `Bash` (read-only).
- **Model:** Sonnet — needs to understand the auth flow.
- **Checks:**
- New `@router.get/post/...` handlers have `Depends(require_admin)` or
`Depends(require_resource_access(ResourceType.X, "..."))`.
- New `ResourceType` values have a `ResourceTypeSpec` registration in
`app/resource_types.py`.
- Consults `agnes-rbac` skill for the gate decision rules.
- **Output:** per-endpoint flag — gated correctly / missing gate / ambiguous.
#### `agnes-reviewer-architecture`
- **When fired:** when the diff touches `src/orchestrator.py`, `src/db.py`,
`connectors/*/extractor.py`, or adds a schema migration.
- **Tools:** `Read`, `Grep`, `Bash` (read-only).
- **Model:** Sonnet.
- **Checks:**
- Extractor changes preserve `_meta` table contract.
- Remote-attach changes preserve `_remote_attach` columns (`alias`,
`extension`, `url`, `token_env`).
- Schema bumps in `src/db.py` include the `vN-1 → vN` migration step, a
CHANGELOG note, and documentation references that reflect the new version.
- Changes to `rebuild()` / `rebuild_source()` hold `_rebuild_lock` on all
write paths.
- Consults `agnes-orchestrator` and `agnes-connectors` skills.
- **Output:** per-invariant punch list — holds / broken / unclear.
### Layer 3 — Releaser subagent (`.claude/agents/agnes-releaser.md`)
- **Tools:** `Read`, `Edit`, `Bash` (including `gh`, `git`).
- **Model:** Sonnet.
Two phases, each invoked explicitly by the user (never auto-fired).
**Phase 1 — pre-merge.** User says "ready to merge".
1. Consults `agnes-release-process` skill.
2. Runs `git log` since the last tag and inspects scope.
3. Decision tree: patch (default) vs minor (asks user) vs major (requires
explicit confirmation).
4. If this PR lands the only `[Unreleased]` content since the last release,
prepares the last commit on the PR: bump `pyproject.toml`, rename
`[Unreleased]` to `[X.Y.Z] - YYYY-MM-DD`, add a new empty `[Unreleased]`.
5. Pushes the prepared commit. **Does not merge** — the user merges via
`gh pr merge` themselves.
**Phase 2 — post-merge.** User says "tag it".
1. Verifies the merge commit contains the release-cut diff.
2. `git tag vX.Y.Z <merge-sha>` and `git push origin vX.Y.Z`.
3. `gh release create vX.Y.Z` with body extracted from the `[X.Y.Z]` section of
CHANGELOG.
4. Returns the GitHub Release URL.
**Never does:** merges the PR (high-blast-radius); force-pushes; amends
published commits.
### Layer 4 — Personal (`~/.claude/agents/<customer>-deploy.md`)
Outside this repo. Customer-specific content lives here so the OSS repo stays
vendor-agnostic per `CLAUDE.md § Vendor-agnostic OSS`. Operators write their
own Layer-4 agent against their deployment.
- **Tools:** `Read`, `Bash` (including the operator's cloud CLI — `gcloud`,
`aws`, etc. — plus `gh`, `git push`).
- **Model:** Sonnet.
- **Knows** (each item is customer-specific; concrete tokens stay in the
operator's home dir / `CLAUDE.local.md`, never committed):
- VM ↔ project ↔ zone ↔ cloud-account mapping (one row per deployment).
- Per-environment deploy ritual (e.g., force-push to `<branch>` to deploy
to `<host>`).
- Cross-references to private infra repos and pinned module tag.
- Default cloud account selection rules; explicit `--account=` overrides.
- Any pre-existing legacy systems still running alongside the deployment.
- **Does not** touch the OSS repo, push to public branches, or participate in
PR review.
## End-to-end PR flow
```
1. User asks for feature X.
2. Main agent creates a worktree (per CLAUDE.local.md), invokes relevant
knowledge skills based on what the feature touches:
- src/orchestrator.py → Skill(agnes-orchestrator)
- app/api/ + app/auth/ → Skill(agnes-rbac)
- connectors/ → Skill(agnes-connectors)
3. Implementation + tests + CHANGELOG bullet (the agnes-release-process skill
reminded the main agent it is needed).
4. Before opening the PR, main agent fires reviewers in parallel — one message,
multiple Agent tool calls:
- Agent(agnes-reviewer-rules) — always
- Agent(agnes-reviewer-rbac) — only if diff touched app/api or app/auth
- Agent(agnes-reviewer-architecture) — only if diff touched src/ or connectors/
5. Main agent aggregates the punch lists, fixes findings, opens the PR.
6. User says "ready to merge" → Agent(agnes-releaser, phase 1) prepares the
release-cut decision and the last commit on the PR.
7. User confirms the version → releaser pushes the prepared commit. User
merges via `gh pr merge` manually.
8. User says "tag it" → Agent(agnes-releaser, phase 2) creates the tag and the
GitHub Release.
```
A separate flow handles personal dev-VM deploys (outside the OSS PR cycle):
```
User says "push to my VM" → main agent invokes Agent(<customer>-deploy) →
that personal-layer agent knows what "push to my VM" means in the operator's
environment (typically a force-push to a deploy-target branch), confirms with
the user (force-push is destructive), pushes.
```
## What lands in the repo
```
.claude/
├── agents/
│ ├── agnes-reviewer-rules.md
│ ├── agnes-reviewer-rbac.md
│ ├── agnes-reviewer-architecture.md
│ └── agnes-releaser.md
└── skills/
├── agnes-orchestrator.md
├── agnes-rbac.md
├── agnes-connectors.md
└── agnes-release-process.md
docs/superpowers/specs/
└── 2026-05-15-agnes-agents-design.md (this document)
```
`CLAUDE.md` gets a short paragraph under "Project conventions" pointing at
`.claude/agents/` and `.claude/skills/`, noting that subagents and skills exist
and how to invoke them.
The personal layer (`~/.claude/agents/<customer>-deploy.md`) is written
separately by each operator and is not part of the in-repo change set.
## Non-goals
- **Not a Claude Code "team".** The architecture, review, and release agents
work sequentially and do not message each other; a team would add the
experimental flag, more tokens, file-conflict risk, and no functional gain.
If `agnes-reviewer-*` ever needs to coordinate across four+ parallel reviews,
re-evaluate.
- **Not a pre-commit hook replacement.** Mechanical checks (CHANGELOG bullet
present, AI-attribution scan) could be a `pre-commit` hook in addition; the
reviewer agent provides judgment-level checks the hook cannot.
- **No auto-merge.** `agnes-releaser` never runs `gh pr merge`. Merge is a
visible, user-controlled action.
## Open questions
- **Auto-trigger reliability for knowledge skills.** Claude Code skill
auto-spawn is description-driven and not always reliable in large skill
catalogs. Mitigation: a one-line pointer at the top of `CLAUDE.md`
("when touching X, invoke skill Y") and explicit invocation in the main
agent's planning step. If reliability is still low after a few weeks, fall
back to a `SessionStart` hook that lists Agnes-specific skills as a reminder.
- **Schema migration as a separate skill?** `agnes-orchestrator` covers
`src/db.py` migration patterns. If migration gotchas grow (versioning beyond
`vN`, multi-step migrations, data backfills), split into a fifth skill
`agnes-schema-migration`.
## Implementation plan
The follow-up step is to invoke the `writing-plans` skill to turn this spec
into a sequenced implementation plan covering: skills first (lowest risk,
testable in isolation), then reviewers, then the releaser, then a CLAUDE.md
pointer. Personal layer lands separately, outside the repo.