Commit graph

9 commits

Author SHA1 Message Date
ZdenekSrotyr
3d58768143 fix: address Devin Review findings — incomplete renames + estimate guard
13 Devin findings across 10 files:

🔴 Critical:
- app/api/v2_catalog.py:42 — `_fetch_hint` returns `da fetch` in /api/v2/catalog
  responses (user-visible in every catalog list)
- cli/skills/agnes-data-querying.md — 11 stale `da fetch`/`da sync` refs in the
  bundled skill markdown
- config/claude_md_template.txt:38 — referenced `agnes pull --docs-only` flag
  that does NOT exist in agnes pull (removed; spec only ships --quiet/--json/
  --dry-run)

🟡 Important:
- app/api/admin.py:252 — `da fetch` in bq_max_scan_bytes hint
- cli/commands/auth.py:119 — `da sync` in import-token docstring (--help text)
- cli/commands/tokens.py:48 — "Export it so `da` can use it" prose
- ARCHITECTURE.md — 4 stale rows in CLI commands table
- README.md — stale paragraphs for analysts (da sync, da analyst setup)

🚩 Substantive observations addressed:
- app/api/query.py:249,302,489 — server-side error/help strings still said
  `da sync`/`da fetch` (returned in API responses to clients)
- cli/commands/snapshot.py:235-241 — DuckDB existence guard incorrectly
  blocked `--estimate` (server-side dry-run that never opens local DB).
  Added test ensuring estimate path skips the guard.

Skipped (intentionally historical):
- app/api/admin.py:2377,2429,2437 — historical comments describing past
  manifest-vs-sync_state bug; past tense, accurate to keep as `da sync`.
2026-05-04 20:05:06 +02:00
ZdenekSrotyr
d25d075ed2 docs(claude-md-template): rewrite verbs + paths for new CLI surface (Task 6)
- Verb renames (da X -> agnes X for surviving verbs; legacy verbs already
  absent from this default template — admin overrides with legacy verbs are
  caught by Task 2's _LEGACY_STRINGS scan + Task 5's admin banner).
- Path renames: data/parquet/ -> server/parquet/, data/duckdb/ ->
  user/duckdb/, data/metadata/ removed entirely (no longer exists per spec).
- Drop user/artifacts/ from directory structure (spec workspace layout
  drops it; surviving paths: server/parquet/, user/duckdb/, user/snapshots/,
  user/sessions/).
- Add AGNES_WORKSPACE.md pointer near top-of-template so analysts know
  where to find human-readable docs.

Cleans Task 0.5's missed sweep on this file (was not in cli/ tree but is
user-visible via /api/welcome).

81 claude_md/welcome_template tests pass.
2026-05-04 17:51:14 +02:00
ZdenekSrotyr
a2157ee807 fix(claude_md): restore full default content (BQ cost guard, hybrid example, ad-hoc table, deeper guidance) 2026-05-04 05:48:04 +02:00
ZdenekSrotyr
f01eb4143d feat(db,repo,renderer): schema v23 + claude_md_template + ClaudeMd renderer
- Bump SCHEMA_VERSION 22 → 23; add claude_md_template singleton table to
  _SYSTEM_SCHEMA and _V22_TO_V23_MIGRATIONS; wire migration + fresh-install seed
- src/repositories/claude_md_template.py: ClaudeMdTemplateRepository (get/set/reset)
  mirroring WelcomeTemplateRepository; defensive re-seed in get()
- src/claude_md.py: compute_default_claude_md / render_claude_md /
  build_claude_md_context — rich renderer with RBAC-filtered tables, metrics,
  and marketplaces; reads override from claude_md_template or falls back to
  config/claude_md_template.txt; raises TemplateError on broken override
- config/claude_md_template.txt: default Jinja2 markdown template restored from
  PR #167 history (tables, metrics, marketplaces, BQ guidance, corporate memory,
  directory structure, per-user footer)
2026-05-03 22:43:56 +02:00
ZdenekSrotyr
c4d23cf235 feat(admin-prompt): update editor UX + docs for banner context
- admin_welcome.html: update subtitle, description, placeholder cheatsheet
  (drop tables/metrics/marketplaces/sync_interval; add user-null note and
  security note). Textarea initial value is now empty (no default template
  to show). Preview pane uses innerHTML (HTML output). refreshStatus sets
  editor to empty when no override. Preview pane styled as light surface.
  Reset modal copy updated (no banner shown, not "OSS-shipped template").
- config/claude_md_template.txt: deleted (markdown template is gone;
  default is now no banner).
- docs/agent-setup-prompt.md: rewritten for variant C — describes the
  /setup banner, smaller placeholder table, security/sanitization notes,
  anonymous-user guard, example HTML snippet.
2026-05-03 16:12:13 +02:00
ZdenekSrotyr
d055417377 feat(config): default welcome template in jinja2 + sync_interval 2026-05-03 16:10:48 +02:00
Vojtech
bd7b8c3233
fix(analyst): document BigQuery remote-query capability in bootstrap CLAUDE.md template (#154)
* fix(analyst): document BigQuery remote-query capability in bootstrap CLAUDE.md template

Closes #153.

The CLAUDE.md template generated by `da analyst bootstrap` (config/claude_md_template.txt)
covered metrics, sync, corporate memory, and directory layout — but had ZERO
mention of query_mode: "remote", da fetch, da query --remote, or --register-bq.
Result: the AI analyst running in a freshly-bootstrapped workspace had no
idea BigQuery-backed tables existed, no path to fetch unsynced data, and no
fallback for tables not in the catalog.

Validated against /Users/<user>/foundry-ai/foundryai-data-analyst/CLAUDE.md
on 2026-05-01: section confirmed missing. Workspace-level (parent-dir)
CLAUDE.md carried legacy SSH-heredoc instructions but the analyst-level
file (which Claude reads as primary project context) had nothing.

## Changes

### config/claude_md_template.txt (+83)

Added a `## Remote Queries (BigQuery)` section covering:

- Discovery first — `da catalog --json | jq '...'` to see all tables
  with their query_mode, then `da schema` and `da describe` for shape.
- Three query patterns:
  - `da fetch` (preferred) — materialize a filtered subset locally,
    query the snapshot, drop when done.
  - `da query --remote` — one-shot server-side execution (cheap probes).
  - `da query --register-bq` — hybrid joins between local + ad-hoc BQ.
- `da fetch` estimate-first discipline — rules of thumb on
  --select / --where / --estimate / snapshot reuse.
- BigQuery SQL flavor cheat sheet for `--where` (DATE literal,
  DATE_SUB, REGEXP_CONTAINS, CAST AS INT64).
- Unknown-table fallback: when a table isn't in `da catalog` at all,
  use ad-hoc `--register-bq` if the agnes server SA has BQ access, or
  ask admin to register with `query_mode: "remote"` for ongoing use.
- Pointer to `da skills show agnes-data-querying` for deeper guidance.

### docs/setup/claude_md_template.txt (deleted)

Stale 359-line template that documented the deprecated SSH-heredoc
remote_query.sh protocol. No code references it (verified via grep
across .py / .sh / .yml / .md). Removing eliminates two failure
modes:
1. A future refactor accidentally pulling it into a workspace and
   shipping deprecated guidance to analyst Claude sessions.
2. Reviewer confusion over which template is canonical.

### CHANGELOG.md

`### Fixed` and `### Removed` entries under [Unreleased].

## Tested

- Manually walked the diff against `da skills show agnes-data-querying`
  output on a live VM (foundryai-development) — patterns + flags
  match the modern CLI exactly.
- Re-bootstrap test deferred: requires network round-trip; pattern
  is identical to existing template substitution path so render is
  not at risk.

## Out of scope

- The companion gap that data_description.md often only enumerates
  query_mode: "local" tables (no signal that other modes exist) —
  separate concern, fix likely belongs in the metadata generator
  on the server side, not in the analyst template.
- Encouraging admins to register frequently-queried BQ tables as
  `query_mode: "remote"` in the registry — workflow improvement, not
  a code bug.

* chore(release): cut 0.28.0

---------

Co-authored-by: ZdenekSrotyr <zdenek.srotyr@keboola.com>
2026-05-01 12:06:41 +02:00
PavelDo
e1108b6112
feat(memory): corporate memory v1+v1.5 + 0.15.0 (#72)
Adds corporate memory v1 (verification flywheel + contradiction detection + confidence scoring) and v1.5 (audience-based distribution + per-item privacy + admin curation). Server: GET /api/memory/bundle returns mandatory + ranked-approved items within a token budget; POST /api/memory/admin/mandate accepts an audience field gated against user_group_members; /api/memory/stats uses SQL aggregation. CLI: da sync writes received items to .claude/rules/km_*.md. Verification detector extracts knowledge candidates from session JSONL files. Auto-tagging via Haiku when ai: is configured. Adapted from the v9-era branch onto v13/v14 RBAC: _is_privileged_viewer + _effective_groups now query user_group_members JOIN user_groups; require_role(Role.KM_ADMIN) replaced with require_admin (km_admin collapsed into admin). Schema v15: knowledge_items context-engineering columns + knowledge_contradictions + session_extraction_state. Schema v16: verification_evidence. Cuts release v0.15.0 (also bundles #116 /me/debug page).
2026-04-29 07:16:22 +02:00
ZdenekSrotyr
c825ead209 feat: add CLAUDE.md template for analyst bootstrap 2026-04-10 19:41:07 +02:00