9f5adbce37
* ci: add actionlint workflow lint, drop superseded deploy.yml stub
* ci: extract rollback into reusable rollback.yml, wire into release smoke-test
* ci: add weekly prune-dev-tags workflow for legacy CalVer tag/image cleanup
* release: 0.54.17 — CI/release workflow consolidation
* fix(ci): warn when rollback.yml receives a non-stable failed_image_tag
* fix(ci): rollback.yml + prune-dev-tags.sh review findings
rollback.yml:
- Pass workflow_dispatch inputs (failed_image_tag, target_image_tag)
through env: instead of textual ${{ }} splicing into bash run blocks
— prevents an actor with workflow_dispatch privilege from injecting
shell via quote/backtick payloads.
- Guard against TARGET == FAILED when only one stable-* tag exists
(fresh repo, or aggressive pruning at month boundary). Fail loudly
rather than re-push the broken image as :stable.
- Add commit SHA to the rollback tracking-issue body — github.sha is
inherited across workflow_call, so on-call no longer has to navigate
rollback run → caller-workflow breadcrumb → failing commit.
prune-dev-tags.sh:
- Replace 'printf … | head -20' preview pipeline with array slice
('"${TO_PRUNE[@]:0:20}"'). Under set -o pipefail, head closing
the pipe early SIGPIPEs printf (exit 141) and aborts the script
before any deletion runs — exactly the multi-month-backlog scenario
the script targets.
- Refactor GHCR-pass: fetch versions JSON once before the loop, then
build a tag→version-id map up-front. Closes two problems:
1. O(N × pages) GHCR API calls collapse to one paginated listing
— months of accumulated CalVer tags no longer risk tripping
abuse detection.
2. The new jq filter excludes any version that ALSO carries a
floating alias (:stable, :dev, *-latest). GHCR DELETE-version
drops the entire manifest, so pruning a CalVer tag that shares
a manifest with :stable (e.g. after a rollback re-tag) would
have vaporized :stable. Now it's skipped with a log line.
lint-workflows.yml:
- Add an explicit shellcheck step. actionlint only walks
.github/workflows/ and the shell embedded in their run: blocks, so
freestanding scripts/ops/*.sh (which are in the workflow's path
filter) were never actually validated despite triggering CI.
* fix(ci): shellcheck --severity=warning to skip pre-existing info findings
The new shellcheck step caught info-level findings (SC1091, SC2015) in
agnes-auto-upgrade.sh / agnes-tls-rotate.sh — pre-existing, not regressed
by this PR. Constrain shellcheck to warning+ severity (real bugs) so info
and style findings don't block CI; mirrors the actionlint step's
continue-on-error initial-rollout posture.
* fix(ci): second-pass review findings — concurrency, walk-back, failure propagation
rollback.yml:
- Add own concurrency block (group: rollback-<repo>-<failed_tag>,
cancel-in-progress: false). The caller release.yml uses
cancel-in-progress: true to avoid duplicate CalVer claims, but a
second push to main mid-rollback would otherwise kill the workflow
between the :stable recovery push and the :deprecated-* audit push,
leaving :stable stuck on the broken image. A reusable workflow's own
concurrency overrides the inherited one.
- Walk back through stable-* tags newest-first, skipping any whose
:deprecated-<stripped> GHCR alias already exists (carries the mark of
a prior failed rollback). The previous 'second-most-recent' heuristic
could re-point :stable at a known-broken image on cascading failures.
- Reorder re-tag step: push :stable recovery FIRST, then the
:deprecated-* audit tag. Defense in depth — even if the concurrency
block somehow misfires, the worst case is missing audit metadata
rather than production stuck on the broken image.
- Move GHCR login before resolve step so 'docker manifest inspect' can
probe for :deprecated-* aliases during walk-back.
- Document the top-level permissions block's dual semantics
(workflow_dispatch grants directly; workflow_call acts as a cap
intersected with the caller's job-level permissions).
release.yml:
- Rewrite the 'issues: write' comment. Old wording ('default for jobs')
was factually wrong — GITHUB_TOKEN's default for issues is never write
— and read as 'this line just documents a default', so a future
cleanup PR could delete it. The line is load-bearing: workflow_call
permissions are bounded by the caller's GITHUB_TOKEN scope, and
removing it would silently 403 rollback.yml's gh issue create step.
prune-dev-tags.sh:
- Drop the '|| echo "[]"' fallback on the GHCR versions fetch. The
fallback turned every API failure (403 missing scope, 429 rate limit,
transient 5xx) into a silent no-op with exit 0 — operators saw a
green run while every TAG fell through to the same 'no eligible
version' skip message used for legitimate manifest-collision skips.
- Reorder: fetch GHCR versions BEFORE any git-tag deletion. Git-tag
delete is irrecoverable (next run rebuilds TO_PRUNE from 'git tag
-l', so an orphan GHCR image is never enumerated again). Fetching
first means an API failure aborts cleanly with no state change.
- Track PRUNE_FAILED flag. 'git push --delete' fallback is no longer
unconditional — local 'git tag -d' is gated on successful remote
push, so a refused remote delete (tag-protection rule, missing
contents:write) leaves the local tag in place for retry. The flag
propagates to a final 'exit 1' so the cron run turns red on any
push or DELETE failure.
lint-workflows.yml:
- shellcheck step now uses 'find scripts/ops -type f -name *.sh' to
match the workflow's recursive 'scripts/ops/**.sh' path filter. The
previous bare 'scripts/ops/*.sh' glob only matched top-level files;
a future script under a subdirectory would have triggered the
workflow but never been linted.
* docs(releasing): document rollback.yml, prune-dev-tags.yml, lint-workflows.yml
Reflects the new operational workflows landing in this release:
- Auto-rollback paragraph in release.yml description (smoke-test job +
rollback-on-smoke-fail → rollback.yml)
- rollback.yml subsection — workflow_call + workflow_dispatch entry
points, walk-back target resolution, immutability + concurrency
guarantees, manual operator gh workflow run examples
- prune-dev-tags.yml subsection — weekly cron, KEEP_MONTHS retention
semantics, floating-alias safety, dry_run preview, failure-propagation
exit-non-zero behavior
- lint-workflows.yml CI quirk — actionlint (continue-on-error) +
shellcheck (--severity=warning blocking) advisory checks
CLAUDE.md non-negotiable rules unchanged — still high-level and
correct (changelog discipline + release-cut belongs to the PR + run the
full test suite).
187 lines
8.2 KiB
TOML
187 lines
8.2 KiB
TOML
[project]
|
||
name = "agnes-the-ai-analyst"
|
||
version = "0.54.17"
|
||
description = "Agnes — AI Data Analyst platform for AI analytical systems"
|
||
requires-python = ">=3.11,<3.14"
|
||
license = "MIT"
|
||
readme = "README.md"
|
||
|
||
dependencies = [
|
||
# Core database
|
||
# 1.5.2 fixes a FK-dependency regression that affected ALTER TABLE on
|
||
# tables referenced by other tables — broke the test_db migration
|
||
# ladder replay on 1.5.1. CI runs 1.5.2; local devs need it too.
|
||
"duckdb>=1.5.2",
|
||
# Web framework (FastAPI)
|
||
"fastapi>=0.115.0",
|
||
"uvicorn[standard]>=0.32.0",
|
||
"python-multipart>=0.0.27",
|
||
"jinja2>=3.1.0",
|
||
"starlette>=0.41.0",
|
||
# Authentication
|
||
"PyJWT>=2.8.0",
|
||
"itsdangerous>=2.1.0",
|
||
"authlib>=1.6.12",
|
||
"argon2-cffi>=23.1.0",
|
||
# HTTP client. `h2` enables HTTP/2 multiplexing for the persistent
|
||
# CLI client used by `agnes pull` (one TCP connection serves N
|
||
# concurrent parquet streams + range chunks). `cli/client.py`
|
||
# gracefully falls back to HTTP/1.1 if h2 is missing, so this
|
||
# extra is for performance, not correctness.
|
||
"httpx>=0.27.0",
|
||
"h2>=4.1.0",
|
||
# CLI
|
||
"typer>=0.12.0",
|
||
"rich>=13.0.0",
|
||
# Configuration
|
||
"python-dotenv>=1.0.0",
|
||
"pyyaml>=6.0",
|
||
# Data processing
|
||
"pandas>=2.0.0",
|
||
"pyarrow>=12.0.0",
|
||
"pytz>=2024.1",
|
||
# SQL parsing — server-side WHERE validator for /api/v2/scan (app/api/where_validator.py)
|
||
# Minimum 30.x — older versions had walk() yielding (node, parent, key)
|
||
# tuples instead of expression nodes, which would silently bypass the
|
||
# WHERE-validator structural checks (isinstance(tuple, exp.Subquery)
|
||
# is always False). 30.x yields nodes directly.
|
||
"sqlglot>=30.0.0",
|
||
# Data source connectors
|
||
"google-cloud-bigquery>=3.0.0",
|
||
"google-cloud-bigquery-storage>=2.0.0",
|
||
# Google Workspace Cloud Identity / Admin SDK (Workspace group membership sync)
|
||
"google-api-python-client>=2.0.0",
|
||
# Profiler visualizations
|
||
"matplotlib>=3.8.0",
|
||
"numpy>=1.24.0",
|
||
# Claude Code marketplace endpoint — pure-Python git server mounted in FastAPI
|
||
"dulwich>=0.22.0",
|
||
"a2wsgi>=1.10.0",
|
||
# In-process TTL cache for marketplace etag (transitively present via
|
||
# google-auth, declared explicitly here because we depend on it directly).
|
||
"cachetools>=5.3.0",
|
||
# Per-IP rate limiting on auth endpoints (#45). In-process counters by
|
||
# default — fine for single-replica deploys. Multi-replica rollouts can
|
||
# swap the storage backend via slowapi's `storage_uri` (Redis, Memcached).
|
||
"slowapi>=0.1.9",
|
||
# LLM provider SDKs — core (not dev) because connectors/llm/*_provider.py
|
||
# is imported by services/{corporate_memory, verification_detector} which
|
||
# the scheduler drives in production. Promoted from [dev] in #176 to fix
|
||
# ModuleNotFoundError boot loops on default Compose deploys.
|
||
"anthropic>=0.30.0",
|
||
"openai>=1.30.0",
|
||
# Keboola Storage API SDK — used by:
|
||
# - `connectors/keboola/client.py` for admin-side bucket / table list
|
||
# (consumed from `app/api/admin.py` discover-and-register, table
|
||
# metadata refresh).
|
||
# Extraction itself uses the lightweight `connectors/keboola/storage_api.py`
|
||
# module (export-async + signed-URL download) which talks to Storage API
|
||
# directly via `requests` — no SDK dependency on the data-path side. The
|
||
# SDK stays for the metadata reads.
|
||
#
|
||
# NOTE: kbcstorage moved to the [server] extra below — see the rationale
|
||
# in [project.optional-dependencies].server. CLI wheels installed via
|
||
# `uv tool install` deliberately ship without it.
|
||
"sse-starlette>=2.0",
|
||
# Optional observability — pure-Python, no compilation. Lazily initialized
|
||
# in src/observability/posthog_client.py and only emits events when
|
||
# POSTHOG_API_KEY is set in the environment. With the key unset the
|
||
# integration is fully off (no network, no init). See docs/observability.md.
|
||
"posthog>=3.7.0",
|
||
# Rust-backed (ammonia) HTML sanitizer for admin-edited rich content
|
||
# (news intro + body, curated marketplace-metadata.json descriptions).
|
||
# Allowlist-based with per-tag attribute scoping; closes the bypass
|
||
# shapes the legacy regex sanitizer in src/welcome_template.py was
|
||
# vulnerable to. Pre-built wheels published for all supported
|
||
# (mac/linux/windows × arm64/x86_64) targets.
|
||
"nh3>=0.2",
|
||
# CommonMark markdown renderer for curator-authored marketplace-metadata.json
|
||
# rich content (plugin description / sample_interaction.assistant). Pure
|
||
# Python, no compilation. Rendered output is funneled through nh3 above.
|
||
"markdown-it-py>=3.0",
|
||
# Cross-platform advisory file locking for the `agnes push` single-instance
|
||
# guard. Wraps fcntl.flock on POSIX and msvcrt.locking on Windows behind
|
||
# a uniform API; OS releases the lock automatically on process exit (no
|
||
# stale-lock detection required). Used by cli/lib/push_lock.py.
|
||
"filelock>=3.13,<4",
|
||
# Transitive dependency hardened directly to dodge 5 dependabot advisories
|
||
# (4 high, 1 medium) flagged on urllib3<2.7.0: cross-origin sensitive
|
||
# header leak on proxied low-level redirects, decompression-bomb bypass
|
||
# + unbounded decompression chain on the streaming API, redirects-when-
|
||
# retries-disabled. The `[server]` extra below adds kbcstorage which
|
||
# transitively caps urllib3<2.0.0; `[tool.uv] override-dependencies`
|
||
# forces 2.7+ in workspace installs (Dockerfile + dev). Wheel consumers
|
||
# who install only the CLI (`uv tool install <wheel>`) get no kbcstorage
|
||
# and no conflict.
|
||
"urllib3>=2.7.0",
|
||
]
|
||
|
||
[project.optional-dependencies]
|
||
# Server-side connectors. The CLI wheel does NOT need these — analysts who
|
||
# `uv tool install` the wheel never reach a kbcstorage import. Splitting it
|
||
# out keeps the wheel's METADATA `Requires-Dist` set free of the
|
||
# `kbcstorage<=0.9.5 → urllib3<2.0.0` cap that conflicts with our
|
||
# `urllib3>=2.7.0` security pin under any fresh resolver context (where
|
||
# `[tool.uv] override-dependencies` does NOT apply — see comment on
|
||
# [tool.uv] below). Server install pulls it in via Dockerfile's
|
||
# `uv pip install --system --no-cache .[server]`.
|
||
server = [
|
||
"kbcstorage>=0.9.0",
|
||
]
|
||
observability = [
|
||
# Already in base dependencies — listed here so operators who want to
|
||
# be explicit can `pip install -e ".[observability]"` and signal intent.
|
||
"posthog>=3.7.0",
|
||
]
|
||
dev = [
|
||
"pytest>=9.0.0",
|
||
"pytest-timeout>=2.0.0",
|
||
"pytest-xdist>=3.0.0",
|
||
# pytest-split shards the suite across parallel CI jobs (`--splits N
|
||
# --group K`); see the `test-shard` matrix in `.github/workflows/ci.yml`.
|
||
# Balanced by the committed `.test_durations` file.
|
||
"pytest-split>=0.9.0",
|
||
"faker>=24.0.0",
|
||
# jsonschema validates the corporate-memory extraction-tool golden fixtures
|
||
# under tests/test_corporate_memory_v1.py (extraction.json, correction.json,
|
||
# confidence_calibration.json). Production code does not depend on it.
|
||
"jsonschema>=4.0.0",
|
||
# FastAPI debug toolbar — gated behind DEBUG=1 env var in app/main.py.
|
||
# Provides per-request panels (headers, routes, timer, profiling, etc.)
|
||
# for local development. Never loaded in production (no DEBUG=1 there).
|
||
"fastapi-debug-toolbar>=0.6.3",
|
||
]
|
||
|
||
[project.scripts]
|
||
agnes = "cli.main:main"
|
||
|
||
[build-system]
|
||
requires = ["hatchling"]
|
||
build-backend = "hatchling.build"
|
||
|
||
[tool.hatch.build.targets.wheel]
|
||
packages = ["app", "src", "connectors", "cli", "services", "config"]
|
||
|
||
[tool.ruff]
|
||
line-length = 120
|
||
target-version = "py313"
|
||
|
||
[tool.uv]
|
||
dev-dependencies = [
|
||
"pytest>=9.0.0",
|
||
"pytest-timeout>=2.0.0",
|
||
"pytest-xdist>=3.0.0",
|
||
"pytest-split>=0.9.0",
|
||
"faker>=24.0.0",
|
||
"anthropic>=0.30.0",
|
||
"openai>=1.30.0",
|
||
"fastapi-debug-toolbar>=0.6.3",
|
||
]
|
||
# Override the urllib3<2.0.0 ceiling kbcstorage 0.9.5 declares (upstream
|
||
# hasn't relaxed it as of 2026-05-12 but the SDK works fine against
|
||
# urllib3 2.x in practice — we only use `Client` + `Tables` from it and
|
||
# both go through `requests`, which natively supports both lines). Lets
|
||
# the resolver pick a urllib3 line that closes Dependabot advisories
|
||
# CVE-2024-37891 / CVE-2025-{xxx}. See `urllib3>=2.7.0` in [project]
|
||
# dependencies above for the security rationale.
|
||
override-dependencies = ["urllib3>=2.7.0"]
|