* feat(observability): optional PostHog integration (errors, LLM traces, replay, flags)
Off by default. Activates when POSTHOG_API_KEY is set in env. Defaults
to PostHog Cloud EU; override host for US Cloud or self-hosted.
Coverage:
- FastAPI 500 handler captures unhandled exceptions
- src/orchestrator.py rebuild + rebuild_source failures
- services/scheduler/ HTTP-job failures
- cli/main.py uncaught CLI errors (Typer.Exit/SystemExit/KeyboardInterrupt
skipped; flushes before re-raise so short-lived CLI invocations don't
drop events)
- connectors/llm/anthropic_provider.py + openai_compat.py emit
$ai_generation events with provider, model, latency, token counts
(prompt/completion bodies stay off unless POSTHOG_LLM_PAYLOADS=1
because LLM prompts here routinely include customer SQL/data)
- Browser snippet injected into every text/html response by
PosthogInjectionMiddleware — registered inside the GZip layer so it
sees uncompressed HTML before compression. Many templates are
standalone (their own DOCTYPE) and never extend base.html, so a
per-template include would miss them.
- Frontend: $pageview, $pageleave, JS error capture via window.error
and unhandledrejection handlers, masked session replay
(maskAllInputs: true plus CSS-selector mask for known data surfaces),
feature flags (browser posthog.isFeatureEnabled + server-side
feature_enabled with fallback for older SDKs).
Identification mode operator-configurable: none / id / email / full.
Default email ships user.id + email but never name. CLI entry point
moves from cli.main:app to cli.main:main (Typer wrapper).
Files:
- src/observability/posthog_client.py — lazy singleton, no network
when disabled, single-process flush on shutdown
- src/observability/llm_tracing.py — trace_generation context manager
- app/middleware/posthog_inject.py — HTML rewrite middleware
- app/web/templates/_posthog.html — browser snippet template
- docs/observability.md — operator guide
- config/.env.template — documented POSTHOG_* knobs
- tests/test_posthog_disabled.py + tests/test_posthog_client.py +
tests/test_llm_tracing.py — 18 tests covering disabled state,
identify-mode payloads, $ai_generation shape, error variant.
CHANGELOG entry under [Unreleased] Added.
* feat(observability): tag every PostHog event with environment + release
Splits PostHog dashboards cleanly between localhost / dev / staging /
production without manual tagging on every capture call.
- POSTHOG_ENVIRONMENT explicit override; auto-resolves to "local" when
LOCAL_DEV_MODE=1, else RELEASE_CHANNEL, else AGNES_DEPLOYMENT_ENV,
else "unknown".
- AGNES_VERSION → RELEASE_CHANNEL fallback feeds the `release` property
for "is this error new in this release?" cohorting.
- Backend gets both via the PostHog SDK's super_properties constructor
arg (every captured event picks them up automatically).
- Browser snippet calls posthog.register({environment, release}) inside
the loaded callback so $pageview, $exception, autocapture, etc. all
carry the same labels.
- request.state.user now populated by auth dependencies so the snippet
can actually call posthog.identify(user_id, {email}) for logged-in
users (previously the user block always resolved to None because
nothing wrote to request.state.user).
4 new tests cover env resolution: explicit > LOCAL_DEV_MODE > channel
> unknown, plus super-properties forwarding into the SDK constructor.
* feat(observability): inline user attrs on every PostHog event + debug throw route
PostHog's UI shows person properties on the Person profile page, not
inline on each event — so a reviewer triaging an exception couldn't tell
which user hit the bug without clicking through. Fix it on both sides.
- Backend capture_exception merges user_id / user_email / user_name into
the event properties (gated by POSTHOG_IDENTIFY_PII: none/id/email/full).
Backed by a new _user_props_for_event helper on PosthogClient.
- Browser snippet registers user_id + user_email + user_name as super-
properties via posthog.register({...}) so every $exception, $pageview,
and custom event coming from posthog.captureException() carries them
inline. Mirrors the backend so cross-referencing client/server events
doesn't require a person-profile lookup.
- /api/debug/throw — debug-only endpoint gated by DEBUG=1 (404 in prod).
Runs Depends(get_current_user) first so request.state.user is set when
the unhandled-exception handler captures the event. Lets operators
exercise the full observability path end-to-end without hand-rolling
a TestClient script. Configurable via ?kind=ValueError&msg=...
7 new tests cover: backend user-attr merge across identify modes,
anonymous request fall-through, browser snippet super-prop emission for
logged-in / anonymous / id-only / full-name cases.
* fix(observability): address minasarustamyan PR #231 review
Two bugs caught in review.
1. PosthogInjectionMiddleware dropped Response.background on every
return path. BaseHTTPMiddleware materialises the body and asks
subclasses to return a fresh Response — three paths in dispatch()
omitted background=, silently cancelling any BackgroundTask /
BackgroundTasks the route attached (audit logging, async webhooks,
email sends) with no log line. Fix: route every return through a
_passthrough() helper that forwards background.
Also adds a _MAX_BUFFER_BYTES (4 MB) cap so a streamed-HTML response
can't balloon RSS during buffering. Bigger bodies short-circuit
through with a warning rather than being injected.
Regression tests in tests/test_posthog_inject_middleware.py exercise
four return paths (snippet present, render-fail, double-injection
guard, non-HTML passthrough) plus the streaming-guard short-circuit.
2. $ai_input / $ai_output_choices were emitted without truncation, so
POSTHOG_LLM_PAYLOADS=1 silently dropped events past PostHog's ~32 KB
per-event ingest limit — exactly the calls (large prompts with
schemas / sample rows / SQL) an operator would want to inspect.
Fix: clip both at POSTHOG_LLM_PAYLOAD_MAX_CHARS (default 30000) with
an explicit "…[truncated N chars]" marker so readers don't mistake
truncated captures for complete ones. Metadata (provider, model,
tokens, latency, error) flows regardless. Three new tests cover
default-cap clipping, env-override, and pass-through under the cap.
37 PostHog tests pass.
199 lines
7.2 KiB
Python
199 lines
7.2 KiB
Python
"""agnes — CLI tool for AI Data Analyst.
|
|
|
|
Primary interface for AI agents. Install: uv tool install agnes-the-ai-analyst
|
|
"""
|
|
|
|
import sys
|
|
from importlib.metadata import PackageNotFoundError
|
|
from importlib.metadata import version as _pkg_version
|
|
|
|
import typer
|
|
|
|
# Force UTF-8 on Windows stdout/stderr at import time. The default Windows
|
|
# console codepage (cp1250 on cs-CZ, cp1252 on en-US, …) cannot encode the
|
|
# Braille spinner glyphs Rich uses for `agnes pull` progress, nor the
|
|
# em-dash / accented chars that show up in skill markdown via
|
|
# `agnes skills list`. Both crash with UnicodeEncodeError /
|
|
# UnicodeDecodeError before any command-level code runs. `reconfigure` is
|
|
# a no-op on non-TextIOWrapper streams (pytest capture, pipes wrapped by
|
|
# other tooling) — swallow the AttributeError there.
|
|
if sys.platform == "win32":
|
|
try:
|
|
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
|
|
sys.stderr.reconfigure(encoding="utf-8", errors="replace")
|
|
except (AttributeError, OSError):
|
|
pass
|
|
|
|
from cli.commands.auth import auth_app
|
|
from cli.commands.init import init_app
|
|
from cli.commands.pull import pull_app
|
|
from cli.commands.push import push_app
|
|
from cli.commands.refresh_marketplace import refresh_marketplace_app
|
|
from cli.commands.query import query_command
|
|
from cli.commands.status import status_app
|
|
from cli.commands.admin import admin_app
|
|
from cli.commands.diagnose import diagnose_app
|
|
from cli.commands.skills import skills_app
|
|
from cli.commands.self_upgrade import self_upgrade_app
|
|
from cli.commands.setup import setup_app
|
|
from cli.commands.server import server_app
|
|
from cli.commands.explore import explore_app
|
|
from cli.commands.catalog import catalog_app
|
|
from cli.commands.schema import schema_app
|
|
from cli.commands.describe import describe
|
|
from cli.commands.snapshot import snapshot_app
|
|
from cli.commands.disk_info import disk_info_app
|
|
from cli.commands.store import store_app
|
|
from cli.commands.my_stack import my_stack_app
|
|
|
|
|
|
def _cli_version() -> str:
|
|
"""Return the installed CLI version from package metadata.
|
|
|
|
Falls back to `"unknown"` when the package is not installed (e.g. running
|
|
from a source checkout without `uv pip install -e .`). Deliberately does
|
|
not read pyproject.toml at runtime — that file is not shipped with the
|
|
wheel and the metadata lookup is the canonical source.
|
|
"""
|
|
try:
|
|
return _pkg_version("agnes-the-ai-analyst")
|
|
except PackageNotFoundError:
|
|
return "unknown"
|
|
|
|
|
|
def _version_callback(value: bool) -> None:
|
|
if value:
|
|
typer.echo(f"agnes {_cli_version()}")
|
|
raise typer.Exit()
|
|
|
|
|
|
app = typer.Typer(
|
|
name="agnes",
|
|
help="Agnes — AI Data Analyst CLI",
|
|
no_args_is_help=True,
|
|
)
|
|
|
|
|
|
@app.callback()
|
|
def _root(
|
|
version: bool = typer.Option(
|
|
None,
|
|
"--version",
|
|
"-V",
|
|
callback=_version_callback,
|
|
is_eager=True,
|
|
help="Show the CLI version and exit.",
|
|
),
|
|
) -> None:
|
|
"""Root callback — carries the --version option and fires the auto-update check.
|
|
|
|
Update check runs before subcommand dispatch but after the --version flag
|
|
(which exits early). It's best-effort: any failure is swallowed so a bad
|
|
network never blocks a working `agnes` command. Disable with
|
|
`AGNES_NO_UPDATE_CHECK=1`.
|
|
"""
|
|
_maybe_warn_outdated()
|
|
|
|
|
|
def _maybe_warn_outdated() -> None:
|
|
"""Hit /cli/latest on the configured server (cached 24h) and emit a
|
|
one-line stderr warning if the installed CLI is older. Never raises."""
|
|
try:
|
|
from cli.config import get_server_url
|
|
from cli.update_check import check, format_outdated_notice
|
|
info = check(get_server_url())
|
|
if info and info.is_outdated():
|
|
typer.echo(format_outdated_notice(info), err=True)
|
|
except Exception:
|
|
pass # best-effort: never fail a command on the probe
|
|
|
|
# Register subcommands
|
|
app.add_typer(auth_app, name="auth")
|
|
app.add_typer(init_app, name="init")
|
|
app.add_typer(pull_app, name="pull")
|
|
app.add_typer(push_app, name="push")
|
|
app.add_typer(refresh_marketplace_app, name="refresh-marketplace")
|
|
app.command("query")(query_command)
|
|
app.add_typer(status_app, name="status")
|
|
app.add_typer(admin_app, name="admin")
|
|
app.add_typer(diagnose_app, name="diagnose")
|
|
app.add_typer(skills_app, name="skills")
|
|
app.add_typer(self_upgrade_app, name="self-upgrade")
|
|
app.add_typer(setup_app, name="setup")
|
|
app.add_typer(server_app, name="server")
|
|
app.add_typer(explore_app, name="explore")
|
|
app.add_typer(catalog_app, name="catalog")
|
|
app.add_typer(schema_app, name="schema")
|
|
app.command("describe")(describe)
|
|
app.add_typer(snapshot_app, name="snapshot")
|
|
app.add_typer(disk_info_app, name="disk-info")
|
|
app.add_typer(store_app, name="store")
|
|
app.add_typer(my_stack_app, name="my-stack")
|
|
|
|
|
|
def _capture_cli_exception(exc: BaseException, kind: str) -> None:
|
|
"""Best-effort PostHog forward for CLI-level errors. No-op when off."""
|
|
try:
|
|
from src.observability import get_posthog
|
|
argv = sys.argv[1:]
|
|
command = argv[0] if argv else "<no-command>"
|
|
get_posthog().capture_exception(
|
|
exc,
|
|
distinct_id="cli",
|
|
properties={
|
|
"component": "cli",
|
|
"command": command,
|
|
"argv": " ".join(argv)[:512],
|
|
"error_kind": kind,
|
|
},
|
|
)
|
|
get_posthog().shutdown()
|
|
except Exception:
|
|
pass # never replace the user-visible error with a tracing failure
|
|
|
|
|
|
def main() -> None:
|
|
"""Wrap ``app()`` so AgnesTransportError (and other typed CLI errors)
|
|
surface as a one-line message + exit, never as a Python traceback. The
|
|
full traceback is already logged to ``~/.config/agnes/last-error.log``
|
|
by the api_* helpers — operators read it from there for support
|
|
forwarding. Anything that escapes this wrapper IS a CLI bug worth
|
|
fixing — log + print "internal error" so the analyst doesn't see a
|
|
Pythonist's traceback either.
|
|
|
|
Also forwards captured exceptions to PostHog (no-op when disabled) so
|
|
operators can see CLI-level failures alongside server-side ones.
|
|
Normal control-flow exits (typer.Exit / SystemExit / KeyboardInterrupt)
|
|
are never reported.
|
|
|
|
Pavel's #185 Phase 3B: previously a `httpx.ReadTimeout` from an
|
|
`agnes query --remote` against a slow BQ view dumped a 30-frame
|
|
traceback to the analyst's terminal. Now: one clean line + a hint,
|
|
return code 1.
|
|
"""
|
|
from cli.client import AgnesTransportError, _log_traceback, _LOG_FILE
|
|
try:
|
|
app()
|
|
except AgnesTransportError as exc:
|
|
_capture_cli_exception(exc, kind="transport")
|
|
typer.echo(f"Error: {exc.user_message}", err=True)
|
|
if exc.hint:
|
|
typer.echo(exc.hint, err=True)
|
|
sys.exit(1)
|
|
except typer.Exit:
|
|
raise
|
|
except (KeyboardInterrupt, SystemExit):
|
|
raise
|
|
except Exception as exc: # last-resort net — escaped exceptions are bugs
|
|
_capture_cli_exception(exc, kind="unhandled")
|
|
log = _log_traceback(exc, context="unhandled at CLI top-level")
|
|
typer.echo(
|
|
f"Error: internal CLI error ({type(exc).__name__}). "
|
|
f"Full traceback logged to {log}.",
|
|
err=True,
|
|
)
|
|
sys.exit(1)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|