agnes-the-ai-analyst/cli/main.py
Vojtech 107195730d
feat(observability): optional PostHog integration (#231)
* feat(observability): optional PostHog integration (errors, LLM traces, replay, flags)

Off by default. Activates when POSTHOG_API_KEY is set in env. Defaults
to PostHog Cloud EU; override host for US Cloud or self-hosted.

Coverage:
  - FastAPI 500 handler captures unhandled exceptions
  - src/orchestrator.py rebuild + rebuild_source failures
  - services/scheduler/ HTTP-job failures
  - cli/main.py uncaught CLI errors (Typer.Exit/SystemExit/KeyboardInterrupt
    skipped; flushes before re-raise so short-lived CLI invocations don't
    drop events)
  - connectors/llm/anthropic_provider.py + openai_compat.py emit
    $ai_generation events with provider, model, latency, token counts
    (prompt/completion bodies stay off unless POSTHOG_LLM_PAYLOADS=1
    because LLM prompts here routinely include customer SQL/data)
  - Browser snippet injected into every text/html response by
    PosthogInjectionMiddleware — registered inside the GZip layer so it
    sees uncompressed HTML before compression. Many templates are
    standalone (their own DOCTYPE) and never extend base.html, so a
    per-template include would miss them.
  - Frontend: $pageview, $pageleave, JS error capture via window.error
    and unhandledrejection handlers, masked session replay
    (maskAllInputs: true plus CSS-selector mask for known data surfaces),
    feature flags (browser posthog.isFeatureEnabled + server-side
    feature_enabled with fallback for older SDKs).

Identification mode operator-configurable: none / id / email / full.
Default email ships user.id + email but never name. CLI entry point
moves from cli.main:app to cli.main:main (Typer wrapper).

Files:
  - src/observability/posthog_client.py — lazy singleton, no network
    when disabled, single-process flush on shutdown
  - src/observability/llm_tracing.py — trace_generation context manager
  - app/middleware/posthog_inject.py — HTML rewrite middleware
  - app/web/templates/_posthog.html — browser snippet template
  - docs/observability.md — operator guide
  - config/.env.template — documented POSTHOG_* knobs
  - tests/test_posthog_disabled.py + tests/test_posthog_client.py +
    tests/test_llm_tracing.py — 18 tests covering disabled state,
    identify-mode payloads, $ai_generation shape, error variant.

CHANGELOG entry under [Unreleased] Added.

* feat(observability): tag every PostHog event with environment + release

Splits PostHog dashboards cleanly between localhost / dev / staging /
production without manual tagging on every capture call.

- POSTHOG_ENVIRONMENT explicit override; auto-resolves to "local" when
  LOCAL_DEV_MODE=1, else RELEASE_CHANNEL, else AGNES_DEPLOYMENT_ENV,
  else "unknown".
- AGNES_VERSION → RELEASE_CHANNEL fallback feeds the `release` property
  for "is this error new in this release?" cohorting.
- Backend gets both via the PostHog SDK's super_properties constructor
  arg (every captured event picks them up automatically).
- Browser snippet calls posthog.register({environment, release}) inside
  the loaded callback so $pageview, $exception, autocapture, etc. all
  carry the same labels.
- request.state.user now populated by auth dependencies so the snippet
  can actually call posthog.identify(user_id, {email}) for logged-in
  users (previously the user block always resolved to None because
  nothing wrote to request.state.user).

4 new tests cover env resolution: explicit > LOCAL_DEV_MODE > channel
> unknown, plus super-properties forwarding into the SDK constructor.

* feat(observability): inline user attrs on every PostHog event + debug throw route

PostHog's UI shows person properties on the Person profile page, not
inline on each event — so a reviewer triaging an exception couldn't tell
which user hit the bug without clicking through. Fix it on both sides.

- Backend capture_exception merges user_id / user_email / user_name into
  the event properties (gated by POSTHOG_IDENTIFY_PII: none/id/email/full).
  Backed by a new _user_props_for_event helper on PosthogClient.
- Browser snippet registers user_id + user_email + user_name as super-
  properties via posthog.register({...}) so every $exception, $pageview,
  and custom event coming from posthog.captureException() carries them
  inline. Mirrors the backend so cross-referencing client/server events
  doesn't require a person-profile lookup.
- /api/debug/throw — debug-only endpoint gated by DEBUG=1 (404 in prod).
  Runs Depends(get_current_user) first so request.state.user is set when
  the unhandled-exception handler captures the event. Lets operators
  exercise the full observability path end-to-end without hand-rolling
  a TestClient script. Configurable via ?kind=ValueError&msg=...

7 new tests cover: backend user-attr merge across identify modes,
anonymous request fall-through, browser snippet super-prop emission for
logged-in / anonymous / id-only / full-name cases.

* fix(observability): address minasarustamyan PR #231 review

Two bugs caught in review.

1. PosthogInjectionMiddleware dropped Response.background on every
   return path. BaseHTTPMiddleware materialises the body and asks
   subclasses to return a fresh Response — three paths in dispatch()
   omitted background=, silently cancelling any BackgroundTask /
   BackgroundTasks the route attached (audit logging, async webhooks,
   email sends) with no log line. Fix: route every return through a
   _passthrough() helper that forwards background.

   Also adds a _MAX_BUFFER_BYTES (4 MB) cap so a streamed-HTML response
   can't balloon RSS during buffering. Bigger bodies short-circuit
   through with a warning rather than being injected.

   Regression tests in tests/test_posthog_inject_middleware.py exercise
   four return paths (snippet present, render-fail, double-injection
   guard, non-HTML passthrough) plus the streaming-guard short-circuit.

2. $ai_input / $ai_output_choices were emitted without truncation, so
   POSTHOG_LLM_PAYLOADS=1 silently dropped events past PostHog's ~32 KB
   per-event ingest limit — exactly the calls (large prompts with
   schemas / sample rows / SQL) an operator would want to inspect.
   Fix: clip both at POSTHOG_LLM_PAYLOAD_MAX_CHARS (default 30000) with
   an explicit "…[truncated N chars]" marker so readers don't mistake
   truncated captures for complete ones. Metadata (provider, model,
   tokens, latency, error) flows regardless. Three new tests cover
   default-cap clipping, env-override, and pass-through under the cap.

37 PostHog tests pass.
2026-05-08 17:57:10 +04:00

199 lines
7.2 KiB
Python

"""agnes — CLI tool for AI Data Analyst.
Primary interface for AI agents. Install: uv tool install agnes-the-ai-analyst
"""
import sys
from importlib.metadata import PackageNotFoundError
from importlib.metadata import version as _pkg_version
import typer
# Force UTF-8 on Windows stdout/stderr at import time. The default Windows
# console codepage (cp1250 on cs-CZ, cp1252 on en-US, …) cannot encode the
# Braille spinner glyphs Rich uses for `agnes pull` progress, nor the
# em-dash / accented chars that show up in skill markdown via
# `agnes skills list`. Both crash with UnicodeEncodeError /
# UnicodeDecodeError before any command-level code runs. `reconfigure` is
# a no-op on non-TextIOWrapper streams (pytest capture, pipes wrapped by
# other tooling) — swallow the AttributeError there.
if sys.platform == "win32":
try:
sys.stdout.reconfigure(encoding="utf-8", errors="replace")
sys.stderr.reconfigure(encoding="utf-8", errors="replace")
except (AttributeError, OSError):
pass
from cli.commands.auth import auth_app
from cli.commands.init import init_app
from cli.commands.pull import pull_app
from cli.commands.push import push_app
from cli.commands.refresh_marketplace import refresh_marketplace_app
from cli.commands.query import query_command
from cli.commands.status import status_app
from cli.commands.admin import admin_app
from cli.commands.diagnose import diagnose_app
from cli.commands.skills import skills_app
from cli.commands.self_upgrade import self_upgrade_app
from cli.commands.setup import setup_app
from cli.commands.server import server_app
from cli.commands.explore import explore_app
from cli.commands.catalog import catalog_app
from cli.commands.schema import schema_app
from cli.commands.describe import describe
from cli.commands.snapshot import snapshot_app
from cli.commands.disk_info import disk_info_app
from cli.commands.store import store_app
from cli.commands.my_stack import my_stack_app
def _cli_version() -> str:
"""Return the installed CLI version from package metadata.
Falls back to `"unknown"` when the package is not installed (e.g. running
from a source checkout without `uv pip install -e .`). Deliberately does
not read pyproject.toml at runtime — that file is not shipped with the
wheel and the metadata lookup is the canonical source.
"""
try:
return _pkg_version("agnes-the-ai-analyst")
except PackageNotFoundError:
return "unknown"
def _version_callback(value: bool) -> None:
if value:
typer.echo(f"agnes {_cli_version()}")
raise typer.Exit()
app = typer.Typer(
name="agnes",
help="Agnes — AI Data Analyst CLI",
no_args_is_help=True,
)
@app.callback()
def _root(
version: bool = typer.Option(
None,
"--version",
"-V",
callback=_version_callback,
is_eager=True,
help="Show the CLI version and exit.",
),
) -> None:
"""Root callback — carries the --version option and fires the auto-update check.
Update check runs before subcommand dispatch but after the --version flag
(which exits early). It's best-effort: any failure is swallowed so a bad
network never blocks a working `agnes` command. Disable with
`AGNES_NO_UPDATE_CHECK=1`.
"""
_maybe_warn_outdated()
def _maybe_warn_outdated() -> None:
"""Hit /cli/latest on the configured server (cached 24h) and emit a
one-line stderr warning if the installed CLI is older. Never raises."""
try:
from cli.config import get_server_url
from cli.update_check import check, format_outdated_notice
info = check(get_server_url())
if info and info.is_outdated():
typer.echo(format_outdated_notice(info), err=True)
except Exception:
pass # best-effort: never fail a command on the probe
# Register subcommands
app.add_typer(auth_app, name="auth")
app.add_typer(init_app, name="init")
app.add_typer(pull_app, name="pull")
app.add_typer(push_app, name="push")
app.add_typer(refresh_marketplace_app, name="refresh-marketplace")
app.command("query")(query_command)
app.add_typer(status_app, name="status")
app.add_typer(admin_app, name="admin")
app.add_typer(diagnose_app, name="diagnose")
app.add_typer(skills_app, name="skills")
app.add_typer(self_upgrade_app, name="self-upgrade")
app.add_typer(setup_app, name="setup")
app.add_typer(server_app, name="server")
app.add_typer(explore_app, name="explore")
app.add_typer(catalog_app, name="catalog")
app.add_typer(schema_app, name="schema")
app.command("describe")(describe)
app.add_typer(snapshot_app, name="snapshot")
app.add_typer(disk_info_app, name="disk-info")
app.add_typer(store_app, name="store")
app.add_typer(my_stack_app, name="my-stack")
def _capture_cli_exception(exc: BaseException, kind: str) -> None:
"""Best-effort PostHog forward for CLI-level errors. No-op when off."""
try:
from src.observability import get_posthog
argv = sys.argv[1:]
command = argv[0] if argv else "<no-command>"
get_posthog().capture_exception(
exc,
distinct_id="cli",
properties={
"component": "cli",
"command": command,
"argv": " ".join(argv)[:512],
"error_kind": kind,
},
)
get_posthog().shutdown()
except Exception:
pass # never replace the user-visible error with a tracing failure
def main() -> None:
"""Wrap ``app()`` so AgnesTransportError (and other typed CLI errors)
surface as a one-line message + exit, never as a Python traceback. The
full traceback is already logged to ``~/.config/agnes/last-error.log``
by the api_* helpers — operators read it from there for support
forwarding. Anything that escapes this wrapper IS a CLI bug worth
fixing — log + print "internal error" so the analyst doesn't see a
Pythonist's traceback either.
Also forwards captured exceptions to PostHog (no-op when disabled) so
operators can see CLI-level failures alongside server-side ones.
Normal control-flow exits (typer.Exit / SystemExit / KeyboardInterrupt)
are never reported.
Pavel's #185 Phase 3B: previously a `httpx.ReadTimeout` from an
`agnes query --remote` against a slow BQ view dumped a 30-frame
traceback to the analyst's terminal. Now: one clean line + a hint,
return code 1.
"""
from cli.client import AgnesTransportError, _log_traceback, _LOG_FILE
try:
app()
except AgnesTransportError as exc:
_capture_cli_exception(exc, kind="transport")
typer.echo(f"Error: {exc.user_message}", err=True)
if exc.hint:
typer.echo(exc.hint, err=True)
sys.exit(1)
except typer.Exit:
raise
except (KeyboardInterrupt, SystemExit):
raise
except Exception as exc: # last-resort net — escaped exceptions are bugs
_capture_cli_exception(exc, kind="unhandled")
log = _log_traceback(exc, context="unhandled at CLI top-level")
typer.echo(
f"Error: internal CLI error ({type(exc).__name__}). "
f"Full traceback logged to {log}.",
err=True,
)
sys.exit(1)
if __name__ == "__main__":
main()