107195730d
* feat(observability): optional PostHog integration (errors, LLM traces, replay, flags)
Off by default. Activates when POSTHOG_API_KEY is set in env. Defaults
to PostHog Cloud EU; override host for US Cloud or self-hosted.
Coverage:
- FastAPI 500 handler captures unhandled exceptions
- src/orchestrator.py rebuild + rebuild_source failures
- services/scheduler/ HTTP-job failures
- cli/main.py uncaught CLI errors (Typer.Exit/SystemExit/KeyboardInterrupt
skipped; flushes before re-raise so short-lived CLI invocations don't
drop events)
- connectors/llm/anthropic_provider.py + openai_compat.py emit
$ai_generation events with provider, model, latency, token counts
(prompt/completion bodies stay off unless POSTHOG_LLM_PAYLOADS=1
because LLM prompts here routinely include customer SQL/data)
- Browser snippet injected into every text/html response by
PosthogInjectionMiddleware — registered inside the GZip layer so it
sees uncompressed HTML before compression. Many templates are
standalone (their own DOCTYPE) and never extend base.html, so a
per-template include would miss them.
- Frontend: $pageview, $pageleave, JS error capture via window.error
and unhandledrejection handlers, masked session replay
(maskAllInputs: true plus CSS-selector mask for known data surfaces),
feature flags (browser posthog.isFeatureEnabled + server-side
feature_enabled with fallback for older SDKs).
Identification mode operator-configurable: none / id / email / full.
Default email ships user.id + email but never name. CLI entry point
moves from cli.main:app to cli.main:main (Typer wrapper).
Files:
- src/observability/posthog_client.py — lazy singleton, no network
when disabled, single-process flush on shutdown
- src/observability/llm_tracing.py — trace_generation context manager
- app/middleware/posthog_inject.py — HTML rewrite middleware
- app/web/templates/_posthog.html — browser snippet template
- docs/observability.md — operator guide
- config/.env.template — documented POSTHOG_* knobs
- tests/test_posthog_disabled.py + tests/test_posthog_client.py +
tests/test_llm_tracing.py — 18 tests covering disabled state,
identify-mode payloads, $ai_generation shape, error variant.
CHANGELOG entry under [Unreleased] Added.
* feat(observability): tag every PostHog event with environment + release
Splits PostHog dashboards cleanly between localhost / dev / staging /
production without manual tagging on every capture call.
- POSTHOG_ENVIRONMENT explicit override; auto-resolves to "local" when
LOCAL_DEV_MODE=1, else RELEASE_CHANNEL, else AGNES_DEPLOYMENT_ENV,
else "unknown".
- AGNES_VERSION → RELEASE_CHANNEL fallback feeds the `release` property
for "is this error new in this release?" cohorting.
- Backend gets both via the PostHog SDK's super_properties constructor
arg (every captured event picks them up automatically).
- Browser snippet calls posthog.register({environment, release}) inside
the loaded callback so $pageview, $exception, autocapture, etc. all
carry the same labels.
- request.state.user now populated by auth dependencies so the snippet
can actually call posthog.identify(user_id, {email}) for logged-in
users (previously the user block always resolved to None because
nothing wrote to request.state.user).
4 new tests cover env resolution: explicit > LOCAL_DEV_MODE > channel
> unknown, plus super-properties forwarding into the SDK constructor.
* feat(observability): inline user attrs on every PostHog event + debug throw route
PostHog's UI shows person properties on the Person profile page, not
inline on each event — so a reviewer triaging an exception couldn't tell
which user hit the bug without clicking through. Fix it on both sides.
- Backend capture_exception merges user_id / user_email / user_name into
the event properties (gated by POSTHOG_IDENTIFY_PII: none/id/email/full).
Backed by a new _user_props_for_event helper on PosthogClient.
- Browser snippet registers user_id + user_email + user_name as super-
properties via posthog.register({...}) so every $exception, $pageview,
and custom event coming from posthog.captureException() carries them
inline. Mirrors the backend so cross-referencing client/server events
doesn't require a person-profile lookup.
- /api/debug/throw — debug-only endpoint gated by DEBUG=1 (404 in prod).
Runs Depends(get_current_user) first so request.state.user is set when
the unhandled-exception handler captures the event. Lets operators
exercise the full observability path end-to-end without hand-rolling
a TestClient script. Configurable via ?kind=ValueError&msg=...
7 new tests cover: backend user-attr merge across identify modes,
anonymous request fall-through, browser snippet super-prop emission for
logged-in / anonymous / id-only / full-name cases.
* fix(observability): address minasarustamyan PR #231 review
Two bugs caught in review.
1. PosthogInjectionMiddleware dropped Response.background on every
return path. BaseHTTPMiddleware materialises the body and asks
subclasses to return a fresh Response — three paths in dispatch()
omitted background=, silently cancelling any BackgroundTask /
BackgroundTasks the route attached (audit logging, async webhooks,
email sends) with no log line. Fix: route every return through a
_passthrough() helper that forwards background.
Also adds a _MAX_BUFFER_BYTES (4 MB) cap so a streamed-HTML response
can't balloon RSS during buffering. Bigger bodies short-circuit
through with a warning rather than being injected.
Regression tests in tests/test_posthog_inject_middleware.py exercise
four return paths (snippet present, render-fail, double-injection
guard, non-HTML passthrough) plus the streaming-guard short-circuit.
2. $ai_input / $ai_output_choices were emitted without truncation, so
POSTHOG_LLM_PAYLOADS=1 silently dropped events past PostHog's ~32 KB
per-event ingest limit — exactly the calls (large prompts with
schemas / sample rows / SQL) an operator would want to inspect.
Fix: clip both at POSTHOG_LLM_PAYLOAD_MAX_CHARS (default 30000) with
an explicit "…[truncated N chars]" marker so readers don't mistake
truncated captures for complete ones. Metadata (provider, model,
tokens, latency, error) flows regardless. Three new tests cover
default-cap clipping, env-override, and pass-through under the cap.
37 PostHog tests pass.
77 lines
2.4 KiB
Python
77 lines
2.4 KiB
Python
"""PostHog integration is fully off when POSTHOG_API_KEY is unset.
|
|
|
|
Pins the contract: zero network, zero side effects, snippet renders empty.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
from unittest.mock import patch
|
|
|
|
import pytest
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _clear_posthog_env(monkeypatch):
|
|
"""Drop every POSTHOG_* var so each test starts from disabled state."""
|
|
for key in list(os.environ):
|
|
if key.startswith("POSTHOG_"):
|
|
monkeypatch.delenv(key, raising=False)
|
|
from src.observability import reset_posthog
|
|
reset_posthog()
|
|
yield
|
|
reset_posthog()
|
|
|
|
|
|
def test_get_posthog_returns_disabled_singleton():
|
|
from src.observability import get_posthog
|
|
|
|
pc = get_posthog()
|
|
|
|
assert pc.enabled is False
|
|
# singleton — second access returns same object
|
|
assert get_posthog() is pc
|
|
|
|
|
|
def test_disabled_client_never_constructs_underlying_sdk():
|
|
"""With key unset, posthog.Posthog() must never be instantiated."""
|
|
with patch("posthog.Posthog") as posthog_ctor:
|
|
from src.observability import reset_posthog, get_posthog
|
|
reset_posthog()
|
|
pc = get_posthog()
|
|
|
|
# Trigger every public method that could touch the SDK.
|
|
pc.capture("custom_event", "user-1", {"k": "v"})
|
|
pc.capture_exception(RuntimeError("boom"), distinct_id="user-1")
|
|
assert pc.is_feature_enabled("flag", "user-1", default=False) is False
|
|
assert pc.get_feature_flag_payload("flag", "user-1") is None
|
|
pc.shutdown()
|
|
|
|
posthog_ctor.assert_not_called()
|
|
|
|
|
|
def test_template_global_reports_disabled():
|
|
"""The Jinja-side `posthog_config` flag must be False when off."""
|
|
from app.web.router import _posthog_config_global
|
|
|
|
cfg = _posthog_config_global()
|
|
|
|
assert cfg == {"enabled": False}
|
|
|
|
|
|
def test_template_user_block_returns_none_when_disabled():
|
|
from app.web.router import _posthog_user_block
|
|
|
|
assert _posthog_user_block(None) is None
|
|
|
|
|
|
def test_trace_generation_yields_noop_when_disabled():
|
|
"""The LLM-tracing context manager must not call into PostHog when off."""
|
|
from src.observability import trace_generation
|
|
|
|
with patch("src.observability.posthog_client.PosthogClient.capture") as cap:
|
|
with trace_generation(provider="anthropic", model="claude-test") as t:
|
|
t.set_input("hello")
|
|
t.set_tokens(input_tokens=10, output_tokens=20)
|
|
|
|
cap.assert_not_called()
|