agnes-the-ai-analyst/docs/archive/superpowers/specs/2026-04-29-dev-debug-toolbar-design.md

Dev Debug Toolbar + Centralized Logging — Design

Date: 2026-04-29 Status: Draft (awaiting user review) Owner: vrysanek

Goal

Give developers a Flask-DebugToolbar-style panel UI in the FastAPI app for local dev, plus consistent log output across the FastAPI app and 5 background services. Make it cheap to leave on in dev, free to ship in production.

Two scopes, one design:

1. Debug toolbar in the FastAPI app — visible only in dev, exposing per-request panels: timer, headers, routes, settings, versions, logging records, pyinstrument profiler, and a custom DuckDB queries panel.
2. Centralized logging — replace 20+ scattered logging.basicConfig(...) calls across app/, services/, connectors/, src/, scripts/ with a single setup_logging() helper that uses rich.logging.RichHandler in dev and JSON to stdout in prod.

Out of scope: production observability, log shipping, distributed tracing, structured logging refactor of every logger.info("foo") call site.

---

Non-goals / explicit rejections

• Not building an in-house toolbar (rejected: 1–2 days build time when fastapi-debug-toolbar covers 80% of the need).
• Not introducing loguru or structlog (rejected: requires rewriting 40+ files using stdlib logging).
• Not auto-deriving service slug via sys._getframe(1) magic (rejected after Gemini adversarial review: brittle on PyPy, decorators, indirect calls). Use explicit setup_logging(__name__).
• Not instrumenting DuckDB queries in production (rejected: per-query overhead, only needed for dev panel).

---

Activation

One environment variable: DEBUG.

DEBUG	FastAPI debug=	Toolbar middleware	Logging handler	DuckDB instrumentation
unset / 0	False	not mounted, not imported	StreamHandler + JSON formatter	no-op (raw duckdb.Connection)
1	True	mounted	RichHandler (color, tracebacks, links)	InstrumentedConnection records queries per request

Orthogonal to LOCAL_DEV_MODE (auth bypass) — they do different things, both can be set independently.

---

Architecture

┌───────────────────────────────────────────────────────────────┐
│ FastAPI app process                                           │
│                                                               │
│  app/main.py                                                  │
│    ├── setup_logging("app")                                   │
│    ├── FastAPI(debug=DEBUG)                                   │
│    ├── app.add_middleware(RequestIdMiddleware)                │
│    └── if DEBUG:                                              │
│          app.add_middleware(DebugToolbarMiddleware,           │
│            panels=[Headers, Routes, Settings, Versions,       │
│                    Timer, Logging, Profiling,                 │
│                    ★ DuckDBPanel ★])                          │
│                                                               │
│  app/logging_config.py        ← setup_logging, _JSONFormatter │
│  app/middleware/request_id.py ← RequestIdMiddleware           │
│  app/debug/duckdb_panel.py    ← DuckDBPanel,                  │
│                                  InstrumentedConnection       │
│  src/db.py (modified)         ← _maybe_instrument helper      │
└───────────────────────────────────────────────────────────────┘

Background services (scheduler, telegram_bot, ws_gateway,
                    corporate_memory, session_collector):
  └── each entrypoint: setup_logging(__name__)
      → kills 20+ scattered basicConfig calls
      → unified format (Rich in dev, JSON in prod)

Component boundaries

Component	Public interface	Depends on
app/logging_config.py	setup_logging(service=None, level=None), request_id_var: ContextVar[str|None]	stdlib logging, rich, env vars
app/middleware/request_id.py	RequestIdMiddleware (Starlette BaseHTTPMiddleware)	request_id_var
app/debug/duckdb_panel.py	DuckDBPanel (toolbar Panel), InstrumentedConnection (duckdb.DuckDBPyConnection-compatible), record_query(), get_request_store()	fastapi-debug-toolbar, duckdb, contextvars
src/db.py (changed)	unchanged signatures; internal _maybe_instrument(con, db_tag) returns raw or wrapped connection based on DEBUG	app.debug.duckdb_panel (lazy import behind DEBUG)

A library module never calls setup_logging(). Only entrypoints (__main__.py, main.py, top-level CLI scripts).

---

Detailed design

app/logging_config.py

import contextvars
import json
import logging
import os
import sys
from datetime import datetime, timezone
from pathlib import Path

request_id_var: contextvars.ContextVar[str | None] = contextvars.ContextVar("request_id", default=None)

_CONFIGURED = False


def setup_logging(service: str | None = None, level: str | None = None) -> None:
    """Configure root logger. Idempotent. Call once per process at entrypoint.

    Pass `__name__` (preferred) or an explicit short service slug.
    """
    global _CONFIGURED
    if _CONFIGURED:
        return

    debug = os.environ.get("DEBUG", "").lower() in ("1", "true", "yes")
    lvl = (level or os.environ.get("LOG_LEVEL") or ("DEBUG" if debug else "INFO")).upper()
    slug = _derive_slug(service)

    if debug:
        from rich.logging import RichHandler
        handler = RichHandler(
            rich_tracebacks=True,
            tracebacks_show_locals=False,
            show_time=True,
            show_path=True,
            markup=False,
            force_terminal=True,        # docker compose: stdout not a TTY but still want color
        )
        handler.setFormatter(logging.Formatter("[%(name)s] %(message)s"))
    else:
        handler = logging.StreamHandler()
        handler.setFormatter(_JSONFormatter(service=slug))

    logging.basicConfig(level=lvl, handlers=[handler], force=True)
    logging.getLogger("uvicorn.access").setLevel(logging.INFO if debug else logging.WARNING)
    logging.getLogger("httpx").setLevel(logging.WARNING)
    _CONFIGURED = True


def _derive_slug(service: str | None) -> str:
    """Turn module name (`__name__`) or override into readable service slug.

    setup_logging("app")                           -> "app"
    setup_logging("services.corporate_memory.collector")
                                                   -> "corporate_memory.collector"
    setup_logging("services.scheduler.__main__")   -> "scheduler"
    setup_logging("__main__") (direct script run)  -> derived from caller's __file__
    """
    if service and not service.startswith("_") and service != "__main__":
        s = service.removeprefix("services.").removeprefix("connectors.").removeprefix("app.")
        s = s.removesuffix(".__main__").removesuffix(".main")
        return s or "app"

    # Fallback: parse caller's __file__
    try:
        frame = sys._getframe(2)
        path = frame.f_globals.get("__file__")
        if path:
            p = Path(path)
            for top in ("services", "connectors", "app"):
                if top in p.parts:
                    i = p.parts.index(top) + 1
                    rest = p.parts[i:]
                    name = ".".join([*rest[:-1], p.stem])
                    return name.removesuffix(".__main__").removesuffix(".main") or top
            return p.stem
    except Exception:
        pass
    return "app"


class _JSONFormatter(logging.Formatter):
    def __init__(self, service: str) -> None:
        super().__init__()
        self.service = service

    def format(self, record: logging.LogRecord) -> str:
        payload = {
            "ts": datetime.fromtimestamp(record.created, tz=timezone.utc).isoformat(),
            "lvl": record.levelname,
            "logger": record.name,
            "service": self.service,
            "msg": record.getMessage(),
        }
        rid = request_id_var.get()
        if rid:
            payload["request_id"] = rid
        if record.exc_info:
            payload["exc"] = self.formatException(record.exc_info)
        return json.dumps(payload, default=str)

Adversarial flaws addressed (per Gemini review 2026-04-29):

Flaw	Mitigation
sys._getframe brittleness	Used only as a fallback when caller can't supply __name__; primary path is explicit setup_logging(__name__)
root.handlers.clear() wipes uvicorn handlers	Replaced with logging.basicConfig(force=True) — semantically equivalent but well-defined
RichHandler color-detection in docker	force_terminal=True — color is intentional in dev, dev never goes to log shippers
Path leakage via show_path=True	Dev only. Prod uses _JSONFormatter, no source paths
Non-idempotent	_CONFIGURED sentinel, repeated calls become no-ops

app/middleware/request_id.py

import uuid
from starlette.middleware.base import BaseHTTPMiddleware
from app.logging_config import request_id_var


class RequestIdMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request, call_next):
        rid = request.headers.get("x-request-id") or uuid.uuid4().hex[:12]
        token = request_id_var.set(rid)
        try:
            response = await call_next(request)
        finally:
            request_id_var.reset(token)
        response.headers["x-request-id"] = rid
        return response

Mounted unconditionally (cheap, useful in prod for correlation too). Sits before DebugToolbarMiddleware so request id is set when the toolbar captures.

app/debug/duckdb_panel.py

from __future__ import annotations
import contextvars
import time
from dataclasses import dataclass
from typing import Any

import duckdb
from debug_toolbar.panels import Panel
from debug_toolbar.types import ServerTiming, Stats


@dataclass
class _Query:
    db: str         # "system" | "analytics"
    sql: str
    params: Any
    ms: float
    rows: int | None
    error: str | None = None


_request_store: contextvars.ContextVar[list[_Query] | None] = contextvars.ContextVar(
    "duckdb_panel_store", default=None
)


def get_request_store() -> list[_Query] | None:
    return _request_store.get()


def record_query(db: str, sql: str, params: Any, started: float,
                 rows: int | None, error: str | None = None) -> None:
    store = _request_store.get()
    if store is None:                  # outside debug request: no-op
        return
    store.append(_Query(
        db=db, sql=sql, params=params,
        ms=(time.perf_counter() - started) * 1000.0,
        rows=rows, error=error,
    ))


class InstrumentedConnection:
    """duckdb.DuckDBPyConnection-compatible wrapper that records queries."""

    def __init__(self, real: duckdb.DuckDBPyConnection, db_tag: str) -> None:
        self._real = real
        self._db = db_tag

    def execute(self, sql: str, params: Any = None, *args, **kwargs):
        started = time.perf_counter()
        err: str | None = None
        result = None
        try:
            result = (self._real.execute(sql, params, *args, **kwargs)
                      if params is not None
                      else self._real.execute(sql, *args, **kwargs))
            return result
        except Exception as e:
            err = repr(e)
            raise
        finally:
            rows: int | None = None
            try:
                if result is not None and hasattr(result, "rowcount"):
                    rows = result.rowcount
            except Exception:
                pass
            record_query(self._db, sql, params, started, rows, err)

    def __getattr__(self, name):       # delegate everything else
        return getattr(self._real, name)


class DuckDBPanel(Panel):
    title = "DuckDB"
    template = "panels/duckdb.html"

    @property
    def nav_subtitle(self) -> str:
        store = get_request_store() or []
        return f"{len(store)} queries · {sum(q.ms for q in store):.1f} ms"

    async def process_request(self, request):
        _request_store.set([])
        return await super().process_request(request)

    async def generate_stats(self, request, response) -> Stats | None:
        store = get_request_store() or []
        return {
            "queries": [q.__dict__ for q in store],
            "total_ms": sum(q.ms for q in store),
            "by_db": {db: sum(q.ms for q in store if q.db == db)
                      for db in {q.db for q in store}},
        }

    async def generate_server_timing(self, request, response) -> ServerTiming:
        store = get_request_store() or []
        return [("DuckDB", "DuckDB queries", sum(q.ms for q in store))]

Template (app/debug/templates/panels/duckdb.html):

<h4>DuckDB queries — {{ stats.queries|length }} ({{ "%.1f"|format(stats.total_ms) }} ms total)</h4>
{% if stats.by_db %}
<p>By DB:
  {% for db, ms in stats.by_db.items() %}<code>{{ db }}</code>: {{ "%.1f"|format(ms) }} ms{% if not loop.last %} · {% endif %}{% endfor %}
</p>
{% endif %}
<table class="djdt-table">
  <thead>
    <tr><th>#</th><th>DB</th><th>ms</th><th>rows</th><th>SQL</th><th>params</th></tr>
  </thead>
  <tbody>
  {% for q in stats.queries %}
    <tr class="{{ 'djdt-error' if q.error else '' }}">
      <td>{{ loop.index }}</td>
      <td>{{ q.db }}</td>
      <td>{{ "%.2f"|format(q.ms) }}</td>
      <td>{{ q.rows if q.rows is not none else '—' }}</td>
      <td>
        <pre>{{ q.sql }}</pre>
        {% if q.error %}<div class="djdt-error">{{ q.error }}</div>{% endif %}
      </td>
      <td><code>{{ q.params }}</code></td>
    </tr>
  {% endfor %}
  </tbody>
</table>

src/db.py integration

import os
from typing import TYPE_CHECKING

_DEBUG = os.environ.get("DEBUG", "").lower() in ("1", "true", "yes")


def _maybe_instrument(con, db_tag: str):
    if not _DEBUG:
        return con
    from app.debug.duckdb_panel import InstrumentedConnection
    return InstrumentedConnection(con, db_tag)


# Each existing connect helper wraps its return:
#   def get_system_conn():    return _maybe_instrument(duckdb.connect(...), "system")
#   def get_analytics_conn(): return _maybe_instrument(duckdb.connect(...), "analytics")

Implementation note: the actual function names in src/db.py are not enumerated in this design — the planner reads the file during plan execution and wraps every public connection-returning helper. Wrapping happens at the single point where a fresh duckdb.connect() is exposed.

Toolbar wiring in app/main.py

from app.logging_config import setup_logging
setup_logging("app")

import os
from fastapi import FastAPI
from app.middleware.request_id import RequestIdMiddleware

DEBUG = os.environ.get("DEBUG", "").lower() in ("1", "true", "yes")

app = FastAPI(debug=DEBUG, ...)
app.add_middleware(RequestIdMiddleware)

if DEBUG:
    try:
        from debug_toolbar.middleware import DebugToolbarMiddleware
        app.add_middleware(
            DebugToolbarMiddleware,
            panels=[
                "debug_toolbar.panels.headers.HeadersPanel",
                "debug_toolbar.panels.routes.RoutesPanel",
                "debug_toolbar.panels.settings.SettingsPanel",
                "debug_toolbar.panels.versions.VersionsPanel",
                "debug_toolbar.panels.timer.TimerPanel",
                "debug_toolbar.panels.logging.LoggingPanel",
                "debug_toolbar.panels.profiling.ProfilingPanel",
                "app.debug.duckdb_panel.DuckDBPanel",
            ],
            settings={"REFRESH_INTERVAL": 5000},
        )
    except ImportError:
        import logging
        logging.getLogger(__name__).warning(
            "DEBUG=1 but fastapi-debug-toolbar not installed; toolbar disabled"
        )

SQLAlchemyPanel and TortoisePanel are intentionally not included — project uses DuckDB directly.

---

File layout

app/
├── logging_config.py          (new)
├── middleware/
│   └── request_id.py          (new)
├── debug/                     (new)
│   ├── __init__.py
│   ├── duckdb_panel.py
│   └── templates/
│       └── panels/
│           └── duckdb.html
├── main.py                    (mod)  setup_logging + RequestIdMiddleware + debug-gated toolbar
└── api/sync.py                (mod)  remove rogue module-level basicConfig (line 104)

src/
├── db.py                      (mod)  _maybe_instrument helper
├── catalog_export.py          (mod)  remove basicConfig
└── profiler.py                (mod)  remove basicConfig

services/
├── scheduler/__main__.py      (mod)  setup_logging(__name__)
├── ws_gateway/gateway.py      (mod)  setup_logging(__name__)
├── telegram_bot/bot.py        (mod)  setup_logging(__name__)
├── corporate_memory/collector.py    (mod)  setup_logging(__name__)
└── session_collector/collector.py   (mod)  setup_logging(__name__)

connectors/
├── keboola/extractor.py       (mod)  remove basicConfig
└── jira/                      (mod)  remove basicConfig in 9 files

scripts/                       (mod)  setup_logging(__name__) in migrate_*.py and generate_sample_data.py

pyproject.toml                 (mod)  fastapi-debug-toolbar in [project.optional-dependencies].dev
.env.template                  (mod)  document DEBUG=1
docs/development.md            (new)  brief usage doc
CHANGELOG.md                   (mod)  Added: dev debug toolbar; centralized logging

tests/
├── test_logging_config.py     (new)
├── test_request_id.py         (new)
├── test_duckdb_panel.py       (new)
└── test_toolbar_integration.py (new)

---

Testing

Test	Type	Asserts
test_logging_config::test_dev_uses_rich_handler	unit	DEBUG=1 → root handler is RichHandler
::test_prod_uses_json	unit	no DEBUG → _JSONFormatter, output parses as JSON
::test_idempotent	unit	calling twice does not duplicate handlers
::test_slug_from_dotted_name	unit	_derive_slug("services.scheduler.__main__") == "scheduler"
::test_slug_from_collector_module	unit	_derive_slug("services.corporate_memory.collector") == "corporate_memory.collector"
::test_slug_explicit_override	unit	_derive_slug("app") == "app"
::test_slug_fallback_via_file	unit	service is None, frame has __file__ → slug derived from path
::test_request_id_in_json_output	unit	with contextvar set → JSON line includes "request_id"
test_request_id::test_assigns_id_when_missing	unit	header missing → 12-char id assigned
::test_passes_through_provided_id	unit	X-Request-ID preserved
::test_propagates_to_response_header	unit	response has X-Request-ID
test_duckdb_panel::test_instrumented_records_query	unit	con.execute("SELECT 1") adds entry with timing
::test_records_error	unit	failing query records error field
::test_db_tag_preserved	unit	system vs analytics tagged correctly
::test_no_op_outside_request	unit	record_query when store is None doesn't raise
::test_passthrough_attributes	unit	con.fetchall() etc. delegated to wrapped connection
test_toolbar_integration::test_html_response_has_toolbar_tab	integration	DEBUG=1 HTML response contains toolbar div
::test_no_toolbar_when_debug_off	integration	no DEBUG → no toolbar markup
::test_duckdb_panel_renders_queries	integration	DuckDB-touching endpoint → panel JSON has 1+ queries
::test_logging_panel_captures_records	integration	logger.info("X") during request → toolbar logging panel contains "X"

Coverage target ≥ 80 % for app.logging_config, app.debug.*, app.middleware.request_id.

Mocking: use real in-memory DuckDB connections (duckdb.connect(":memory:")) — fast, catches API drift. monkeypatch.setenv("DEBUG", "1") per test.

---

Rollout phases

Phase	Deliverable	Verifies
P1	app/logging_config.py + tests	pytest tests/test_logging_config.py green
P2	RequestIdMiddleware + tests, wired in app/main.py	response has X-Request-ID; JSON logs include same id
P3	Migrate app/main.py + 5 service entrypoints	each service starts; no duplicate log lines; dev colors render
P4	Migrate connectors/, src/, scripts/ (~14 files)	full suite passes; only test files keep basicConfig
P5	Add fastapi-debug-toolbar dev dep + wire middleware behind DEBUG=1 (no DuckDBPanel yet)	with DEBUG=1, visit /admin/access → toolbar tab visible. Verify under both uvicorn --reload AND docker compose up (compose run intentionally exercises non-TTY stdout — force_terminal=True must still emit color).
P6	DuckDBPanel + InstrumentedConnection + integrate in src/db.py	DuckDB panel populated on any DB-touching request. Re-verify under docker compose up after the wiring lands.
P7	docs/development.md + CHANGELOG + .env.template DEBUG=1 line	onboarding doc covers usage

Each phase = one commit. P5 is opt-in (DEBUG flag) so prod is unaffected throughout.

Compose verification is explicitly named in P5/P6 so multi-process startup ordering, env-var propagation through docker-compose.yml, and force_terminal=True color rendering on non-TTY streams are caught before merge — running only under bare uvicorn would mask docker-only failure modes.

---

Production safeguards

• Toolbar code path never imported in prod: import is inside if DEBUG: block.
• pyproject.toml puts fastapi-debug-toolbar in [project.optional-dependencies].dev only — production image build (uv pip install . without [dev]) does not ship it.
• _maybe_instrument(con, ...) is a no-op in prod: returns the raw duckdb.Connection unchanged. Zero per-query overhead.
• JSON log format unchanged from existing prod expectations. New optional fields: service, request_id. Both omitted when not set.
• DEBUG env var documented in .env.template; default unset = production behavior.

Dev-time exposure (accepted)

DuckDBPanel renders raw SQL and parameter bindings in the toolbar. In dev that means anything stored in system.duckdb or analytics.duckdb — including user emails, PATs, session-collector context — is visible to whoever has access to the running dev process. Documented as accepted dev-time exposure: the toolbar is gated by DEBUG=1, never reachable in prod, and the dev is already authorized to see their local DB. No redaction or masking. If a future deployment ever runs DEBUG=1 against shared data, the operator's exposure model is the same as LOCAL_DEV_MODE (auth bypass) and is the operator's responsibility.

The toolbar's LoggingPanel and ProfilingPanel carry similar caveats (any PII emitted via logger.info(...) ends up in the panel). Same gate, same accepted risk.

---

Failure-mode self-check

Mode	Mitigation
Hallucinated actions	Plan execution reads src/db.py to find actual connect helper names before wrapping — flagged in §"src/db.py integration"
Scope creep	Only entrypoint files + 1 broken module-level basicConfig in app/api/sync.py:104. No drive-by refactors.
Cascading errors	InstrumentedConnection.execute re-raises; record_query runs in finally, never swallows.
Context loss	setup_logging idempotent + _CONFIGURED sentinel; uvicorn loggers re-routed via propagate=True to root
Tool misuse	Native Edit/Write for code, pytest for tests, browser/playwright-cli for UI verification in P5/P6

---

Open questions

None at design time. Implementation will resolve:

• Exact function names in src/db.py to wrap (read during P6).
• Whether duckdb executemany / sql / query helpers also need instrumentation (audit during P6).
• Whether the toolbar's built-in LoggingPanel plays nicely with our RichHandler (verify in P5; fall back to StreamHandler if conflict).

---

References

• fastapi-debug-toolbar docs
• GitHub: mongkok/fastapi-debug-toolbar — v0.6.3 (2024-05), 167 stars, low velocity but stable
• PyPI: fastapi-debug-toolbar 0.6.3
• Rich logging docs
• Gemini 2.5 Flash adversarial review run 2026-04-29 (validated setup_logging design; recommended explicit __name__ over frame-walk auto-derive)