* fix(security+ops): #82 #85 #87 — auth hardening, API validation, deploy posture Security and operational hardening across three issue groups: - M23: docker-compose.override.yml → docker-compose.dev.yml (BREAKING, prod foot-gun) - C13: Container runs as non-root user 'agnes' (USER directive in Dockerfile) - M21: Docker resource limits (mem_limit, cpus) on app + scheduler - M22: Caddyfile security headers (X-Frame-Options, X-Content-Type-Options, Referrer-Policy, -Server) - M17: /api/health split into minimal (unauth) + /api/health/detailed (auth) (BREAKING) - M26: release.yml restricts build-and-push to main + workflow_dispatch; paths-ignore for docs - C2: table_id traversal validation on /api/data/{table_id}/download - M4: Upload streaming (chunk-read + temp file) instead of full-buffer; /local-md hashed filename - C5: reset_token removed from POST /api/users/{id}/reset-password response - C8: Startup WARNING when no user has password_hash (bootstrap window visible) - M9: Audit log on failed web form login (mirrors /auth/token endpoint) - M10: Atomic magic-link consume via compare-and-swap (CONSUMED: marker + DuckDB conflict catch) Also: SSRF protection on /api/admin/configure (#46), memory stats SQL aggregation (#90) Generated with [Devin](https://cli.devin.ai/docs) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(review): SSRF 169.254.x.x + IPv6 multicast; M10 marker cleanup safety Review fixes: - Add 169.254.0.0/16 (link-local, cloud metadata) to SSRF regex — was missing, allowing requests to AWS/GCP/Azure metadata endpoints - Add ff[0-9a-f]{2}: (IPv6 multicast) to SSRF regex - M10: wrap Step 3 (CONSUMED marker cleanup) in try-except with warning log — prevents unhandled exception if DB write fails after successful token consumption - Add test for 169.254.169.254 SSRF rejection Generated with [Devin](https://cli.devin.ai/docs) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(review): SSRF IPv6 bypass, CLI health endpoint, upload FD leak Address Devin Review findings on PR #104: 1. SSRF IPv6 bypass: Replace hostname regex with DNS resolution + ipaddress module checks. The old regex patterns like `fe80:` only matched up to the first colon, missing real IPv6 addresses like `fe80::1`, `fc00::1`, `ff02::1`. The new approach resolves the hostname via getaddrinfo and checks each resulting IP against ipaddress.is_private/is_loopback/is_link_local/is_reserved/is_multicast. 2. CLI commands broken: `da setup test-connection`, `da setup verify`, `da diagnose`, `da status` all called /api/health expecting the old format (status=="healthy", services dict). Now they call /api/health/detailed for service-level checks (with graceful fallback to the minimal endpoint when auth is not configured). 3. Temp file handle leak: _stream_to_temp returns an open NamedTemporaryFile; callers now close it before shutil.move() to prevent FD leaks until GC. Also adds IPv6 SSRF test cases (loopback, link-local, unique-local, multicast) with mocked DNS resolution for test environment independence. Generated with [Devin](https://cli.devin.ai/docs) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * fix(review): download regex blocks hyphenated IDs; document health split Address Devin Review round-3 findings on PR #104: 1. _SAFE_IDENTIFIER regex blocked hyphenated table IDs: The download endpoint used the strict SQL-identifier regex which does not allow dots or hyphens, but Keboola table IDs like in.c-crm.orders contain both. Switched to _SAFE_QUOTED_IDENTIFIER which allows dots and hyphens while still blocking path-traversal chars (/, .., \) and quote/control characters. Added test for hyphenated/dotted IDs. 2. Documented health endpoint split in DEPLOYMENT.md: Added Health checks & external monitoring section explaining both endpoints (minimal unauth /api/health vs authenticated /api/health/detailed) and how to wire external monitoring tools to the detailed endpoint with a PAT. Generated with [Devin](https://cli.devin.ai/docs) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> * release(0.12.1): cut hotfix for snapshot integrity + #82/#85/#87 hardening * fix(security): apply CAS pattern to password reset confirm (#82/M10 follow-up) Devin review on the rebased PR flagged the asymmetry: magic-link verify got the atomic compare-and-swap pattern in the original M10 fix, but password reset confirm at /auth/password/reset/confirm was still using read-validate-clear. Two concurrent POSTs with the same valid reset token could both succeed in setting different new passwords (last-write- wins). Lower severity than the magic-link race because the attacker would need the reset token AND to race the legitimate user, but the asymmetry was a polish gap. Mirrors app/auth/providers/email.py::_consume_token CAS exactly: write unique CONSUMED:<random> marker via UPDATE...WHERE token=old_token, then SELECT to verify our marker won, then proceed. Only the winner clears the marker and applies the password change. New regression test_concurrent_reset_only_one_wins in tests/test_password_flows.py::TestResetConfirm pins the contract: two ThreadPoolExecutor workers + Barrier hit /reset/confirm with the same token; exactly one gets 302 (password applied), the other gets 200 with 'Invalid or expired'. Sanity-checked against the pre-CAS code — both POSTs got 302 (race confirmed). --------- Co-authored-by: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com>
253 lines
9.2 KiB
Python
253 lines
9.2 KiB
Python
"""Setup commands — da setup init/bootstrap/test-connection/first-sync/verify."""
|
|
|
|
import json
|
|
import os
|
|
|
|
import typer
|
|
|
|
from cli.client import api_get, api_post
|
|
|
|
setup_app = typer.Typer(help="Instance setup (guided by AI agent)")
|
|
|
|
|
|
@setup_app.command("init")
|
|
def setup_init(
|
|
server: str = typer.Option("http://localhost:8000", help="Server URL"),
|
|
):
|
|
"""Initialize CLI config to point at a server."""
|
|
typer.echo(f"Server: {server}")
|
|
|
|
from cli.config import _config_dir
|
|
config_dir = _config_dir()
|
|
config_file = config_dir / "config.yaml"
|
|
|
|
import yaml
|
|
config = {"server": server}
|
|
config_file.write_text(yaml.dump(config))
|
|
typer.echo(f"Config saved to {config_file}")
|
|
os.environ["DA_SERVER"] = server
|
|
typer.echo("\nNext: da setup bootstrap --email admin@company.com")
|
|
|
|
|
|
@setup_app.command("bootstrap")
|
|
def bootstrap(
|
|
email: str = typer.Argument(..., help="Admin email"),
|
|
name: str = typer.Option("", help="Display name"),
|
|
password: str = typer.Option("", help="Optional password"),
|
|
server: str = typer.Option(None, help="Server URL override"),
|
|
):
|
|
"""Create the first admin user on a fresh instance.
|
|
|
|
Only works when the database has zero users.
|
|
After this, use 'da login' for normal auth.
|
|
"""
|
|
if server:
|
|
os.environ["DA_SERVER"] = server
|
|
|
|
typer.echo("Bootstrapping first admin user...")
|
|
try:
|
|
resp = api_post("/auth/bootstrap", json={
|
|
"email": email,
|
|
"name": name or email.split("@")[0],
|
|
"password": password,
|
|
})
|
|
if resp.status_code == 200:
|
|
data = resp.json()
|
|
# Save token automatically
|
|
from cli.config import save_token
|
|
save_token(data["access_token"], data["email"], data["role"])
|
|
typer.echo(f"Admin user created: {data['email']}")
|
|
typer.echo(f"Token saved — you are now logged in as admin.")
|
|
typer.echo("\nNext: da setup test-connection")
|
|
elif resp.status_code == 403:
|
|
typer.echo(f"Bootstrap disabled: {resp.json().get('detail', '')}")
|
|
typer.echo("Users already exist. Use: da login --email your@email.com")
|
|
else:
|
|
typer.echo(f"Failed: {resp.text}", err=True)
|
|
raise typer.Exit(1)
|
|
except Exception as e:
|
|
typer.echo(f"Connection error: {e}", err=True)
|
|
typer.echo("Is the server running? Check: docker compose ps")
|
|
raise typer.Exit(1)
|
|
|
|
|
|
@setup_app.command("test-connection")
|
|
def test_connection():
|
|
"""Test connection to the server and data source."""
|
|
typer.echo("Testing server connection...")
|
|
try:
|
|
# Quick unauth ping first
|
|
resp = api_get("/api/health")
|
|
health = resp.json()
|
|
if health.get("status") != "ok":
|
|
typer.echo(f" Server: unexpected status {health.get('status')}")
|
|
raise typer.Exit(1)
|
|
typer.echo(" Server: reachable")
|
|
|
|
# Detailed health (auth required) for service-level checks
|
|
try:
|
|
resp = api_get("/api/health/detailed")
|
|
detailed = resp.json()
|
|
typer.echo(f" Health: {detailed.get('status', 'unknown')}")
|
|
for svc, info in detailed.get("services", {}).items():
|
|
typer.echo(f" {svc}: {info.get('status', '?')}")
|
|
if detailed.get("status") == "healthy":
|
|
typer.echo("\nServer is healthy.")
|
|
else:
|
|
typer.echo("\nServer has issues. Check: da diagnose --json")
|
|
except Exception:
|
|
# Auth may not be configured yet — minimal check is sufficient
|
|
typer.echo("\nServer is reachable (detailed check requires auth).")
|
|
|
|
except Exception as e:
|
|
typer.echo(f" FAILED: {e}", err=True)
|
|
raise typer.Exit(1)
|
|
|
|
typer.echo("\nNext: da setup first-sync")
|
|
|
|
|
|
@setup_app.command("first-sync")
|
|
def first_sync():
|
|
"""Trigger the first data sync."""
|
|
typer.echo("Triggering initial data sync...")
|
|
try:
|
|
resp = api_post("/api/sync/trigger")
|
|
if resp.status_code == 200:
|
|
data = resp.json()
|
|
typer.echo(f" Status: {data.get('status', '?')}")
|
|
typer.echo(f" {data.get('message', '')}")
|
|
elif resp.status_code == 403:
|
|
typer.echo(" Permission denied. Are you logged in as admin?")
|
|
typer.echo(" Run: da login --email admin@company.com")
|
|
raise typer.Exit(1)
|
|
else:
|
|
typer.echo(f" Failed: {resp.text}", err=True)
|
|
raise typer.Exit(1)
|
|
except Exception as e:
|
|
typer.echo(f" Error: {e}", err=True)
|
|
raise typer.Exit(1)
|
|
|
|
typer.echo("\nWait for sync to complete, then: da setup verify")
|
|
|
|
|
|
@setup_app.command("verify")
|
|
def verify(as_json: bool = typer.Option(False, "--json", help="Output as JSON")):
|
|
"""Verify the instance is working end-to-end.
|
|
|
|
Checks: server health → auth → data sync → manifest → query capability.
|
|
Returns structured report for AI agents.
|
|
"""
|
|
checks = []
|
|
|
|
# 1. Server reachable
|
|
try:
|
|
resp = api_get("/api/health")
|
|
h = resp.json()
|
|
# Minimal health returns {"status": "ok"} — try detailed for richer check
|
|
try:
|
|
resp_d = api_get("/api/health/detailed")
|
|
hd = resp_d.json()
|
|
checks.append({
|
|
"name": "server",
|
|
"status": "pass" if hd.get("status") == "healthy" else "warn",
|
|
"detail": hd.get("status"),
|
|
})
|
|
except Exception:
|
|
# Auth not configured yet — minimal reachability is enough
|
|
checks.append({
|
|
"name": "server",
|
|
"status": "pass" if h.get("status") == "ok" else "warn",
|
|
"detail": h.get("status"),
|
|
})
|
|
except Exception as e:
|
|
checks.append({"name": "server", "status": "fail", "detail": str(e)})
|
|
_report(checks, as_json)
|
|
return
|
|
|
|
# 2. Auth works (token valid)
|
|
from cli.config import get_token
|
|
token = get_token()
|
|
if token:
|
|
try:
|
|
resp = api_get("/api/sync/manifest")
|
|
if resp.status_code == 200:
|
|
checks.append({"name": "auth", "status": "pass", "detail": "token valid"})
|
|
else:
|
|
checks.append({"name": "auth", "status": "fail", "detail": f"HTTP {resp.status_code}"})
|
|
except Exception as e:
|
|
checks.append({"name": "auth", "status": "fail", "detail": str(e)})
|
|
else:
|
|
checks.append({"name": "auth", "status": "fail", "detail": "no token — run: da login"})
|
|
|
|
# 3. Data available
|
|
try:
|
|
resp = api_get("/api/sync/manifest")
|
|
m = resp.json()
|
|
table_count = len(m.get("tables", {}))
|
|
total_rows = sum(t.get("rows", 0) for t in m.get("tables", {}).values())
|
|
if table_count > 0:
|
|
checks.append({"name": "data", "status": "pass", "detail": f"{table_count} tables, {total_rows:,} rows"})
|
|
else:
|
|
checks.append({"name": "data", "status": "warn", "detail": "0 tables — run: da setup first-sync"})
|
|
except Exception as e:
|
|
checks.append({"name": "data", "status": "fail", "detail": str(e)})
|
|
|
|
# 4. Users exist
|
|
try:
|
|
resp = api_get("/api/users")
|
|
if resp.status_code == 200:
|
|
count = len(resp.json())
|
|
checks.append({"name": "users", "status": "pass", "detail": f"{count} users"})
|
|
elif resp.status_code == 403:
|
|
checks.append({"name": "users", "status": "pass", "detail": "exists (need admin for count)"})
|
|
else:
|
|
checks.append({"name": "users", "status": "warn", "detail": f"HTTP {resp.status_code}"})
|
|
except Exception as e:
|
|
checks.append({"name": "users", "status": "fail", "detail": str(e)})
|
|
|
|
# 5. Web UI accessible
|
|
try:
|
|
resp = api_get("/login")
|
|
checks.append({
|
|
"name": "web_ui",
|
|
"status": "pass" if resp.status_code == 200 else "fail",
|
|
"detail": f"HTTP {resp.status_code}, {len(resp.content)} bytes",
|
|
})
|
|
except Exception as e:
|
|
checks.append({"name": "web_ui", "status": "fail", "detail": str(e)})
|
|
|
|
# 6. Swagger docs
|
|
try:
|
|
resp = api_get("/docs")
|
|
checks.append({
|
|
"name": "api_docs",
|
|
"status": "pass" if resp.status_code == 200 else "fail",
|
|
"detail": f"HTTP {resp.status_code}",
|
|
})
|
|
except Exception as e:
|
|
checks.append({"name": "api_docs", "status": "fail", "detail": str(e)})
|
|
|
|
_report(checks, as_json)
|
|
|
|
|
|
def _report(checks: list, as_json: bool):
|
|
all_pass = all(c["status"] == "pass" for c in checks)
|
|
has_fail = any(c["status"] == "fail" for c in checks)
|
|
|
|
if as_json:
|
|
typer.echo(json.dumps({
|
|
"overall": "pass" if all_pass else ("fail" if has_fail else "warn"),
|
|
"checks": checks,
|
|
}, indent=2))
|
|
else:
|
|
for c in checks:
|
|
icon = {"pass": "OK", "fail": "FAIL", "warn": "WARN"}[c["status"]]
|
|
typer.echo(f" [{icon:4s}] {c['name']}: {c['detail']}")
|
|
typer.echo("")
|
|
if all_pass:
|
|
typer.echo("All checks passed! Instance is ready.")
|
|
elif has_fail:
|
|
typer.echo("Some checks FAILED. See above for details.")
|
|
raise typer.Exit(1)
|
|
else:
|
|
typer.echo("Instance is running but some items need attention.")
|