Commit graph

261 commits

Author SHA1 Message Date
ZdenekSrotyr
dbc57d1de3 Merge pull request #1 from keboola/feature/v2-fastapi-duckdb-docker-cli
feat: multi-instance deployment (14 must-have items)
2026-04-10 18:08:03 +02:00
ZdenekSrotyr
5836bcde4c fix: /setup redirects to /login instead of /dashboard for unauthenticated users 2026-04-10 17:38:08 +02:00
ZdenekSrotyr
795f602348 fix: verify_token → test_connection, url → stack_url (Devin round 6)
- KeboolaClient has test_connection() not verify_token() — every
  /api/admin/configure call for Keboola was failing with AttributeError
- Renamed data_source.keboola.url → stack_url to match
  instance.yaml.example (line 106) and avoid user confusion

663 tests pass.
2026-04-10 15:34:17 +02:00
ZdenekSrotyr
44b99f25ca fix: address Devin review round 5 — empty secret file, CI .env
- secrets.py: validate file content is non-empty before using it;
  regenerate if file exists but is empty/corrupted
- release.yml: touch .env before docker compose in smoke test
  (env_file: .env in docker-compose.yml requires the file to exist)

663 tests pass.
2026-04-10 14:55:31 +02:00
ZdenekSrotyr
40cca627be fix: address Devin review round 4 — bash arithmetic, CalVer max, docs
- smoke-test.sh: replace ((PASS++)) with PASS=$((PASS + 1)) to avoid
  set -e abort when counter is 0 (bash returns exit 1 for ((0)))
- CalVer: use max(N) from existing tags instead of count, safe when
  tags are deleted (e.g. deprecated version cleanup)
- CLAUDE.md: update schema version from v2 to v3

663 tests pass.
2026-04-10 14:39:16 +02:00
ZdenekSrotyr
dc8a9275e6 fix: address Devin review round 3 — retry exhaustion, discover path, WAL snapshot
- CalVer retry loop now exits with error if all 5 attempts fail
  (prevents pushing Docker image with unclaimed version tag)
- discover_tables endpoint reads data_source.keboola.url (consistent
  with configure_instance and _discover_and_register_tables)
- Pre-migration snapshot flushes WAL via CHECKPOINT before copying
  and copies .wal file if it still exists after flush

663 tests pass.
2026-04-10 14:11:17 +02:00
ZdenekSrotyr
c79d85f87c fix: config path mismatch + CalVer race condition (Devin review round 2)
- _discover_and_register_tables reads from data_source.keboola.url
  (matches what /api/admin/configure writes) instead of top-level
  keboola.url which doesn't exist
- CalVer: claim git tag BEFORE Docker build with retry loop (up to 5
  attempts). Prevents race where two concurrent CI runs get same N.
  Git tag acts as a distributed lock for version uniqueness.

663 tests pass.
2026-04-10 13:30:05 +02:00
ZdenekSrotyr
49f109bf73 fix: address PR review findings — config write, CalVer, error handling
- Config writes to DATA_DIR/state/instance.yaml (writable) instead of
  CONFIG_DIR (read-only :ro in Docker)
- instance_config.py checks DATA_DIR/state/ first, then falls back to
  CONFIG_DIR for backward compat
- CalVer counter is now global across channels (*-YYYY.MM.*) per spec
- Keboola error messages sanitized — log full error, return generic msg
- chmod in secrets.py wrapped in try/except for Windows compat
- Setup wizard JS handles 401 (expired JWT) with user-facing message
- deploy.yml changed to workflow_dispatch only (no duplicate test runs)
- Smoke test uses docker-compose.prod.yml + AGNES_TAG instead of sed
- docker-compose.prod.yml uses ${AGNES_TAG:-stable} env var

663 tests pass. 8 E2E verification tests pass.
2026-04-10 13:16:40 +02:00
ZdenekSrotyr
6c53082295 feat: multi-instance deployment — all 14 must-have items from spec
CalVer CI (release.yml) with stable/dev channels, health endpoint
with version/channel/schema_version, JWT secret auto-generation with
file persistence, smoke test script + Docker-in-CI, pre-migration
snapshot, /api/admin/configure for headless setup, /api/admin/
discover-and-register, /setup wizard, OpenAPI snapshot test, custom
connector mount support, CHANGELOG, migration safety tests, startup
banner.

663 tests pass (6 new migration safety + 3 OpenAPI snapshot + 1
updated JWT test).
2026-04-10 11:57:42 +02:00
ZdenekSrotyr
cce179f114 docs: add versioned tags per channel (dev-YYYY.MM.N, stable-YYYY.MM.N) 2026-04-10 06:44:25 +02:00
ZdenekSrotyr
4ea22232ef docs: multi-instance deployment and versioning design spec 2026-04-09 21:14:21 +02:00
ZdenekSrotyr
b7a3c8dd13 fix: hide Google login button when OAuth is not configured 2026-04-09 19:44:59 +02:00
ZdenekSrotyr
582e06c859 fix: cookie secure flag based on DOMAIN env — allows HTTP for dev/staging 2026-04-09 19:37:25 +02:00
ZdenekSrotyr
5ae13b199c feat: add web login handler — form POST sets cookie and redirects to dashboard 2026-04-09 19:33:25 +02:00
ZdenekSrotyr
d49844c1fe fix: Flask url_for compatibility shim + login template routes 2026-04-09 19:28:37 +02:00
ZdenekSrotyr
ad73af47b7 fix: /login/password shows login form, not account setup form 2026-04-09 19:20:44 +02:00
ZdenekSrotyr
86042f17d8 fix: add missing /login/password and /login/email web routes 2026-04-09 19:15:05 +02:00
ZdenekSrotyr
c8e232e43e docs: update stale v1 docs to v2 Docker/FastAPI/DuckDB architecture
- CONFIGURATION.md: remove Flask/SendGrid/WEBAPP_SECRET_KEY references,
  update env vars to JWT_SECRET_KEY and SESSION_SECRET, point to
  config/.env.template and config/instance.yaml.example
- disaster-recovery.md: rewrite for Docker volumes; cover GCP disk
  snapshot backup/restore and full VM rebuild; drop systemd/nginx/SSH
- server.md: strip rsync, systemd, nginx, Linux group, and sudo
  sections; keep Docker Compose operations, log viewing, health checks,
  sync/admin CLI, and Jira webhook procedures
2026-04-09 18:44:25 +02:00
ZdenekSrotyr
7d036760f5 fix: wrap Google OAuth DB connection in try/finally to ensure it is always closed
The system DB connection opened in google_callback is now closed in a
finally block, so it is released even when an exception occurs between
open and close.
2026-04-09 18:42:56 +02:00
ZdenekSrotyr
471982d3f9 fix: route admin_edit through KnowledgeRepository.update instead of raw SQL 2026-04-09 18:42:52 +02:00
ZdenekSrotyr
7e0cb80ed2 fix: move argon2 imports to top-level and catch VerifyMismatchError specifically
PasswordHasher and VerifyMismatchError are now imported at module level in
router.py and providers/password.py. Wrong-password errors are caught as
VerifyMismatchError (401); unexpected errors fall through to a 500 with logging.
2026-04-09 18:42:51 +02:00
ZdenekSrotyr
f6d2d1487f fix: remove duplicate Path alias in upload.py, replace _Path with Path 2026-04-09 18:42:48 +02:00
ZdenekSrotyr
5fe177c309 feat: add audit logging for authentication events
Log token_created, login_failed, and bootstrap_completed events via
AuditRepository. Extracts a shared _audit() helper that swallows
errors so audit failures never block auth. Also tightens password
verification to catch VerifyMismatchError specifically and log
unexpected errors at 500 rather than silently swallowing them.
2026-04-09 18:42:38 +02:00
ZdenekSrotyr
49cb940729 fix: strip HTML from table and column descriptions in OpenMetadata enricher
Imports strip_html from transformer and applies it to both table-level
and column-level descriptions parsed from the OpenMetadata API response.
2026-04-09 18:42:37 +02:00
ZdenekSrotyr
30987eef16 fix: add union_by_name=true to read_parquet calls in profiler
Handles schema evolution across partitions when profiling tables
with multiple parquet files that may have different column sets.
2026-04-09 18:42:33 +02:00
ZdenekSrotyr
5e0e4ceb9e fix: rewrite Makefile and scripts/README.md
Makefile simplified to four targets (test, dev, docker, lint) aligned
with the current FastAPI/Docker architecture. scripts/README.md rewritten
to list only the active and migration scripts that still exist.
2026-04-09 17:16:04 +02:00
ZdenekSrotyr
22cfbfe5fb docs: update references to deleted files
- QUICKSTART.md: replace data_description.md.example copy step with
  note that tables are registered via the admin API or web UI
- NOTIFICATIONS.md: replace examples/ section with planned-feature note
- telegram_bot.md: remove examples/notifications/ rows from deployment
  table and example scripts section; note feature is planned
- dev_docs/README.md: remove plan-corporate-memory.md entry
- duckdb_manager.py: update comment from remote_query.py to query API endpoint
2026-04-09 17:15:19 +02:00
ZdenekSrotyr
f3bd378b47 chore: remove 17 dead files from v1 architecture
Removes unused scripts (collect_session, generate_user_sync_configs,
standalone_profiler, remote_query, update, setup_views, test_sync,
activate_venv, backfill_gap, sync_config_template), legacy config
(data_description.md.example), llms.txt, completed planning docs
(plan-rsync-fix, plan_parquet_types_fix, plan-corporate-memory), and
notification examples/ directory.
2026-04-09 17:14:06 +02:00
ZdenekSrotyr
988cdb4320 docs: add production deployment sections to DEPLOYMENT.md
Add GHCR pre-built images, HTTPS/Caddy, multi-instance (Terraform + manual), and instance update procedures.
2026-04-09 16:41:26 +02:00
ZdenekSrotyr
fa30298589 fix: use DATA_DIR env var instead of hardcoded /data paths
- services/telegram_bot/config.py: NOTIFICATIONS_DIR now uses DATA_DIR fallback
- src/profiler.py: DATA_DIR now uses main DATA_DIR env var instead of PROFILER_DATA_DIR
- services/telegram_bot/dispatch.py: WS_GATEWAY_SOCKET_PATH now uses WS_GATEWAY_SOCKET env var
2026-04-09 16:39:44 +02:00
ZdenekSrotyr
d814eaa503 feat: add Caddy HTTPS reverse proxy and production compose override 2026-04-09 16:39:23 +02:00
ZdenekSrotyr
622c5005aa feat(infra): add GCS remote backend and instance.yaml generation to startup script 2026-04-09 16:39:07 +02:00
ZdenekSrotyr
510e1a8178 fix: add restart policy and config mount to app, scheduler, extract services 2026-04-09 16:38:58 +02:00
ZdenekSrotyr
816f217d8e feat: add commit SHA tag to Docker image push for rollback capability 2026-04-09 16:38:38 +02:00
ZdenekSrotyr
69e029fb53 docs: expand .env.template with all ~20 env vars, organize by section 2026-04-09 16:38:26 +02:00
ZdenekSrotyr
cf59abe6dd fix: update tests to provide password after OAuth token bypass fix 2026-04-09 16:35:15 +02:00
ZdenekSrotyr
7f523788c2 fix: correct YAML path for instance name and subtitle
get_instance_name and get_instance_subtitle now look up the nested
instance.name and instance.subtitle keys to match the YAML structure.
2026-04-09 16:31:56 +02:00
ZdenekSrotyr
7bada9f32b fix: force secure cookie in production, reduce max_age to 1 day
Use TESTING env var to detect production instead of relying on
request scheme, and align cookie max_age with JWT expiry (86400s).
2026-04-09 16:31:50 +02:00
ZdenekSrotyr
c55dd02196 fix: stop leaking server file paths in upload responses
Return filename instead of full server-side path in upload_session
and upload_artifact responses.
2026-04-09 16:31:46 +02:00
ZdenekSrotyr
2043594670 fix: restrict script execution endpoints to analyst/admin roles
deploy, run, and run-deployed require analyst; undeploy requires admin.
Update test to use admin token for undeploy.
2026-04-09 16:31:42 +02:00
ZdenekSrotyr
449053bf8a fix: enforce per-table access control on catalog profile endpoints
Add can_access_table check to GET /api/catalog/profile/{table_name} and
POST /api/catalog/profile/{table_name}/refresh, returning 403 for
unauthorized tables. Update test_api_complete to cover new 403 behaviour
and fix the existing 404 test to use admin token.
2026-04-09 16:30:24 +02:00
ZdenekSrotyr
ad6b3a96e4 fix: enforce role guards on admin web pages
Add require_role(Role.ADMIN) to /admin/tables and /admin/permissions,
and require_role(Role.KM_ADMIN) to /corporate-memory/admin so that
non-admin users receive 403 instead of being served the page.

Fix admin_cookie test fixture to supply a password_hash (required since
the /auth/token endpoint blocks passwordless requests). Add analyst
fixture and TestAdminRoleGuards tests verifying analysts get 403 and
admins get 200 on the protected routes.
2026-04-09 16:30:13 +02:00
ZdenekSrotyr
8e7913b93a fix: anchor auth/ gitignore rule to repo root only 2026-04-09 16:29:53 +02:00
ZdenekSrotyr
3205a8d300 fix: block /auth/token for OAuth-only users without password_hash
Users without a password_hash (Google OAuth / magic-link accounts) could
obtain a JWT by simply posting their email to /auth/token. Add an else
clause that rejects such requests with 401, directing them to their
configured auth provider. Update and extend tests accordingly.
2026-04-09 16:29:47 +02:00
ZdenekSrotyr
55515266ea fix: block DuckDB metadata functions and relative paths in query endpoint
Add information_schema, duckdb_* introspection functions, pragma_* functions,
and relative path traversal patterns to the SQL blocklist so users cannot
enumerate schema metadata regardless of RBAC. Add six corresponding tests.
2026-04-09 16:29:11 +02:00
ZdenekSrotyr
86fe4b411d fix: upgrade urllib3 1.26→2.6.3 — resolves all 4 Dependabot security alerts
Removed kbcstorage from all dependency groups (optional + dev) so urllib3
is no longer pinned to <2.0. Legacy Keboola client is available via
manual install: pip install kbcstorage
2026-04-09 14:53:30 +02:00
ZdenekSrotyr
809448e02b fix: move kbcstorage to optional dep — unblocks urllib3 security updates
kbcstorage pins urllib3<2.0.0 which blocks Dependabot security patches.
Moved to [project.optional-dependencies] keboola-legacy since the primary
extraction path uses the DuckDB Keboola extension, not kbcstorage.
Legacy fallback uses lazy import — app works without it installed.
2026-04-09 14:46:50 +02:00
ZdenekSrotyr
22b4d830e5 chore: upgrade docker actions to Node.js 24 (login-action@v4, build-push-action@v7) 2026-04-09 14:22:11 +02:00
ZdenekSrotyr
6ebfc15010 fix: setup-uv@v7 (v8 major tag doesn't exist yet) 2026-04-09 14:19:32 +02:00
ZdenekSrotyr
1ebf50bd78 fix: upgrade setup-uv@v5 → v8 (Node.js 24 native), add uv.lock
- setup-uv@v8 runs on Node.js 24 natively — no more deprecation warnings
- Removed FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 workaround (no longer needed)
- Added uv.lock for reproducible dependency resolution
2026-04-09 14:16:55 +02:00