AI-Cognitive-Leap/agnes-the-ai-analyst
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
903 commits 1 branch 0 tags 13 MiB
Commit graph

316 commits

Author SHA1 Message Date
ZdenekSrotyr
78f003f5b5 fix: reject empty table name in register-table endpoint 
Fixes #8 — empty name created orphaned record that couldn't be deleted.
 2026-03-31 12:18:58 +02:00
ZdenekSrotyr
e1e2d6d903 feat: add SEED_ADMIN_EMAIL for Docker test environments 
app/main.py: seed admin user on startup when SEED_ADMIN_EMAIL is set.
docker-compose.test.yml: expose port 8000, add seed env var.
 2026-03-31 09:48:12 +02:00
ZdenekSrotyr
617e724d21 feat: add E2E test suite — API, extractor, Docker 
tests/conftest.py: shared fixtures (e2e_env, seeded_app, create_mock_extract)
tests/test_e2e_api.py: 11 tests — full sync flow, RBAC, table lifecycle
tests/test_e2e_extract.py: 6 tests — Keboola/BQ/Jira pipelines, multi-source, corrupt handling
tests/test_e2e_docker.py: 3 tests — Docker health + full flow (opt-in via -m docker)

Fix admin update route (duplicate id kwarg, .dict() → .model_dump()).

705 tests passing.
 2026-03-31 08:18:54 +02:00
ZdenekSrotyr
caa60a507d feat: add centralized RBAC module — replace Linux group auth 
New src/rbac.py: Role enum, hierarchy, get_user_role(), has_role(),
is_admin(), is_km_admin(), has_dataset_access(), set_user_role().

webapp/auth.py: admin_required + km_admin_required now use DuckDB
roles instead of Linux groups (pwd.getpwnam + sudo/data-ops check).

app/auth/dependencies.py: imports Role from src/rbac.py (single source).

11 RBAC tests passing.
 2026-03-31 08:04:35 +02:00
ZdenekSrotyr
b502bd8bdd refactor: delete old sync pipeline — 9,500 lines removed 
Phase 5 cleanup: remove all code replaced by extract.duckdb architecture.

Deleted modules:
- src/config.py (653) — replaced by DuckDB table_registry
- src/parquet_manager.py (755) — replaced by DuckDB COPY TO
- src/data_sync.py (734) — replaced by SyncOrchestrator
- src/remote_query.py (636) — replaced by DuckDB BigQuery ATTACH
- src/table_registry.py (464) — replaced by DuckDB repository
- connectors/keboola/adapter.py (820) — replaced by extractor.py
- connectors/bigquery/adapter.py (665) — replaced by extractor.py
- connectors/bigquery/client.py (644) — replaced by DuckDB BQ extension

Updated all imports in webapp, catalog_export, enricher, router,
sync_settings_service, generate_sample_data. Kept keboola/client.py
as fallback (removed src.config dependency).

704 tests passing.
 2026-03-31 07:50:37 +02:00
ZdenekSrotyr
1bf97c725c feat: wire orchestrator into API — replace DataSyncManager 
sync.py: _run_sync() now calls extractor + SyncOrchestrator.rebuild()
data.py: parquet lookup searches /data/extracts/ first, legacy fallback
catalog.py: list tables from DuckDB table_registry instead of src.config
admin.py: discover-tables uses KeboolaClient directly, remove old TableRegistry dep
 2026-03-30 20:16:33 +02:00
ZdenekSrotyr
18e5f0b6e8 feat: implement extract.duckdb contract — orchestrator + extractors 
Phase 0: extend table_registry schema (v1→v2 migration), add
source_type/bucket/source_table/query_mode columns.

Phase 1: SyncOrchestrator ATTACHes extract.duckdb files into master
analytics.duckdb. Keboola extractor uses DuckDB extension with
legacy client fallback. BigQuery extractor is remote-only via
DuckDB BQ extension (no data download).

62 tests passing.
 2026-03-30 20:12:56 +02:00
ZdenekSrotyr
7b0a161d3d fix: handle timezone-naive timestamps in health check 2026-03-30 14:19:40 +02:00
ZdenekSrotyr
bca5e91826 feat: add bootstrap endpoint + deploy skill for AI agents 
- POST /auth/bootstrap — creates first admin, self-deactivates after
- da setup bootstrap — CLI command for agent-driven setup
- da setup verify — structured health check (JSON output for agents)
- cli/skills/deploy.md — complete deployment guide for AI agents
- 6 bootstrap tests including full agent deployment flow simulation
- 156 total tests passing
 2026-03-30 14:01:01 +02:00
ZdenekSrotyr
0b91d4ac47 feat: complete web UI + auth providers + template compatibility 
All 7 web pages rendering (200):
  /login, /dashboard, /catalog, /corporate-memory,
  /corporate-memory/admin, /activity-center, /admin/tables

All 13 API endpoints working (200):
  health, sync, data, query, users, memory, scripts,
  settings, telegram, admin, catalog

Auth providers: Google OAuth, Password (argon2), Email magic link
Cookie-based JWT auth for web UI after OAuth redirect
FlexDict for Flask→FastAPI template compatibility
150 tests passing
 2026-03-27 17:34:39 +01:00
ZdenekSrotyr
1a7939c594 feat: add auth providers (Google OAuth, Password, Email magic link) + web UI fixes 
- Google OAuth with authlib + auto user creation + cookie-based JWT
- Password auth with argon2 hash + setup token flow
- Email magic link with SMTP/SendGrid support
- Cookie-based auth for web UI (after OAuth redirect)
- Dashboard template compatibility (user_info, activity, desktop status)
- 150 tests passing
 2026-03-27 17:07:59 +01:00
ZdenekSrotyr
fb1e60d8e1 fix: fix TemplateResponse API for Starlette compatibility 
Use new TemplateResponse(request, name, context) signature.
Add Flask compat shims (get_flashed_messages, url_for, session).
 2026-03-27 16:59:04 +01:00
ZdenekSrotyr
1287e63ed9 feat: complete system — web UI, all API endpoints, governance, admin, CLI commands 
Major additions:
- Web UI: Jinja2 templates in FastAPI (login, dashboard, catalog, corporate memory, admin)
- API: catalog profiles/metrics, telegram verify/unlink/status, admin table registry CRUD
- Corporate memory governance: approve/reject/mandate/revoke/edit/batch + audit log
- Sync: real DataSyncManager trigger, sync-settings, table-subscriptions
- CLI: setup (init/test/deploy/verify), server (logs/restart/deploy/backup), explore
- Instance config integration (instance.yaml loaded at startup)
- 140 tests passing (25 new)
 2026-03-27 16:52:22 +01:00
ZdenekSrotyr
c5527ec153 fix: harden script sandbox and SQL query security 
Fixes found by E2E QA agent:
- Script sandbox: block os, sys, socket, eval, exec, open, __import__,
  getattr, pathlib and 20+ other dangerous patterns
- SQL query: block COPY, ATTACH, read_csv, semicolons, non-SELECT
- Added 24 security tests covering all attack vectors
 2026-03-27 16:11:05 +01:00
ZdenekSrotyr
e0ce91ddb9 feat: add dataset permissions, script execution, Kamal config, CI/CD 
- SyncSettingsRepository + DatasetPermissionRepository with RBAC
- Script deploy/run/undeploy API with import sandboxing
- User sync settings API with permission checks
- 4 CLI skills (connectors, security, notifications, corporate-memory)
- Kamal production + staging configs
- GitHub Actions CI + deploy workflows
- 91 total tests passing
 2026-03-27 15:40:11 +01:00
ZdenekSrotyr
a3918d3833 feat: add FastAPI server with auth, RBAC, and all API endpoints 
- JWT auth with role-based access control (viewer/analyst/admin/km_admin)
- Endpoints: health, sync manifest, data download, query, users CRUD,
  corporate memory, session/artifact upload
- 18 API tests covering auth, RBAC, all endpoints
 2026-03-27 15:19:18 +01:00