Petr
b99ec576ca
Add self-service data onboarding system
...
Table Registry as central source of truth (JSON) with atomic writes,
optimistic locking, audit logging, and data_description.md generation.
Existing readers (config.py, profiler.py) need zero changes.
Phase 1 - Discovery API:
- discover_tables() on DataSource ABC + Keboola implementation
- admin_required decorator with server-side recomputation
- GET /api/admin/discover-tables endpoint
Phase 2 - Table Registry:
- src/table_registry.py with CRUD, validation, migration from MD
- Admin API: register/update/unregister with version locking
- DELETE cascade cleans up per-user subscriptions
Phase 3 - Auto-Profiling:
- profile_changed_tables() for incremental profiling
- Non-fatal hook in sync_all() after successful sync
Phase 4 - Per-Table Subscriptions:
- table_mode (all/explicit) with per-table toggles
- GET/POST /api/table-subscriptions endpoints
- Subscription status in catalog and dashboard views
Phase 5 - Smart Sync:
- Python-generated rsync filter files (not shell YAML parsing)
- sync_data.sh uses --filter="merge ..." for explicit mode
Phase 6 - Admin UI:
- /admin/tables with discovery, registration modal, registry mgmt
- Vanilla JS, matching existing design system
2026-03-09 14:25:37 +01:00
Petr
38b86127ed
Branding cleanup: remove Keboola-specific references from docs and config
...
- server/deploy.sh: KEBOOLA_ENV_FILE -> SYNC_ENV_FILE
- server/ws-gateway.service, notify-bot.service: remove Keboola from descriptions
- .gitignore: generic comment for data directory
- CLAUDE.md, README.md, ARCHITECTURE.md: update paths from src/adapters to connectors/
- docs/DATA_SOURCES.md: update custom connector guide to connectors/ pattern
- connectors/jira/README.md: keboola-analyst -> data-analyst in config paths
- dev_docs/desktop-app.md: KeboolaAnalyst -> DataAnalyst branding
2026-03-09 12:22:27 +01:00
Petr
266e8573d3
Extract Keboola into connectors/keboola module
...
Move all Keboola-specific code out of src/ into connectors/keboola/:
- git mv src/keboola_client.py -> connectors/keboola/client.py
- Extract LocalKeboolaSource (855 lines) from data_sync.py -> connectors/keboola/adapter.py
- Rename to KeboolaDataSource with full env var validation
- Extend DataSource ABC with get_column_metadata() and get_source_name()
- Add dynamic connector registry via importlib in create_data_source()
- Refactor _generate_schema_yaml to use ABC methods (source_type, _schema_version: 2)
- Remove src/adapters/ (redundant facade layer)
- Remove Keboola validation from src/config.py (connector validates itself)
- Add 14 tests for factory, ABC defaults, env validation, dynamic lookup
2026-03-09 12:22:16 +01:00
Petr
86edd27655
Extract Jira into connectors/jira module
...
Move all Jira-specific code into a self-contained connector module:
- 22 files moved via git mv (transform, service, webhook, scripts,
systemd units, tests, docs, bin helper)
- All imports updated to use connectors.jira.* paths
- Jira is now conditional: auto-detected via JIRA_DOMAIN env var
- Webapp registers Jira blueprint only when available
- Health service monitors Jira timers only when enabled
- Profiler loads Jira tables dynamically from filesystem
- Sync settings uses config-driven dependency validation
- Renamed keboola_platform_url -> custom_url in transform
- Updated deploy.sh, sudoers-deploy, backfill_gap.sh paths
- Fixed pytest.ini to skip live tests by default
2026-03-09 11:17:50 +01:00
