7c9007a8f9
Add auth providers, standalone services, and service patterns to project structure in README, ARCHITECTURE, and CLAUDE.md. Reflects the completed extraction of auth, telegram bot, ws gateway, corporate memory, and session collector.
190 lines
7.4 KiB
Markdown
190 lines
7.4 KiB
Markdown
# AI Data Analyst
|
|
|
|
Open-source data distribution platform for AI analytical systems. Syncs data from various sources, converts to Parquet, and distributes to analysts who use Claude Code for local analysis.
|
|
|
|
## First-Time Setup
|
|
|
|
When a user opens this project for the first time, guide them through interactive setup:
|
|
|
|
### Step 1: Gather Information
|
|
Ask the user for:
|
|
1. Company domain (e.g., "acme.com") - used for Google OAuth
|
|
2. Data source type: keboola / csv / bigquery (future)
|
|
3. Instance name (e.g., "Acme Data Analyst")
|
|
|
|
### Step 2: Generate Configuration
|
|
1. Copy `config/instance.yaml.example` to `config/instance.yaml`
|
|
2. Fill in values from Step 1
|
|
3. If Keboola: ask for Storage API token, stack URL, project ID
|
|
4. Create `.env` from `config/.env.template`
|
|
|
|
### Step 3: Generate Data Description
|
|
1. If Keboola adapter: use the API to fetch table metadata and generate `docs/data_description.md`
|
|
2. If CSV: ask user to describe their data files
|
|
3. The file defines tables, sync strategies, and schema
|
|
|
|
### Step 4: Server Setup (if deploying)
|
|
1. Guide VM provisioning (or use existing server)
|
|
2. Run `server/setup.sh` on the target VM
|
|
3. Run `server/webapp-setup.sh` for the web portal
|
|
4. Set up CI/CD from `.github/workflows/deploy.yml.example`
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
├── src/ # Core data sync engine (vendor-neutral)
|
|
│ ├── config.py # Configuration from data_description.md
|
|
│ ├── data_sync.py # Sync orchestration + DataSource ABC
|
|
│ ├── parquet_manager.py # Parquet file management
|
|
│ └── profiler.py # Data profiling
|
|
├── connectors/ # Data source connectors (pluggable)
|
|
│ ├── keboola/ # Keboola Storage connector
|
|
│ └── jira/ # Jira webhook connector
|
|
├── auth/ # Authentication providers (pluggable)
|
|
│ ├── google/ # Google OAuth provider
|
|
│ ├── password/ # Email/password provider
|
|
│ └── desktop/ # Desktop JWT provider (API-only)
|
|
├── services/ # Standalone services (own systemd units)
|
|
│ ├── telegram_bot/ # Telegram notification bot
|
|
│ ├── ws_gateway/ # WebSocket notification gateway
|
|
│ ├── corporate_memory/ # AI knowledge aggregation
|
|
│ └── session_collector/ # Claude Code session collector
|
|
├── webapp/ # Flask web portal (login, dashboard, API)
|
|
├── server/ # Deployment infrastructure only
|
|
├── scripts/ # Utility scripts (sync, DuckDB setup, dev)
|
|
├── config/ # Configuration templates
|
|
│ ├── instance.yaml.example
|
|
│ └── data_description.md.example
|
|
├── docs/ # Documentation
|
|
└── tests/ # Test suite
|
|
```
|
|
|
|
## Architecture
|
|
|
|
```
|
|
Data Source (Keboola / CSV / BigQuery)
|
|
│
|
|
▼
|
|
┌─────────────────────────────────┐
|
|
│ Data Broker Server │
|
|
│ ├── /data/src_data/parquet/ │ Converted data
|
|
│ ├── /data/docs/ │ Documentation
|
|
│ └── /data/scripts/ │ Helper scripts
|
|
└─────────────────────────────────┘
|
|
│ rsync (via ~/server/ symlinks)
|
|
▼
|
|
┌─────────────────────────────────┐
|
|
│ Analyst (local machine) │
|
|
│ ├── ./server/ (read-only) │ parquet, docs, scripts
|
|
│ └── ./user/ (workspace) │ duckdb, notifications
|
|
└─────────────────────────────────┘
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Instance-specific config is in `config/instance.yaml`. See `config/instance.yaml.example` for all options.
|
|
|
|
Environment variables go in `.env` (never committed to git).
|
|
|
|
Data schema is defined in `docs/data_description.md` (YAML blocks in markdown).
|
|
|
|
## Development
|
|
|
|
```bash
|
|
# Setup
|
|
python3 -m venv .venv
|
|
source .venv/bin/activate
|
|
pip install -r requirements.txt
|
|
|
|
# Run webapp locally
|
|
flask --app webapp.app run --debug
|
|
|
|
# Run tests
|
|
pytest tests/ -v
|
|
|
|
# Sync data
|
|
python -m src.data_sync
|
|
```
|
|
|
|
## Extensibility
|
|
|
|
### Data Sources
|
|
Pluggable data source connectors in `connectors/`:
|
|
- **Keboola** (`keboola`): Syncs from Keboola Storage API
|
|
- **CSV** (`csv`): Import from local CSV files (planned)
|
|
- New connector = `connectors/<name>/adapter.py` implementing `DataSource`
|
|
|
|
### Authentication
|
|
Pluggable auth providers in `auth/`:
|
|
- **Google** (`google`): OAuth via Google
|
|
- **Password** (`password`): Email/password with magic links
|
|
- **Desktop** (`desktop`): JWT for desktop app API
|
|
- New provider = `auth/<name>/provider.py` implementing `AuthProvider`
|
|
|
|
Configure data source in `config/instance.yaml` under `data_source.type`.
|
|
|
|
## Server Management
|
|
|
|
```bash
|
|
# Add analyst user
|
|
sudo add-analyst username "ssh-rsa AAAA..."
|
|
|
|
# Add privileged analyst
|
|
sudo add-analyst username "ssh-rsa AAAA..." --private
|
|
|
|
# List analysts
|
|
list-analysts
|
|
|
|
# Server monitoring
|
|
uptime && free -h && df -h /data
|
|
```
|
|
|
|
## Returning Users
|
|
|
|
When reopening the project in Claude Code:
|
|
1. Sync latest data: `bash server/scripts/sync_data.sh`
|
|
2. Verify DuckDB: `ls -lh user/duckdb/analytics.duckdb`
|
|
3. Start analyzing with Claude Code
|
|
|
|
## Key Implementation Details
|
|
|
|
### Config Loading Chain
|
|
1. `config/loader.py` loads `instance.yaml` (checks `$CONFIG_DIR`, then `./config/`)
|
|
2. `webapp/config.py` calls `_load_instance_config()` at module level
|
|
3. `_get(config, *keys, default="")` traverses nested dicts safely
|
|
4. `inject_config()` context processor exposes `Config` to all Jinja templates
|
|
5. Templates use `{{ config.INSTANCE_NAME }}`, `{{ config.INSTANCE_SUBTITLE }}`, etc.
|
|
|
|
### Connector Pattern
|
|
- ABC: `DataSource` class in `src/data_sync.py`
|
|
- Registry: `create_data_source()` in `src/data_sync.py` auto-discovers connectors in `connectors/`
|
|
- Keboola: `connectors/keboola/adapter.py` -> `KeboolaDataSource` implementing `DataSource`
|
|
- Core Keboola logic: `connectors/keboola/client.py` (Keboola Storage API wrapper)
|
|
|
|
### Auth Provider Pattern
|
|
- ABC: `AuthProvider` class in `auth/__init__.py`
|
|
- Discovery: `discover_providers()` scans `auth/*/provider.py`
|
|
- Providers: google, password, desktop (each exports `provider` instance)
|
|
- Session contract: all providers set `session["user"] = {"email", "name", "picture"}`
|
|
|
|
### Service Pattern
|
|
- Self-contained modules in `services/` with `__main__.py` for `python -m services.<name>`
|
|
- Systemd files in `services/<name>/systemd/`, auto-discovered by `deploy.sh`
|
|
- Services: telegram_bot, ws_gateway, corporate_memory, session_collector
|
|
|
|
### Server Patterns
|
|
- Atomic JSON writes: `tempfile.mkstemp()` + `os.fchmod(fd, 0o660)` + `os.replace()`
|
|
- User home writes: `sudo /usr/bin/install -o {user} -g {user}` pattern
|
|
- Staging dir: `/tmp/data_analyst_staging` (deploy.sh creates it with setgid)
|
|
- Dev docs: `dev_docs/server.md` documents all established patterns
|
|
|
|
### Files NOT to modify (stable infrastructure)
|
|
- `src/parquet_manager.py` - Parquet conversion engine
|
|
- `connectors/jira/file_lock.py` - Advisory file locking
|
|
- `connectors/jira/incremental_transform.py` - Jira monthly Parquet transform
|
|
- `services/ws_gateway/` - WebSocket notification gateway
|
|
|
|
## Git Commits & Pull Requests
|
|
|
|
- Keep commit messages clean and concise
|
|
- Do not include AI attribution in commits or PRs
|