AI-Cognitive-Leap/agnes-the-ai-analyst
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Rewrite auto-install guide with dual-repo architecture

Browse source 
Document the full end-to-end workflow: OSS repo (code) + private
instance repo (config/secrets). Covers SSH key isolation per repo,
symlink bridging, and ongoing deployment workflow.
This commit is contained in:
Petr 2026-03-10 11:38:41 +01:00
parent 1ac868d787
commit 495940d6b8
1 changed files with 254 additions and 168 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

422
docs/auto-install.md
View file

@ -1,50 +1,74 @@
# Automated Installation Log # Automated Installation Guide
Step-by-step record of deploying the platform on a clean Ubuntu 24.04 VM (DigitalOcean). Step-by-step deployment of AI Data Analyst on a clean Ubuntu 24.04 VM.
## Infrastructure Two repos are involved:
- **OSS repo** (public/private): application code (`padak/tmp_oss`)
- **Instance repo** (private): your config, secrets template, data schema (`padak/tmp_oss_cfg`)
- **Provider**: DigitalOcean ## Architecture on Server
- **Droplet**: s-1vcpu-2gb (1 vCPU, 2GB RAM, 50GB disk)
- **Region**: ams3 (Amsterdam)
- **OS**: Ubuntu 24.04.3 LTS
- **IP**: 165.22.199.226
## Prerequisites Discovered ```
/opt/data-analyst/
1. **DigitalOcean API token needs `ssh_key` scope** to register SSH keys ├── repo/ # OSS repo clone
- Without `ssh_keys` field in droplet creation, DO forces password expiry │ ├── config/
- Cloud-init `user_data` cannot override this (DO scripts run after cloud-init) │ │ └── instance.yaml -> ../../instance/config/instance.yaml (symlink)
- Solution: register key via `/v2/account/keys`, then reference in `ssh_keys` array │ ├── webapp/
│ ├── server/
2. **`python3-venv` must be installed** before `server/setup.sh` │ └── ...
- Ubuntu 24.04 doesn't include it by default ├── instance/ # Private instance repo clone
- Fix: `apt install python3.12-venv` before running setup │ ├── config/
│ │ ├── instance.yaml # Branding, auth domains, data source
## Step 0: Create GitHub Repo & Push │ │ └── data_description.md # Data schema (when configured)
│ ├── docs/setup/ # Custom CLAUDE.md template, etc.
```bash │ ├── .env.example # Secrets template
# Repo was created on GitHub: padak/tmp_oss (private) │ └── README.md
git remote add origin https://github.com/padak/tmp_oss.git ├── .env # Secrets (not in git, from .env.example)
git push -u origin main ├── .venv/ # Python virtual environment
└── logs/ # Application logs
``` ```
## Step 1: VM Setup Key principle: OSS repo has no secrets/config. Instance repo has no code. Symlinks bridge them.
### 1a: Create Droplet via API ## Prerequisites
1. **DigitalOcean API token** with `ssh_key` scope (or any Ubuntu 24.04 VM)
2. **Two GitHub repos**: one for OSS code, one for private instance config
3. **SSH key** on your local machine for server access
### Known Issues
- `python3-venv` must be installed before `server/setup.sh` (Ubuntu 24.04 omits it)
- `webapp-setup.sh` generates SSL nginx config - use HTTP-only for IP-only deployments
- DigitalOcean cloud-init cannot override password expiry; must use `ssh_keys` API field
## Step 0: Create Repos
```bash ```bash
# First: register SSH key (requires ssh_key scope) # Push OSS code to GitHub
git remote add origin git@github.com:YOUR_ORG/YOUR_OSS_REPO.git
git push -u origin main
# Create private instance config repo on GitHub (empty, private)
# We'll populate it from the server after setup
```
## Step 1: Provision VM
### 1a: Create Droplet (DigitalOcean)
```bash
# Register SSH key (requires ssh_key scope on API token)
curl -s -X POST -H 'Content-Type: application/json' \ curl -s -X POST -H 'Content-Type: application/json' \
-H "Authorization: Bearer $DO_TOKEN" \ -H "Authorization: Bearer $DO_TOKEN" \
-d '{"name":"my-key","public_key":"ssh-ed25519 AAAA..."}' \ -d '{"name":"my-key","public_key":"ssh-ed25519 AAAA..."}' \
"https://api.digitalocean.com/v2/account/keys" "https://api.digitalocean.com/v2/account/keys"
# Get key ID from response, then create droplet # Create droplet with SSH key
curl -s -X POST -H 'Content-Type: application/json' \ curl -s -X POST -H 'Content-Type: application/json' \
-H "Authorization: Bearer $DO_TOKEN" \ -H "Authorization: Bearer $DO_TOKEN" \
-d '{ -d '{
"name":"oss-devel-1", "name":"data-analyst-1",
"size":"s-1vcpu-2gb", "size":"s-1vcpu-2gb",
"region":"ams3", "region":"ams3",
"image":"ubuntu-24-04-x64", "image":"ubuntu-24-04-x64",
@ -58,58 +82,79 @@ curl -s -X POST -H 'Content-Type: application/json' \
```bash ```bash
ssh root@DROPLET_IP ssh root@DROPLET_IP
# Wait for apt lock to release (auto-updates run on first boot) # Wait for apt lock (auto-updates run on first boot)
# Then install python3-venv apt update && apt install -y python3.12-venv python3-pip
apt install -y python3.12-venv python3-pip
``` ```
### 1c: Clone Repo & Run Setup ### 1c: Generate Deploy Keys
Two separate keys - one per repo, for security isolation:
```bash ```bash
# Generate deploy key on VM # Key for OSS repo
ssh-keygen -t ed25519 -f /root/.ssh/deploy_key -N "" ssh-keygen -t ed25519 -f /root/.ssh/deploy_key -N "" -C "oss-app@$(hostname)"
# Add deploy_key.pub to GitHub repo as deploy key
# Configure SSH for GitHub # Key for private instance config repo
cat > /root/.ssh/config << 'EOF' ssh-keygen -t ed25519 -f /root/.ssh/instance_key -N "" -C "instance-config@$(hostname)"
Host github.com
IdentityFile /root/.ssh/deploy_key
StrictHostKeyChecking no
EOF
# Clone and setup
git clone git@github.com:YOUR_ORG/YOUR_REPO.git /opt/data-analyst/repo
cd /opt/data-analyst/repo
REPO_URL="git@github.com:YOUR_ORG/YOUR_REPO.git" bash server/setup.sh
``` ```
### Step 1 Checklist Results Add each public key as a **deploy key** on its respective GitHub repo:
- `deploy_key.pub` -> OSS repo Settings > Deploy Keys
- `instance_key.pub` -> Instance repo Settings > Deploy Keys
| # | Check | Result | Configure SSH to use the right key per repo:
|---|-------|--------|
| 1.1 | Groups created | data-ops, dataread, data-private - OK | ```bash
| 1.2 | Deploy user exists | uid=999(deploy), groups: deploy, data-ops - OK | cat > /root/.ssh/config << 'EOF'
| 1.3 | Directory structure | /opt/data-analyst/{repo,.venv,logs} - OK | # OSS application repo
| 1.4 | Python venv works | Flask 3.1.3 loaded - OK | Host github-oss
| 1.5 | Management scripts | add-analyst, list-analysts, add-admin, remove-analyst - OK | HostName github.com
IdentityFile /root/.ssh/deploy_key
StrictHostKeyChecking no
# Instance config repo (private)
Host github-cfg
HostName github.com
IdentityFile /root/.ssh/instance_key
StrictHostKeyChecking no
EOF
chmod 600 /root/.ssh/config
```
### 1d: Clone OSS Repo & Run Setup
```bash
git clone git@github-oss:YOUR_ORG/YOUR_OSS_REPO.git /opt/data-analyst/repo
cd /opt/data-analyst/repo
REPO_URL="git@github-oss:YOUR_ORG/YOUR_OSS_REPO.git" bash server/setup.sh
```
### Step 1 Checklist
| # | Check | Expected |
|---|-------|----------|
| 1.1 | Groups | data-ops, dataread, data-private exist |
| 1.2 | Deploy user | uid deploy, groups: deploy, data-ops |
| 1.3 | Directories | /opt/data-analyst/{repo,.venv,logs} |
| 1.4 | Python venv | Flask loads in .venv |
| 1.5 | Scripts | add-analyst, list-analysts in /usr/local/bin |
## Step 2: Webapp Setup ## Step 2: Webapp Setup
### 2a: Run webapp-setup.sh ### 2a: Run webapp-setup.sh
```bash ```bash
export SERVER_HOSTNAME="165.22.199.226" # or your domain export SERVER_HOSTNAME="your-domain-or-ip"
bash server/webapp-setup.sh bash server/webapp-setup.sh
``` ```
**Issue**: Nginx config assumes SSL/domain. For IP-only testing, replace with HTTP-only config: For IP-only (no SSL), replace nginx config:
```bash ```bash
cat > /etc/nginx/sites-available/webapp << 'NGINX' cat > /etc/nginx/sites-available/webapp << 'NGINX'
server { server {
listen 80; listen 80;
server_name _; server_name _;
location / { location / {
proxy_pass http://unix:/run/webapp/webapp.sock; proxy_pass http://unix:/run/webapp/webapp.sock;
proxy_set_header Host $host; proxy_set_header Host $host;
@ -120,12 +165,10 @@ server {
proxy_set_header Upgrade $http_upgrade; proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade"; proxy_set_header Connection "upgrade";
} }
location /static/ { location /static/ {
alias /opt/data-analyst/repo/webapp/static/; alias /opt/data-analyst/repo/webapp/static/;
expires 1d; expires 1d;
} }
location /health { location /health {
proxy_pass http://unix:/run/webapp/webapp.sock; proxy_pass http://unix:/run/webapp/webapp.sock;
proxy_set_header Host $host; proxy_set_header Host $host;
@ -133,12 +176,11 @@ server {
} }
} }
NGINX NGINX
rm -f /etc/nginx/sites-enabled/default rm -f /etc/nginx/sites-enabled/default
nginx -t && systemctl restart nginx nginx -t && systemctl restart nginx
``` ```
### 2b: Configure .env ### 2b: Create .env
```bash ```bash
SECRET_KEY=$(python3 -c 'import secrets; print(secrets.token_hex(32))') SECRET_KEY=$(python3 -c 'import secrets; print(secrets.token_hex(32))')
@ -147,8 +189,8 @@ cat > /opt/data-analyst/.env << EOF
WEBAPP_SECRET_KEY="${SECRET_KEY}" WEBAPP_SECRET_KEY="${SECRET_KEY}"
SERVER_HOST="YOUR_IP" SERVER_HOST="YOUR_IP"
SERVER_HOSTNAME="YOUR_IP_OR_DOMAIN" SERVER_HOSTNAME="YOUR_IP_OR_DOMAIN"
GOOGLE_CLIENT_ID="your-google-client-id" GOOGLE_CLIENT_ID="placeholder"
GOOGLE_CLIENT_SECRET="your-google-client-secret" GOOGLE_CLIENT_SECRET="placeholder"
DATA_SOURCE="local" DATA_SOURCE="local"
DATA_DIR="/data/src_data" DATA_DIR="/data/src_data"
EOF EOF
@ -172,49 +214,46 @@ systemctl start webapp
systemctl enable webapp systemctl enable webapp
``` ```
### Step 2 Checklist Results ### Step 2 Checklist
| # | Check | Result | | # | Check | Expected |
|---|-------|--------| |---|-------|----------|
| 2.1 | Nginx running | active - OK | | 2.1 | Nginx | active, port 80 |
| 2.2 | Webapp running | active (gunicorn with 2 workers) - OK | | 2.2 | Webapp | active (gunicorn) |
| 2.3 | SSL cert | SKIPPED (IP-only, no domain) | | 2.3 | Health | `curl http://IP/health` returns JSON |
| 2.4 | Health endpoint | Returns JSON with disk/load/services - OK | | 2.4 | Login page | HTTP 200 at /login |
| 2.5 | Login page loads | HTTP 200 - OK |
## Issues Found & Fixes ## Step 3: Instance Configuration (Private Repo)
### Issue 1: `python3-venv` not installed ### 3a: Clone Instance Repo
- **Symptom**: `server/setup.sh` fails at venv creation
- **Fix**: `apt install python3.12-venv` before running setup
- **TODO**: Add to `server/setup.sh` package list
### Issue 2: Nginx SSL config with IP address
- **Symptom**: Nginx fails to start - no SSL cert for "YOUR_DOMAIN"
- **Fix**: Replace nginx config with HTTP-only version for IP-only deployments
- **TODO**: `webapp-setup.sh` should detect IP vs domain and generate appropriate config
### Issue 3: DigitalOcean cloud-init limitations
- **Symptom**: `user_data` cloud-init cannot prevent password expiry
- **Fix**: Must use `ssh_keys` API field with registered key
- **Lesson**: DO initialization scripts run after cloud-init and override password settings
## Step 3: Instance Configuration
```bash ```bash
cat > /opt/data-analyst/repo/config/instance.yaml << 'YAML' git clone git@github-cfg:YOUR_ORG/YOUR_INSTANCE_REPO.git /opt/data-analyst/instance
chown -R root:data-ops /opt/data-analyst/instance
chmod -R 770 /opt/data-analyst/instance
```
### 3b: Initialize Instance Config (if empty repo)
If this is a fresh instance repo, create the initial config:
```bash
cd /opt/data-analyst/instance
mkdir -p config docs/setup
cat > config/instance.yaml << 'YAML'
instance: instance:
name: "OSS Data Analyst" name: "My Data Analyst"
subtitle: "Test Deployment" subtitle: "My Organization"
copyright: "Test" copyright: "My Org"
server: server:
hostname: "165.22.199.226" hostname: "YOUR_IP_OR_DOMAIN"
host: "165.22.199.226" host: "YOUR_IP"
app_dir: "/opt/data-analyst" app_dir: "/opt/data-analyst"
auth: auth:
allowed_domain: "test.com" # any domain for testing allowed_domain: "mycompany.com"
webapp_secret_key: "${WEBAPP_SECRET_KEY}" webapp_secret_key: "${WEBAPP_SECRET_KEY}"
data_source: data_source:
@ -224,103 +263,150 @@ catalog:
categories: {} categories: {}
YAML YAML
# Create .env.example as a template for future deployments
cat > .env.example << 'ENV'
WEBAPP_SECRET_KEY="generate-with: python3 -c 'import secrets; print(secrets.token_hex(32))'"
SERVER_HOST="server-ip"
SERVER_HOSTNAME="server-ip-or-domain"
GOOGLE_CLIENT_ID="placeholder"
GOOGLE_CLIENT_SECRET="placeholder"
DATA_SOURCE="local"
DATA_DIR="/data/src_data"
ENV
cat > .gitignore << 'GI'
.env
.env.local
*.swp
*~
.DS_Store
GI
git add -A && git commit -m "Initial instance config" && git push origin main
```
### 3c: Symlink Config into OSS Repo
```bash
# Remove any existing instance.yaml (from manual setup) and symlink
rm -f /opt/data-analyst/repo/config/instance.yaml
ln -s /opt/data-analyst/instance/config/instance.yaml /opt/data-analyst/repo/config/instance.yaml
# Optional: symlink data_description.md when ready
# ln -s /opt/data-analyst/instance/config/data_description.md /opt/data-analyst/repo/docs/data_description.md
systemctl restart webapp systemctl restart webapp
``` ```
### Step 3 Checklist Results ### Step 3 Checklist
| # | Check | Result | | # | Check | Expected |
|---|-------|--------| |---|-------|----------|
| 3.1 | Config loads | OK - webapp starts without errors | | 3.1 | Instance repo | /opt/data-analyst/instance/ exists |
| 3.2 | Instance name shown | "OSS Data Analyst" on login page | | 3.2 | Symlink | config/instance.yaml -> ../../instance/config/instance.yaml |
| 3.3 | Webapp loads | Instance name shown on login page |
## Step 4: Authentication (Email Magic Link) ## Step 4: Authentication
No Google OAuth needed! The email magic link provider works without any external service. Email magic link works without any external service.
### How it works 1. Login page shows "Sign in with Email"
2. User enters email with allowed domain
3. Without SMTP: magic link shown in browser (dev mode)
4. With SMTP: link sent via email
5. Click link -> logged in -> dashboard
1. Login page shows "Sign in with Email" button Optional: add Google OAuth by setting real `GOOGLE_CLIENT_ID`/`GOOGLE_CLIENT_SECRET`.
2. User enters email with allowed domain (e.g., `user@test.com`)
3. System generates a signed magic link (valid 15 minutes)
4. Without SMTP: link shown directly in browser (development mode)
5. With SMTP: link sent via email
6. Click link -> logged in, redirected to dashboard
### Test Results ### Step 4 Checklist
```bash | # | Check | Expected |
# Login page shows both providers |---|-------|----------|
curl -s http://localhost/login | grep 'Sign in with' | 4.1 | Email auth | "Sign in with Email" on login page |
# Sign in with Google | 4.2 | Magic link | Generated for valid domain email |
# Sign in with Email | 4.3 | Domain check | Rejects wrong domains |
| 4.4 | Login flow | Magic link -> dashboard with session |
# Email form accessible ## Step 5: Onboarding Flow (End-User)
curl -s -o /dev/null -w "%{http_code}" http://localhost/login/email
# 200
# Magic link generated (dev mode - shown in browser) After server is set up, analysts self-onboard via the webapp:
curl -s -X POST -d "email=admin@test.com" http://localhost/login/email/send
# Shows magic link URL
# Click magic link -> redirect to dashboard 1. Visit `http://YOUR_SERVER/login` and sign in with email
curl -s -D - -L -c cookies.txt "http://localhost/login/email/verify/TOKEN" 2. Dashboard shows "Get Started" with 4 steps:
# HTTP 302 -> /dashboard -> HTTP 200 - Create project folder (`mkdir -p data-analyst && cd data-analyst`)
- Generate SSH key (`ssh-keygen -t ed25519 -f ~/.ssh/data_analyst_server -N ''`)
- Copy public key (`cat ~/.ssh/data_analyst_server.pub`)
- Paste key into form, click "Create Account"
3. After account creation, dashboard shows "Set up your local environment"
4. User runs `claude` in their project folder, pastes setup instructions
5. Claude Code configures SSH, rsyncs data, sets up Python + DuckDB
## Step 6: Data Source (Next)
Configure a real data source in `instance/config/instance.yaml`:
```yaml
data_source:
type: "keboola"
keboola:
storage_token: "${KEBOOLA_STORAGE_TOKEN}"
stack_url: "https://connection.keboola.com"
project_id: "12345"
``` ```
### Step 4 Checklist Results Add the token to `.env` and create `config/data_description.md` with table schemas.
| # | Check | Result | ## Deployment Workflow (Ongoing)
|---|-------|--------|
| 4.1 | Email auth available | "Sign in with Email" shown on login page |
| 4.2 | Magic link generated | Token URL generated for valid domain email |
| 4.3 | Domain restriction | Rejects emails from wrong domain |
| 4.4 | Login works | Magic link redirects to /dashboard with session |
| 4.5 | Dev mode works | Link shown in browser when no SMTP configured |
## Issues Found & Fixes
### Issue 1: `python3-venv` not installed
- **Symptom**: `server/setup.sh` fails at venv creation
- **Fix**: `apt install python3.12-venv` before running setup
- **TODO**: Add to `server/setup.sh` package list
### Issue 2: Nginx SSL config with IP address
- **Symptom**: Nginx fails to start - no SSL cert for "YOUR_DOMAIN"
- **Fix**: Replace nginx config with HTTP-only version for IP-only deployments
- **TODO**: `webapp-setup.sh` should detect IP vs domain and generate appropriate config
### Issue 3: DigitalOcean cloud-init limitations
- **Symptom**: `user_data` cloud-init cannot prevent password expiry
- **Fix**: Must use `ssh_keys` API field with registered key
- **Lesson**: DO initialization scripts run after cloud-init and override password settings
## Current State
- **Step 0**: GitHub repo created and pushed - DONE
- **Step 1**: VM setup (groups, users, venv, scripts) - DONE
- **Step 2**: Webapp (nginx, gunicorn, .env) - DONE
- **Step 3**: Instance configuration - DONE
- **Step 4**: Authentication (email magic link) - DONE
- **Step 5+**: Discovery API, Table Registry, Data Sync - NEXT (needs data source)
## Server Access
### Update OSS code
```bash ```bash
ssh -i ~/.ssh/id_ed25519 root@165.22.199.226 cd /opt/data-analyst/repo && git pull
bash server/deploy.sh # restarts services, syncs scripts/docs
```
### Update instance config
```bash
cd /opt/data-analyst/instance && git pull
systemctl restart webapp # picks up new instance.yaml via symlink
```
### Both at once
```bash
cd /opt/data-analyst/repo && git pull
cd /opt/data-analyst/instance && git pull
bash server/deploy.sh
```
## Server Layout Summary
```
/opt/data-analyst/
├── repo/ -> git@github-oss:ORG/OSS_REPO.git
├── instance/ -> git@github-cfg:ORG/INSTANCE_REPO.git
├── .env # Secrets (not in git)
├── .venv/ # Python
└── logs/ # App logs
/root/.ssh/
├── deploy_key # For OSS repo (github-oss alias)
├── instance_key # For instance repo (github-cfg alias)
└── config # Maps aliases to keys
Symlinks:
repo/config/instance.yaml -> instance/config/instance.yaml
repo/docs/data_description.md -> instance/config/data_description.md (optional)
``` ```
## Quick Verification ## Quick Verification
```bash ```bash
# Health check # Health check
curl http://165.22.199.226/health | python3 -m json.tool curl http://YOUR_IP/health | python3 -m json.tool
# Login page # Login page
curl -s -o /dev/null -w "%{http_code}" http://165.22.199.226/login curl -s -o /dev/null -w "%{http_code}" http://YOUR_IP/login
# Expected: 200 # Expected: 200
# Email auth form # Instance config loaded
curl -s -o /dev/null -w "%{http_code}" http://165.22.199.226/login/email curl -s http://YOUR_IP/login | grep 'YOUR_INSTANCE_NAME'
# Expected: 200
``` ```