AI-Cognitive-Leap/agnes-the-ai-analyst
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: add HACKATHON.md — condensed deploy + dev playbooks (#21)

Browse source 
Written for both humans and AI agents — explicit commands, expected
outputs, troubleshooting tables, 'safe to run anytime' vs 'requires
thought' sections, pitfalls checklist.

Three parts:
1. Deploy for a new customer (45 min target, 7 steps)
2. Develop against Agnes (branch → image → dev VM loop, common tasks)
3. AI agent checklist (guardrails, verification, common pitfalls)

Complements the deep docs (ONBOARDING.md, DEPLOYMENT.md, architecture.md)
with a practical quick-reference for hackathon-style deploys.
This commit is contained in:
ZdenekSrotyr 2026-04-21 21:33:06 +02:00 committed by GitHub
parent ada9fb75f6
commit 1ca5295d54
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 363 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

1
README.md
View file

@ -133,6 +133,7 @@ See `config/instance.yaml.example` for all available options.
## Documentation ## Documentation
- [Hackathon TL;DR](docs/HACKATHON.md) — condensed deploy + dev playbooks (for both humans and AI agents)
- [Onboarding Guide](docs/ONBOARDING.md) — end-to-end Terraform deployment into a GCP project (recommended for production) - [Onboarding Guide](docs/ONBOARDING.md) — end-to-end Terraform deployment into a GCP project (recommended for production)
- [Deployment Guide](docs/DEPLOYMENT.md) — chooses between Terraform and Docker Compose; covers OSS self-host - [Deployment Guide](docs/DEPLOYMENT.md) — chooses between Terraform and Docker Compose; covers OSS self-host
- [Configuration Reference](docs/CONFIGURATION.md) — `instance.yaml`, env vars, per-instance options - [Configuration Reference](docs/CONFIGURATION.md) — `instance.yaml`, env vars, per-instance options

362
docs/HACKATHON.md Normal file
View file

@ -0,0 +1,362 @@
# Hackathon TL;DR — Deploy & Develop
Two condensed playbooks, written to be copy-pasteable by both humans and AI agents. For depth see [`ONBOARDING.md`](ONBOARDING.md) and [`DEPLOYMENT.md`](DEPLOYMENT.md).
- [Part 1 — Deploy for a new customer](#part-1--deploy-for-a-new-customer)
- [Part 2 — Develop against Agnes](#part-2--develop-against-agnes)
- [Part 3 — AI agent checklist](#part-3--ai-agent-checklist)
---
## Part 1 — Deploy for a new customer
**Goal:** Agnes running in `https://<customer-name>'s own GCP project`, accessible on an IP. Target time: **45 minutes**.
### Prerequisites (verify first)
```bash
gcloud --version # ≥ 500.0.0
terraform --version # ≥ 1.5
gh --version # any recent
gh auth status # must be logged in with repo + workflow + admin:repo_hook scopes
```
Plus:
- GCP project with **billing linked**; you have `roles/owner` or equivalent
- Keboola Storage token (if `data_source = "keboola"`)
### Step 1 — Bootstrap GCP project
```bash
# Download + run bootstrap (creates deploy SA, tfstate bucket, enables APIs)
curl -fsSL https://raw.githubusercontent.com/keboola/agnes-the-ai-analyst/main/scripts/bootstrap-gcp.sh -o bootstrap-gcp.sh
chmod +x bootstrap-gcp.sh
./bootstrap-gcp.sh <GCP_PROJECT_ID>
```
Outputs you need from the run:
- SA email `agnes-deploy@<project>.iam.gserviceaccount.com`
- Bucket `gs://agnes-<project>-tfstate`
- Key file `./agnes-deploy-<project>-key.json` — move to `~/.agnes-keys/` and `chmod 600`. **Never git-commit.**
### Step 2 — Keboola token secret
Skip if `data_source` is not keboola.
```bash
echo -n "<KEBOOLA_STORAGE_TOKEN>" | gcloud secrets create keboola-storage-token \
--data-file=- --replication-policy=automatic --project=<GCP_PROJECT_ID>
```
### Step 3 — Create customer's private infra repo from template
```bash
# <customer-org> can be your own org (if Keboola is deploying) or customer's org
gh repo create <customer-org>/agnes-infra-<customer-name> \
--template keboola/agnes-infra-template \
--private \
--clone
cd agnes-infra-<customer-name>
# SA key to GitHub secret
gh secret set GCP_SA_KEY < ~/.agnes-keys/agnes-deploy-<project>-key.json
```
### Step 4 — GitHub environments + auto-merge
```bash
# dev environment — no protection
gh api -X PUT repos/<customer-org>/agnes-infra-<customer-name>/environments/dev
# prod environment — branch policy (main only), no reviewer here; add manually via UI
echo '{"deployment_branch_policy":{"protected_branches":true,"custom_branch_policies":false}}' \
| gh api -X PUT repos/<customer-org>/agnes-infra-<customer-name>/environments/prod --input -
# Settings → Environments → prod → Add required reviewer
# Allow auto-merge on this repo (for Renovate)
gh api -X PATCH repos/<customer-org>/agnes-infra-<customer-name> \
-f allow_auto_merge=true -f delete_branch_on_merge=true
# Install Renovate GitHub App on this repo:
# https://github.com/apps/renovate → Configure → <customer-org>/agnes-infra-<customer-name>
```
### Step 5 — Edit `terraform/main.tf` and `terraform.tfvars`
```hcl
# terraform/main.tf — replace both placeholders
backend "gcs" {
bucket = "agnes-<GCP_PROJECT_ID>-tfstate"
prefix = "<customer-name>"
}
```
```bash
cp terraform/terraform.tfvars.example terraform/terraform.tfvars
# Edit, minimum required:
# gcp_project_id = "<GCP_PROJECT_ID>"
# customer_name = "<customer-name>" # lowercase, 2–21 chars
# seed_admin_email = "admin@<customer>.com"
# keboola_stack_url = "https://connection.<region>.gcp.keboola.com/"
```
### Step 6 — First apply (from local, before CI is set up)
```bash
cd terraform
export GOOGLE_APPLICATION_CREDENTIALS=~/.agnes-keys/agnes-deploy-<project>-key.json
terraform init
terraform plan # verify counts: ~20 resources to add
terraform apply # type 'yes'
# Expected: ~5 min. Outputs include prod_ip.
terraform output -raw prod_ip
```
Alternative: commit tfvars + push to main — GitHub Actions `apply-dev` auto-runs, `apply-prod` waits for reviewer.
### Step 7 — Bootstrap the admin + log in
```bash
PROD_IP=$(terraform output -raw prod_ip)
curl -X POST "http://$PROD_IP:8000/auth/bootstrap" \
-H "Content-Type: application/json" \
-d '{"email":"<seed_admin_email from tfvars>","password":"<STRONG_PASSWORD>"}'
```
Expected: 200 OK + JSON with `role: "admin"`.
Open `http://<prod_ip>:8000/login` → sign in. Done.
### Verify (smoke tests)
```bash
curl -s "http://$PROD_IP:8000/api/health" | jq '{status, version, channel}'
curl -s "http://$PROD_IP:8000/api/version" | jq
```
Both should return JSON. Badge in UI footer shows `channel-version · deployed Xs ago`.
### Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| `PERMISSION_DENIED: monitoring.uptimeCheckConfigs.create` | monitoring.googleapis.com not enabled / role missing | Re-run `bootstrap-gcp.sh` (grants `roles/monitoring.editor`) |
| `At least one of the pre-conditions you specified did not hold` | Stale GCS state lock | Remove `gs://…/tfstate/<prefix>/default.tflock` + retry |
| `Error acquiring the state lock` | Active apply concurrent | Wait, or `terraform force-unlock <LOCK_ID>` |
| Docker pull fails `ghcr.io/…:<tag>: not found` | `image_tag` pins a branch/tag that was never built | Fix `image_tag` in tfvars (use `stable` or `dev` floating) |
| VM up but `/api/health` 000 / connection refused | Startup script still running (takes 60–90 s) | Tail `/var/log/agnes-startup.log` via `gcloud compute ssh` |
| `/auth/bootstrap` returns 403 `already have passwords` | Someone already bootstrapped. | Use normal login at `/login/password` |
---
## Part 2 — Develop against Agnes
**Loop:** branch in public repo → auto-built `:dev-<slug>` image → point a dev VM at that tag → iterate.
### Development architecture (mental model)
```
keboola/agnes-the-ai-analyst (public)
│
│ push branch zs/my-edit
├──▶ release.yml builds ghcr.io/…:dev-zs-my-edit (one image per push to branch)
│
│ push to main
└──▶ release.yml builds :stable + :stable-YYYY.MM.N
<customer-infra-repo>/terraform.tfvars
│
│ dev_instances = [..., { name = "agnes-zs", image_tag = "dev-zs-my-edit" }, ...]
└──▶ apply-dev recreates VM agnes-zs, pinned to that tag, cron pulls new digests every 5 min
```
### Step 1 — Branch and push (in `keboola/agnes-the-ai-analyst`)
```bash
git checkout -b zs/my-edit # or feature/xyz, fix/bar — any slash prefix works
# …edit code…
git commit -am "wip: my experiment"
git push origin zs/my-edit
# → release.yml builds ghcr.io/keboola/agnes-the-ai-analyst:dev-zs-my-edit (~5 min)
```
**Slug rule:** `<branch>` is slugified — non-`[a-z0-9-]` → `-`, lowercased, max 50 chars. Leading `feature/` is stripped. So:
- `zs/my-edit` → `:dev-zs-my-edit`
- `feature/alice/dashboard` → `:dev-alice-dashboard`
- `fix/issue_42` → `:dev-fix-issue-42`
Verify the image exists before continuing:
```bash
docker manifest inspect ghcr.io/keboola/agnes-the-ai-analyst:dev-zs-my-edit
```
### Step 2 — Open PR in the customer's infra repo
```bash
cd <customer-infra-repo>
git checkout -b add-dev-vm-zs
# Edit terraform/variables.tf or terraform.tfvars:
# dev_instances = [
# { name = "agnes-dev", image_tag = "dev" },
# { name = "agnes-zs-edit", image_tag = "dev-zs-my-edit" }, # <-- added
# ]
git commit -am "add: dev VM pinned to zs/my-edit"
git push origin add-dev-vm-zs
gh pr create
```
`plan.yml` comments on PR with diff. Review + merge → `apply-dev` creates VM (~2 min).
### Step 3 — Access your VM
```bash
cd terraform
terraform output -json instance_ips # grep your VM's name
# Open http://<ip>:8000/login
# Use the customer's admin credentials (seed + password)
```
### Step 4 — Iterate
Every push to your branch:
1. release.yml rebuilds `:dev-zs-my-edit`
2. Cron on your VM (every 5 min) detects new digest, pulls, restarts containers
3. Within ~6 min of your push, your VM runs the new code
No manual apply needed.
### Step 5 — Merge to main
Open PR on public repo → review → merge. This:
1. Builds `:stable + :stable-YYYY.MM.N` (main)
2. Smoke test in CI
3. Cron on **all prod VMs** (every customer!) pulls new `:stable` within 5 min
Your branch's `:dev-zs-my-edit` tag persists in GHCR but is no longer updated. Your dev VM still runs the last build of your branch until you change its `image_tag`.
### Step 6 — Clean up
PR in customer infra repo removing the entry:
```diff
dev_instances = [
{ name = "agnes-dev", image_tag = "dev" },
- { name = "agnes-zs-edit", image_tag = "dev-zs-my-edit" },
]
```
Merge → apply-dev destroys the VM + data disk + IP + monitoring resources. Daily snapshot (if enabled) retains data for 30 days.
### Common development tasks
| Task | Where | How |
|---|---|---|
| Write code | public repo | Normal git workflow |
| Run tests locally | public repo | `TESTING=1 pytest tests/ -v` |
| Bump infra module | public repo | Edit `infra/modules/customer-instance/`, PR, merge, create `infra-vX.Y.Z` tag |
| Point customer at new module | customer infra repo | Renovate opens PR; or edit `ref=` in main.tf manually |
| Force-propagate startup script change | customer infra repo | Actions → Terraform Apply → Run workflow → `recreate_targets=module.agnes.google_compute_instance.vm["agnes-prod"]` |
| Add dev VM for someone else | customer infra repo | Add entry to `dev_instances`, PR, merge |
| Rotate Keboola token | customer GCP + VM | `gcloud secrets versions add keboola-storage-token --data-file=-` then SSH + `sudo /usr/local/bin/agnes-auto-upgrade.sh` (no manual edits to `.env`) |
| Restart app manually | customer VM | `sudo docker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.host-mount.yml restart app` |
| See current deployed version | any | Footer badge on UI, or `curl /api/version` |
### Rules
- **Never `gcloud compute ssh` + edit `/opt/agnes/.env`** — gets wiped on next VM recreate. Route all config through Terraform or Secret Manager.
- **Never edit inside a VM's `/opt/agnes/`** — same reason. If you need a change to survive, it belongs in the module / compose files / tfvars.
- **Never bump `infra-v*` by tagging upstream without first testing on a dev VM** — a broken module propagates to all customer repos via Renovate.
- **Never delete a customer's data disk without a manual `gcloud compute disks snapshot` first** if in doubt about what's on it.
---
## Part 3 — AI agent checklist
These are guardrails/verification steps an AI agent should follow autonomously.
### Before taking destructive action
Run and read, don't assume:
```bash
terraform plan # what will actually change?
gh run list --limit 3 # any CI failures?
curl -s <PROD_IP>:8000/api/health | jq .status # is prod actually healthy?
```
### When propagating module bumps
1. Read `docs/superpowers/plans/2026-04-21-deployment-log.md` for context on iteration history.
2. Check current `ref=` in customer infra repo against latest `infra-v*` tag in upstream.
3. Prefer Renovate PR over manual edit — has automatic `terraform validate` gate.
4. For startup-script changes (not just module-resource changes), use `workflow_dispatch` → `recreate_targets` to force VM recreate. Normal apply won't propagate (`ignore_changes`).
### When a customer reports "it's broken"
```bash
# What version is deployed?
curl -s http://<ip>:8000/api/version | jq
# Recent deploys?
gh run list --repo <customer-org>/agnes-infra-<customer> --limit 5
# VM state?
gcloud compute instances list --project=<customer-project> --filter="name~agnes-"
# App logs (last 50)
gcloud compute ssh agnes-prod --zone=... --project=... \
--command="sudo docker logs agnes-app-1 --tail 50"
# Startup script log (if VM just booted)
gcloud compute ssh agnes-prod --zone=... --project=... \
--command="sudo tail -30 /var/log/agnes-startup.log"
```
### When you're unsure
Prefer non-destructive paths first:
1. `terraform plan` (read-only) before `apply`
2. Add a new resource before deleting an old one
3. Snapshot before destroying a disk
4. Dev VM before touching prod — always
### Common pitfalls to detect
| Pitfall | Check |
|---|---|
| Uncommitted local changes on operator's laptop | `git status -s` in infra repo |
| Multiple concurrent applies (state lock) | `gsutil ls gs://.../tflock` |
| `image_tag` points at non-existent GHCR image | `docker manifest inspect ghcr.io/…:<tag>` |
| Seed user without password on fresh deploy | `curl /api/health | jq .services.users.count` — if 1 and nobody has logged in, `/auth/bootstrap` is still open |
| Main branch protection prevents direct push | Use PR + auto-merge; never force-push to main |
| Renovate not installed → module bumps don't happen | Check `https://github.com/<org>/<repo>/pulls?q=author%3Aapp%2Frenovate` |
| `/opt/agnes/.env` edited manually → drift | `git diff` against module's expected `.env` shape |
### Safe-to-run anytime
- `curl /api/health`, `curl /api/version` — no auth, no side effects
- `terraform plan` — read-only
- `gh run list`, `gh pr list` — read-only
- `gcloud ... describe` / `list` — read-only
- `docker logs` / `docker inspect` — read-only (on the VM)
### Requires thought
- `terraform apply` — mutates infra
- `gh workflow run` with `recreate_targets` — destroys + recreates VMs
- `gcloud compute instances delete` — unrecoverable after 30 days
- `gcloud secrets versions destroy` — unrecoverable
- `gh repo delete` — unrecoverable
---
## Reference links
- Full onboarding: [`docs/ONBOARDING.md`](ONBOARDING.md)
- Deployment comparison: [`docs/DEPLOYMENT.md`](DEPLOYMENT.md)
- Spec (architecture rationale): [`docs/superpowers/specs/2026-04-21-multi-customer-deployment-spec.md`](superpowers/specs/2026-04-21-multi-customer-deployment-spec.md)
- Deployment log (what we actually built, with iterations and known limitations): [`docs/superpowers/plans/2026-04-21-deployment-log.md`](superpowers/plans/2026-04-21-deployment-log.md)
- Module source: [`infra/modules/customer-instance/`](../infra/modules/customer-instance/)
- Upstream issues: https://github.com/keboola/agnes-the-ai-analyst/issues