Commit graph

10 commits

Author SHA1 Message Date
ZdenekSrotyr
921094ae40 feat(infra): address code review — scoped SA, fail-fast secrets, firewall split, cron reads .env, merge fix
Critical fixes:
- C1: VM SA now gets secretmanager.secretAccessor only on specific secrets
  (JWT + each entry in runtime_secrets). Previously project-wide.
- C3: chmod 640 on /var/log/agnes-startup.log (defense in depth)
- C4: Remove '|| echo ""' fallback on keboola-storage-token — boot now fails
  loudly if the secret is missing instead of starting a broken app.
- C5: Cron auto-upgrade script sources /opt/agnes/.env for AGNES_TAG. If an
  operator edits .env to pin a specific stable-YYYY.MM.N, cron picks it up
  immediately with no drift. Removed AGNES_TAG from crontab entry.
- C7: explicit depends_on = [IAM bindings, secret_version] on VM — prevents
  race where VM boots before IAM propagates.

Important fixes:
- I1: Split firewall into web (80/443 + conditional 8000) and ssh (port 22 with
  configurable source_ranges, default IAP range only).
- I4: Fetch docker-compose files from compose_ref (default 'main'), so customers
  can pin a specific tag for reproducibility.
- I5+I6: Merge order fixed — user-supplied dev_instances values now override
  defaults (was the other way around). Dev tls_mode default flipped to 'none'.
- I7: Remove '|| true' on Caddyfile fetch; surface failures loudly.
- New acme_email variable (falls back to seed_admin_email if empty).

Out-of-module:
- Comments translated from Czech to English where applicable (M1).
2026-04-21 19:39:53 +02:00
ZdenekSrotyr
0842debf8a feat(infra): add daily backup snapshot + monitoring alerts
- google_compute_resource_policy.daily_backup: daily snapshot at 02:00,
  30-day retention, labels (app=agnes, customer=<name>)
- google_compute_disk_resource_policy_attachment.data_backup: attach policy
  to each data disk (prod + dev)
- google_monitoring_uptime_check_config.health: per-VM /api/health uptime
  check every 60s, 10s timeout
- google_monitoring_alert_policy.health_failure: alert when uptime check
  fails for > 5 min

New opt-out: enable_monitoring = false (default true)
New opt-in:  notification_channel_ids = [...] to wire alerts to email/Slack

Module API unchanged; existing customers pick up backups + monitoring on
next module upgrade. TF provider requirement unchanged.
2026-04-21 19:01:56 +02:00
ZdenekSrotyr
1acc89c486 fix(ci): move bind-mount of /data to separate overlay, fix CI smoke test
The CI smoke test failed because docker-compose.prod.yml forced a bind mount
to /data on the host — which doesn't exist on GitHub runners.

Split the bind mount into docker-compose.host-mount.yml, which is only
composed by the VM startup script (/data exists there, mounted from the
persistent disk). CI continues to use the default named volume.

Module startup script + auto-upgrade cron now compose all three:
  -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose.host-mount.yml
2026-04-21 16:54:18 +02:00
ZdenekSrotyr
cbd85c52ed fix(infra): replace watchtower with cron for auto-upgrade
Watchtower container has Docker API mismatch (client 1.25 vs daemon 1.54+)
that can't be worked around without upstream fix. Simple cron job does the
same thing more reliably:
- Every 5 min: docker compose pull + detect digest change + up -d if changed
- Logs to /var/log/agnes-auto-upgrade.log

This removes the watchtower container and a Docker daemon dependency.
2026-04-21 16:46:55 +02:00
ZdenekSrotyr
a2c05a5d97 infra: refactor Terraform into reusable customer-instance module
Breaking changes:
- infra/main.tf, variables.tf, outputs.tf, terraform.tfvars.example removed
- Single-file monolith replaced by reusable module + example

New structure:
- infra/modules/customer-instance/ — the module:
  - main.tf: VMs, disks, firewall, Secret Manager, dedicated VM SA
  - variables.tf: prod_instance + dev_instances flexible schema
  - outputs.tf: IPs, SA email, JWT secret reference
  - startup-script.sh.tpl: bootstraps VM, fetches secrets, runs compose,
    adds Watchtower for auto-upgrade
- infra/examples/minimal/ — OSS self-host quickstart using the module

Supports:
- Per-customer GCP project isolation
- Branch-aware dev VMs via dev_instances list (any image_tag)
- Persistent /data disk (rebuild-safe)
- OS Login (no per-user SSH keys)
- Caddy TLS mode (opt-in via tls_mode="caddy" + domain)
- Watchtower auto-upgrade (opt-in via upgrade_mode="auto")
2026-04-21 16:18:35 +02:00
ZdenekSrotyr
622c5005aa feat(infra): add GCS remote backend and instance.yaml generation to startup script 2026-04-09 16:39:07 +02:00
ZdenekSrotyr
89154d043b chore: clean repo for public release — fix references, remove drafts
- Replace padak/tmp_oss → keboola/agnes-the-ai-analyst in all docs, infra, CLI
- Replace your-org/ai-data-analyst → keboola/agnes-the-ai-analyst in README, Jira docs
- Remove real GCP project ID from terraform.tfvars.example
- Delete internal draft documents (dev_docs/draft/)
- Update infra/main.tf to clone from main branch
2026-04-08 19:27:25 +02:00
ZdenekSrotyr
04c5aecc58 fix: update Terraform for extract.duckdb architecture
- Create /data/extracts instead of /data/src_data/parquet
- Add admin_email variable for SEED_ADMIN_EMAIL
2026-03-31 09:49:32 +02:00
ZdenekSrotyr
a4944dba4a feat: auto-generate JWT secret in Terraform, remove manual variable 2026-03-30 16:03:19 +02:00
ZdenekSrotyr
b6a94add67 feat: add Terraform config for GCP deployment
- GCE e2-small with Ubuntu 24.04 + Docker
- Static IP, firewall rules, SSD boot disk
- Startup script: installs Docker, clones repo, creates .env, starts compose
- Outputs: IP, SSH command, API URL, bootstrap command, CLI setup
- ~7$/month for always-on server
2026-03-30 15:55:26 +02:00