Commit graph

27 commits

Author SHA1 Message Date
ZdenekSrotyr
963db420fe
ci(release): push dev-<user-prefix>-latest alias for <user>/* branches (#31)
Adds a second tag to dev-channel image builds: when a branch is in the
form <prefix>/<whatever>, the image is also pushed as
ghcr.io/keboola/agnes-the-ai-analyst:dev-<prefix>-latest.

Enables per-developer dev VMs on GRPN (and elsewhere) to auto-deploy
without knowing the specific branch slug. Each VM pins its .env to
AGNES_TAG=dev-<prefix>-latest, and the auto-upgrade cron (5 min tick)
picks up the newly pushed image on the next run.

Common Git Flow prefixes are deliberately skipped so feature/*, fix/*,
hotfix/* etc. don't create noise tags. Matched list:
feature, fix, hotfix, bugfix, docs, chore, test, ci, ops, refactor,
perf, style, build.

Verified locally against several branch names:
  zs/my-feature     -> dev-zs-latest
  vr/foo            -> dev-vr-latest
  pc/bar-baz        -> dev-pc-latest
  feature/xyz       -> (skipped)
  fix/bug           -> (skipped)
  main              -> (no-op, stable channel)
  test-no-slash     -> (no-op, no slash)
2026-04-22 14:02:59 +02:00
ZdenekSrotyr
4f381dc103
fix(ci): propagate-infra-tag fail-soft on branch push / missing secret (#24)
Job-level 'if: secrets.X != ""' did not prevent workflow from being
scheduled on branch pushes (GitHub reports failure with 0 jobs in that
case). Refactored: first step is a guard that checks both the tag ref
pattern and the secret presence; downstream steps skip when the guard
says no.

Result: workflow now reports success with a clear warning annotation on
branch pushes or when the secret is absent; only real infra-v* tag
pushes with the secret set perform the bump.
2026-04-21 21:59:10 +02:00
ZdenekSrotyr
e2eb51f657
ci(release): build image for all branches, not just feature/** (#19)
* dryrun: intentional failing test (will be reverted)

* feat(auth): optional SEED_ADMIN_PASSWORD to pre-hash seed admin (dev helper)

Terraform gains enable_seed_password + seed_admin_password (sensitive) vars
on the customer-instance module; when enabled the password is piped via
startup-script into /opt/agnes/.env as SEED_ADMIN_PASSWORD. On first boot
app/main.py argon2-hashes it onto the seed user so the admin can log in
immediately without going through /auth/bootstrap. Never overwrites an
existing password_hash — safe against accidental reset on terraform apply.

* ci(release): build :dev-<slug> on any branch, not just feature/**

Before: only 'feature/**' branches triggered release.yml, so pushing
'zs/my-edit' or 'fix/bug' did not publish an image. dev_instances entry
pinning image_tag = 'dev-zs-my-edit' then crashed VM startup with
'image not found'.

Now: any branch push (except main, which produces :stable) publishes
:dev-<slug>. Slug strips a leading 'feature/' and replaces non-[a-z0-9-]
with '-', keeping existing feature/** behavior identical.

* Revert "dryrun: intentional failing test (will be reverted)"

This reverts commit cf9cc06a7884bb401ff29fc5cb6d8baf84dc3daa.
2026-04-21 21:33:57 +02:00
ZdenekSrotyr
2cbffce85f
ci: propagate infra-v* tags to template repo + auto-merge rules (#17)
* dryrun: verify per-branch GHCR tag

* ci: propagate infra-v* tag bumps to template repo

On push of any infra-v* tag, opens a PR in keboola/agnes-infra-template
that bumps the module ref in terraform/main.tf. Auto-merge rules in the
template (Renovate + CI validate + GitHub native auto-merge) land it
without manual work on patch/minor bumps.

Requires repo secret TEMPLATE_REPO_TOKEN (fine-grained PAT with
Contents:write + Pull requests:write on keboola/agnes-infra-template).

Fail-soft: if secret is missing the job is skipped and Renovate on the
template repo picks up the new tag on its next cycle as a fallback.

* docs(onboarding): 'Keeping the template up-to-date' maintainer section

Documents the two mechanisms (upstream release hook + Renovate), the
required repo settings (allow_auto_merge, validate.yml gate), the TOKEN
secret setup, and the one-time setup checklist. Notes the difference
between template repo (auto-merge on) and customer infra repos
(human approval).
2026-04-21 21:32:58 +02:00
ZdenekSrotyr
1c7cc8aa29 fix(image): add AGNES_COMMIT_SHA build-arg to Dockerfile + release.yml
Completes the previous commit — bakes the full git SHA into the image ENV
at build time so the UI badge shows a real commit, not a sha256 digest
(which was the floating manifest digest and unhelpful for debugging).
2026-04-21 21:00:30 +02:00
ZdenekSrotyr
5188bd9127 ci: add per-branch image tag :dev-<slug> for branch-aware dev deploys
Extracts branch name from GITHUB_REF, slugifies it, and adds as extra tag
on feature branch builds. Main branch is unaffected (no branch_slug output).

Enables dev_instances tfvar with image_tag pinning specific feature branches.
2026-04-21 18:47:01 +02:00
ZdenekSrotyr
5bbd82bacd fix: address Devin review — docker-e2e .env, jira webhook test isolation
- Create empty .env before docker compose up in CI (env_file: .env is required)
- Mock get_jira_service in webhook HMAC test to isolate signature check
  from Jira API availability — strict assert 200 instead of permissive 500
2026-04-13 14:36:31 +02:00
ZdenekSrotyr
5bfff6616c ci: add parallel test execution and nightly Docker E2E job 2026-04-12 14:15:46 +02:00
ZdenekSrotyr
44b99f25ca fix: address Devin review round 5 — empty secret file, CI .env
- secrets.py: validate file content is non-empty before using it;
  regenerate if file exists but is empty/corrupted
- release.yml: touch .env before docker compose in smoke test
  (env_file: .env in docker-compose.yml requires the file to exist)

663 tests pass.
2026-04-10 14:55:31 +02:00
ZdenekSrotyr
40cca627be fix: address Devin review round 4 — bash arithmetic, CalVer max, docs
- smoke-test.sh: replace ((PASS++)) with PASS=$((PASS + 1)) to avoid
  set -e abort when counter is 0 (bash returns exit 1 for ((0)))
- CalVer: use max(N) from existing tags instead of count, safe when
  tags are deleted (e.g. deprecated version cleanup)
- CLAUDE.md: update schema version from v2 to v3

663 tests pass.
2026-04-10 14:39:16 +02:00
ZdenekSrotyr
dc8a9275e6 fix: address Devin review round 3 — retry exhaustion, discover path, WAL snapshot
- CalVer retry loop now exits with error if all 5 attempts fail
  (prevents pushing Docker image with unclaimed version tag)
- discover_tables endpoint reads data_source.keboola.url (consistent
  with configure_instance and _discover_and_register_tables)
- Pre-migration snapshot flushes WAL via CHECKPOINT before copying
  and copies .wal file if it still exists after flush

663 tests pass.
2026-04-10 14:11:17 +02:00
ZdenekSrotyr
c79d85f87c fix: config path mismatch + CalVer race condition (Devin review round 2)
- _discover_and_register_tables reads from data_source.keboola.url
  (matches what /api/admin/configure writes) instead of top-level
  keboola.url which doesn't exist
- CalVer: claim git tag BEFORE Docker build with retry loop (up to 5
  attempts). Prevents race where two concurrent CI runs get same N.
  Git tag acts as a distributed lock for version uniqueness.

663 tests pass.
2026-04-10 13:30:05 +02:00
ZdenekSrotyr
49f109bf73 fix: address PR review findings — config write, CalVer, error handling
- Config writes to DATA_DIR/state/instance.yaml (writable) instead of
  CONFIG_DIR (read-only :ro in Docker)
- instance_config.py checks DATA_DIR/state/ first, then falls back to
  CONFIG_DIR for backward compat
- CalVer counter is now global across channels (*-YYYY.MM.*) per spec
- Keboola error messages sanitized — log full error, return generic msg
- chmod in secrets.py wrapped in try/except for Windows compat
- Setup wizard JS handles 401 (expired JWT) with user-facing message
- deploy.yml changed to workflow_dispatch only (no duplicate test runs)
- Smoke test uses docker-compose.prod.yml + AGNES_TAG instead of sed
- docker-compose.prod.yml uses ${AGNES_TAG:-stable} env var

663 tests pass. 8 E2E verification tests pass.
2026-04-10 13:16:40 +02:00
ZdenekSrotyr
6c53082295 feat: multi-instance deployment — all 14 must-have items from spec
CalVer CI (release.yml) with stable/dev channels, health endpoint
with version/channel/schema_version, JWT secret auto-generation with
file persistence, smoke test script + Docker-in-CI, pre-migration
snapshot, /api/admin/configure for headless setup, /api/admin/
discover-and-register, /setup wizard, OpenAPI snapshot test, custom
connector mount support, CHANGELOG, migration safety tests, startup
banner.

663 tests pass (6 new migration safety + 3 OpenAPI snapshot + 1
updated JWT test).
2026-04-10 11:57:42 +02:00
ZdenekSrotyr
816f217d8e feat: add commit SHA tag to Docker image push for rollback capability 2026-04-09 16:38:38 +02:00
ZdenekSrotyr
22b4d830e5 chore: upgrade docker actions to Node.js 24 (login-action@v4, build-push-action@v7) 2026-04-09 14:22:11 +02:00
ZdenekSrotyr
6ebfc15010 fix: setup-uv@v7 (v8 major tag doesn't exist yet) 2026-04-09 14:19:32 +02:00
ZdenekSrotyr
1ebf50bd78 fix: upgrade setup-uv@v5 → v8 (Node.js 24 native), add uv.lock
- setup-uv@v8 runs on Node.js 24 natively — no more deprecation warnings
- Removed FORCE_JAVASCRIPT_ACTIONS_TO_NODE24 workaround (no longer needed)
- Added uv.lock for reproducible dependency resolution
2026-04-09 14:16:55 +02:00
ZdenekSrotyr
554ba0d9f2 fix: remove Kamal deploy job (no server configured), force Node.js 24 in CI
- Removed deploy-production job — Kamal config has placeholder IPs, no secrets
- Renamed workflow to "Build & Push" — test + Docker image to GHCR
- Added FORCE_JAVASCRIPT_ACTIONS_TO_NODE24=true to suppress Node.js 20 warnings
2026-04-09 14:10:37 +02:00
ZdenekSrotyr
0279cc06fa refactor: consolidate deps into pyproject.toml, remove requirements.txt
- All dependencies now in pyproject.toml [project.dependencies]
- Dev/test deps in [project.optional-dependencies] dev and [tool.uv]
- Dockerfile uses uv pip install . from pyproject.toml
- CI uses uv pip install ".[dev]"
- Deleted requirements.txt and requirements-dev.txt
- Updated README, CLAUDE.md install instructions
- Enhanced .dockerignore (exclude tests, docs, infra from image)
2026-04-09 13:17:59 +02:00
ZdenekSrotyr
fa3aef652f chore: update GitHub Actions to Node.js 24 compatible versions (checkout@v5, setup-python@v6, setup-uv@v5) 2026-04-09 12:48:14 +02:00
ZdenekSrotyr
f9fae6e895 fix: CI installs requirements-dev.txt (faker needed for tests), set TESTING=1 2026-04-09 09:10:29 +02:00
ZdenekSrotyr
2635f77974 ci: add CI test suite + deploy pipeline
- ci.yml: runs 607 tests + Docker build on push/PR
- deploy.yml: tests → build → GHCR push → Kamal deploy on main
2026-04-08 18:24:05 +02:00
ZdenekSrotyr
cfa08c4b4c chore: remove obsolete CI workflows (deploy-guard, deploy.yml.example)
deploy-guard.yml referenced deleted tests and sudoers files.
deploy.yml.example used legacy SSH-based deployment.
Updated ci.yml and deploy.yml are in .gitignore (need workflow scope to push).
2026-04-08 18:16:48 +02:00
ZdenekSrotyr
a74f69d6b1 chore: exclude CI workflow from push (needs workflow scope) 2026-03-27 17:41:27 +01:00
ZdenekSrotyr
e0ce91ddb9 feat: add dataset permissions, script execution, Kamal config, CI/CD
- SyncSettingsRepository + DatasetPermissionRepository with RBAC
- Script deploy/run/undeploy API with import sandboxing
- User sync settings API with permission checks
- 4 CLI skills (connectors, security, notifications, corporate-memory)
- Kamal production + staging configs
- GitHub Actions CI + deploy workflows
- 91 total tests passing
2026-03-27 15:40:11 +01:00
Petr
c56905d34f Initial commit: OSS data distribution platform
Open-source AI data analyst platform extracted from internal repo.
Includes data sync engine, Keboola adapter, Flask web portal,
server deployment scripts, and configuration templates.
2026-03-08 23:31:28 +01:00