agnes-the-ai-analyst/tests
ZdenekSrotyr 24e81fb671
fix(security): gate Script-API /run on admin role (#44) (#92)
* fix(security): gate Script-API /run on admin role (#44)

The AST + string-blocklist sandbox in `_execute_script` is defense-in-depth,
not a primary trust boundary. It does not block `vars()`, `type()`, or
`__class__.__bases__` introspection chains, and the string blocklist is
trivially evadable via concatenation/dunder encoding. Treat the role gate
as the actual barrier: only admin can run scripts.

- `POST /api/scripts/run` and `POST /api/scripts/{id}/run` now require admin.
- `POST /api/scripts/deploy` stays analyst-accessible (storing != executing).
- Existing /run tests retargeted to admin_token; added regression tests
  asserting analyst → 403 on both endpoints.
- CHANGELOG: BREAKING (security) bullet under Unreleased/Changed.

Closes #44.

* fix(security): admin-gate /deploy + harden sandbox blocklist (review #92)

Reviewer of PR #92 flagged three MUST-FIXes that #44 wasn't fully closed:

1. /api/scripts/deploy still accepted analyst → planted-script attack
   path (analyst plants malicious source, waits for admin to /run).
   Now: /deploy also requires admin; the entire Script API is admin-only.

2. The "Minimum (same-day)" blocklist mitigations from issue #44 weren't
   applied. Added the introspection-chain dunders that the issue PoC
   pivots through: __subclasses__, __globals__, __class__, __base__,
   __bases__, __mro__, __dict__, __code__, __builtins__. Plus `vars`
   in BLOCKED_FUNCTIONS. Deliberately NOT adding __init__ /
   __getattribute__ (substring match would flag every legit `def __init__`)
   nor `type`/`dir` (frequent in legitimate admin scripts). Documented
   the trade-off inline.

3. Tests didn't cover the actual PoC payload nor non-analyst non-admin
   roles. Added test_run_pwn_payload_blocked parametrized over the issue's
   own PoC + two equivalent variants (lambda+__globals__, __mro__
   traversal); these stay green only as long as the dunder list does.
   test_*_requires_admin tests now parametrize over (analyst, viewer,
   km_admin) so all three non-admin core roles are pinned at 403.

Conftest extension: seeded_app now exposes viewer_token and
km_admin_token as siblings to admin_token / analyst_token.

CHANGELOG bullet updated to reflect /deploy gate change and new
internal regression tests. 35/35 scripts tests pass locally.

Refs review of #92.

* fix(tests): test_security TestScriptSandbox needs admin token after #44 hardening

CI failure on PR #92 caught a missed test file. tests/test_security.py
seeded only an analyst user and used the analyst token to drive sandbox
tests. After the #44 admin-gate (deploy + run both admin-only), every
sandbox test got 403 from the role gate before the AST/string check
could run, so 'blocks os.system' / 'blocks eval' / etc. all failed.

Fix: extend the fixture to also seed an admin user and return the admin
token. Sandbox tests now reach the sandbox layer; access-control tests
further down in the module continue to use the analyst that was kept
around. 41/41 test_security.py tests pass locally.

* fix(security): #92 round-3 — gate GET /api/scripts on admin role

Devin Review caught: GET /api/scripts (app/api/scripts.py:44-51) was
left on Depends(get_current_user) when the rest of the API moved to
admin-only. ScriptRepository.list_all() does SELECT * FROM script_registry
which returns ALL columns including 'source' (the full script body).
So any authenticated user (viewer / analyst / km_admin) could read
admin-deployed scripts — leak of code that may contain credentials,
business logic, or admin-only operational details.

CHANGELOG already says 'The entire Script API is now admin-only',
which was true for /deploy, /run, /{id}/run, DELETE — just not for
GET. Now consistent: every Script endpoint requires admin.

Tests:
- New parametrized test_list_scripts_requires_admin over (analyst,
  viewer, km_admin) tokens — all assert 403.
- Updated test_list_scripts_empty in both test_scripts_api.py and
  test_api_scripts.py to use admin_token.

79 tests pass.

Refs Devin Review of #92.

* fix: cleanup unused imports, stale docstrings, and incomplete CHANGELOG

- Remove unused imports: Path, List, get_current_user (ruff F401)
- Trim docstrings to describe current behavior, not change history
- CHANGELOG now lists GET /api/scripts among admin-gated endpoints
- Remove diff-commenting inline comments from tests

Co-Authored-By: zdenek.srotyr <zdenek.srotyr@keboola.com>

* fix: merge duplicate Changed sections into one per CLAUDE.md convention

Co-Authored-By: zdenek.srotyr <zdenek.srotyr@keboola.com>

---------

Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
2026-04-27 21:13:56 +02:00
..
helpers fix: address code review findings — duplicate fixture, JWT key length, async deprecation 2026-04-13 13:47:51 +02:00
snapshots feat: multi-instance deployment — all 14 must-have items from spec 2026-04-10 11:57:42 +02:00
__init__.py Initial commit: OSS data distribution platform 2026-03-08 23:31:28 +01:00
conftest.py fix(security): gate Script-API /run on admin role (#44) (#92) 2026-04-27 21:13:56 +02:00
test_access_control.py refactor: remove legacy webapp + add missing tests + housekeeping 2026-03-31 13:44:06 +02:00
test_access_requests_api.py test: add 132 API gap tests across 8 endpoint modules 2026-04-12 11:13:24 +02:00
test_admin_configure_api.py test: add 132 API gap tests across 8 endpoint modules 2026-04-12 11:13:24 +02:00
test_admin_role_mapping_ui.py feat(auth): unified role management — UI + REST API + CLI + schema v9 (v0.11.4) (#73) 2026-04-27 02:23:01 +02:00
test_admin_tokens_ui.py feat(auth): Google Workspace groups on /profile + tag-triggered Keboola deploy workflow (#56) 2026-04-26 00:56:44 +02:00
test_admin_user_capabilities_ui.py feat(auth): unified role management — UI + REST API + CLI + schema v9 (v0.11.4) (#73) 2026-04-27 02:23:01 +02:00
test_analyst_bootstrap.py feat: add da analyst setup command with bootstrap flow 2026-04-10 19:43:36 +02:00
test_api.py feat: add POST /api/query/hybrid endpoint for two-phase BQ+DuckDB queries 2026-04-11 11:09:42 +02:00
test_api_complete.py fix: return filename instead of absolute path in upload responses 2026-04-12 14:23:51 +02:00
test_api_role_management.py feat(auth): unified role management — UI + REST API + CLI + schema v9 (v0.11.4) (#73) 2026-04-27 02:23:01 +02:00
test_api_scripts.py fix(security): gate Script-API /run on admin role (#44) (#92) 2026-04-27 21:13:56 +02:00
test_app_version.py release(2.1.0): durable sync, CLI auto-update, versioned wheel URL, version unification (#43) 2026-04-22 21:18:18 +02:00
test_auth_providers.py release(0.11.2): LOCAL_DEV_GROUPS dev mock + Makefile defaults + docs/local-development.md (#70) 2026-04-26 16:48:55 +02:00
test_auto_profiling.py Add self-service data onboarding system 2026-03-09 14:25:37 +01:00
test_bigquery_extractor.py feat: add graceful shutdown handler 2026-04-09 07:03:45 +02:00
test_bigquery_extractor_full.py test: add connector test suite (Block D) — 5 files, 58 tests 2026-04-12 11:12:50 +02:00
test_bootstrap.py fix(auth): /auth/bootstrap activates seed users, disabled only by real password 2026-04-21 20:01:20 +02:00
test_catalog_export.py chore(oss): isolate customer-specific deploy bits from scripts/grpn/ (#88, wave 1) (#94) 2026-04-27 20:24:34 +02:00
test_cli.py release(2.1.0): durable sync, CLI auto-update, versioned wheel URL, version unification (#43) 2026-04-22 21:18:18 +02:00
test_cli_admin.py User management + PAT + CLI distribution + HTML auth redirect (#9 #10 #11 #12) (#28) 2026-04-22 14:24:28 +02:00
test_cli_admin_role.py feat(auth): unified role management — UI + REST API + CLI + schema v9 (v0.11.4) (#73) 2026-04-27 02:23:01 +02:00
test_cli_analyst.py test: add CLI gap tests for all 9 command groups 2026-04-12 11:13:15 +02:00
test_cli_artifacts.py release(2.1.0): durable sync, CLI auto-update, versioned wheel URL, version unification (#43) 2026-04-22 21:18:18 +02:00
test_cli_auth.py User management + PAT + CLI distribution + HTML auth redirect (#9 #10 #11 #12) (#28) 2026-04-22 14:24:28 +02:00
test_cli_diagnose.py test: add CLI gap tests for all 9 command groups 2026-04-12 11:13:15 +02:00
test_cli_explore.py test: add CLI gap tests for all 9 command groups 2026-04-12 11:13:15 +02:00
test_cli_metrics.py test: add CLI gap tests for all 9 command groups 2026-04-12 11:13:15 +02:00
test_cli_query.py test: add CLI gap tests for all 9 command groups 2026-04-12 11:13:15 +02:00
test_cli_server.py test: add CLI gap tests for all 9 command groups 2026-04-12 11:13:15 +02:00
test_cli_sync.py release(2.1.0): durable sync, CLI auto-update, versioned wheel URL, version unification (#43) 2026-04-22 21:18:18 +02:00
test_cli_update_check.py release(2.1.0): durable sync, CLI auto-update, versioned wheel URL, version unification (#43) 2026-04-22 21:18:18 +02:00
test_column_metadata.py feat: add ColumnMetadataRepository with CRUD and proposal import 2026-04-10 19:41:53 +02:00
test_connector_kit_poc.py User management + PAT + CLI distribution + HTML auth redirect (#9 #10 #11 #12) (#28) 2026-04-22 14:24:28 +02:00
test_corporate_memory_collector.py test: add Block C services tests (68 tests across 6 files) 2026-04-12 11:11:48 +02:00
test_db.py test: add correctness test for _reattach_remote_extensions 2026-04-12 08:40:12 +02:00
test_docker_full.py fix(tests): refresh nightly docker-e2e asserts after auth + health refactors (#69) 2026-04-26 16:12:20 +02:00
test_duckdb_manager.py chore(oss): isolate customer-specific deploy bits from scripts/grpn/ (#88, wave 1) (#94) 2026-04-27 20:24:34 +02:00
test_e2e_api.py refactor: remove legacy webapp + add missing tests + housekeeping 2026-03-31 13:44:06 +02:00
test_e2e_docker.py fix(tests): refresh nightly docker-e2e asserts after auth + health refactors (#69) 2026-04-26 16:12:20 +02:00
test_e2e_extract.py fix: use SCHEMA_VERSION constant in e2e migration test 2026-04-10 19:39:19 +02:00
test_generate_sample_data.py Add --format parquet using project's ParquetManager 2026-03-10 21:46:20 +01:00
test_instance_config.py fix: address code review findings — duplicate fixture, JWT key length, async deprecation 2026-04-13 13:47:51 +02:00
test_jira_incremental.py test: add connector test suite (Block D) — 5 files, 58 tests 2026-04-12 11:12:50 +02:00
test_jira_service.py test: add missing coverage for web UI, Jira extract, instance config, and concurrent rebuild 2026-04-09 07:15:14 +02:00
test_jira_service_full.py fix(security): close Jira webhook fail-open + path traversal (#83) (#93) 2026-04-27 19:53:55 +02:00
test_jira_validation.py fix(security): close Jira webhook fail-open + path traversal (#83) (#93) 2026-04-27 19:53:55 +02:00
test_jira_webhooks.py fix(security): close Jira webhook fail-open + path traversal (#83) (#93) 2026-04-27 19:53:55 +02:00
test_journey_analyst.py test: add E2E journey tests (J1-J8) covering full user flows 2026-04-12 11:13:51 +02:00
test_journey_bootstrap_auth.py test: add E2E journey tests (J1-J8) covering full user flows 2026-04-12 11:13:51 +02:00
test_journey_hybrid.py test: add E2E journey tests (J1-J8) covering full user flows 2026-04-12 11:13:51 +02:00
test_journey_jira.py fix(security): close Jira webhook fail-open + path traversal (#83) (#93) 2026-04-27 19:53:55 +02:00
test_journey_memory.py test: add E2E journey tests (J1-J8) covering full user flows 2026-04-12 11:13:51 +02:00
test_journey_multisource.py test: add E2E journey tests (J1-J8) covering full user flows 2026-04-12 11:13:51 +02:00
test_journey_rbac.py test: add E2E journey tests (J1-J8) covering full user flows 2026-04-12 11:13:51 +02:00
test_journey_sync_query.py test: add E2E journey tests (J1-J8) covering full user flows 2026-04-12 11:13:51 +02:00
test_keboola_extractor.py feat: add graceful shutdown handler 2026-04-09 07:03:45 +02:00
test_keboola_extractor_full.py test: add connector test suite (Block D) — 5 files, 58 tests 2026-04-12 11:12:50 +02:00
test_live_bigquery.py test: add Docker E2E and live connector test files 2026-04-12 11:10:06 +02:00
test_live_jira.py test: add Docker E2E and live connector test files 2026-04-12 11:10:06 +02:00
test_live_keboola.py test: add Docker E2E and live connector test files 2026-04-12 11:10:06 +02:00
test_llm_connector.py Add modular LLM connector for Corporate Memory 2026-03-23 12:08:33 +01:00
test_llm_providers_full.py test: add connector test suite (Block D) — 5 files, 58 tests 2026-04-12 11:12:50 +02:00
test_memory_api.py test: add 132 API gap tests across 8 endpoint modules 2026-04-12 11:13:24 +02:00
test_metadata_api.py test: add 132 API gap tests across 8 endpoint modules 2026-04-12 11:13:24 +02:00
test_metrics.py fix: address code review — path injection, multi-table search, metrics import API, error handling 2026-04-10 19:56:00 +02:00
test_migration.py fix: replace os.environ direct assignment with monkeypatch.setenv in test fixtures 2026-04-09 07:11:36 +02:00
test_openapi_snapshot.py feat: multi-instance deployment — all 14 must-have items from spec 2026-04-10 11:57:42 +02:00
test_openmetadata_client.py Implement OpenMetadata catalog integration (Phase 1) 2026-03-12 14:07:13 +01:00
test_openmetadata_enricher.py chore(oss): isolate customer-specific deploy bits from scripts/grpn/ (#88, wave 1) (#94) 2026-04-27 20:24:34 +02:00
test_openmetadata_transformer.py chore(oss): isolate customer-specific deploy bits from scripts/grpn/ (#88, wave 1) (#94) 2026-04-27 20:24:34 +02:00
test_orchestrator.py test: add missing coverage for web UI, Jira extract, instance config, and concurrent rebuild 2026-04-09 07:15:14 +02:00
test_password_flows.py feat(auth): password reset & invite flows for web + admin (#34) (#37) 2026-04-22 17:43:57 +02:00
test_pat.py feat(auth): Google Workspace groups on /profile + tag-triggered Keboola deploy workflow (#56) 2026-04-26 00:56:44 +02:00
test_permissions.py fix: replace os.environ direct assignment with monkeypatch.setenv in test fixtures 2026-04-09 07:11:36 +02:00
test_permissions_api.py test: add 132 API gap tests across 8 endpoint modules 2026-04-12 11:13:24 +02:00
test_profiler.py Initial commit: OSS data distribution platform 2026-03-08 23:31:28 +01:00
test_rbac.py fix: replace os.environ direct assignment with monkeypatch.setenv in test fixtures 2026-04-09 07:11:36 +02:00
test_remote_query.py fix: BQ COUNT subquery alias, wrap ImportError in RemoteQueryError 2026-04-11 20:29:03 +02:00
test_repositories.py fix: replace os.environ direct assignment with monkeypatch.setenv in test fixtures 2026-04-09 07:11:36 +02:00
test_role_resolver.py feat(auth): unified role management — UI + REST API + CLI + schema v9 (v0.11.4) (#73) 2026-04-27 02:23:01 +02:00
test_scheduler.py Support multiple daily sync times (e.g., "daily 07:00,13:00,18:00") 2026-03-16 23:09:48 +01:00
test_scheduler_full.py test: add Block C services tests (68 tests across 6 files) 2026-04-12 11:11:48 +02:00
test_schema_v9_migration.py feat(auth): unified role management — UI + REST API + CLI + schema v9 (v0.11.4) (#73) 2026-04-27 02:23:01 +02:00
test_scripts_api.py fix(security): gate Script-API /run on admin role (#44) (#92) 2026-04-27 21:13:56 +02:00
test_security.py fix(security): gate Script-API /run on admin role (#44) (#92) 2026-04-27 21:13:56 +02:00
test_selective_gzip.py release(2.1.0): durable sync, CLI auto-update, versioned wheel URL, version unification (#43) 2026-04-22 21:18:18 +02:00
test_session_collector.py test: add Block C services tests (68 tests across 6 files) 2026-04-12 11:11:48 +02:00
test_settings_api.py test: add 132 API gap tests across 8 endpoint modules 2026-04-12 11:13:24 +02:00
test_setup_instructions.py release(2.1.0): durable sync, CLI auto-update, versioned wheel URL, version unification (#43) 2026-04-22 21:18:18 +02:00
test_telegram_api.py test: add telegram API endpoint tests (verify, unlink, status) 2026-04-12 14:12:28 +02:00
test_telegram_bot.py fix: address code review findings — duplicate fixture, JWT key length, async deprecation 2026-04-13 13:47:51 +02:00
test_telegram_storage.py test: add Block C services tests (68 tests across 6 files) 2026-04-12 11:11:48 +02:00
test_upload_api.py test: add 132 API gap tests across 8 endpoint modules 2026-04-12 11:13:24 +02:00
test_user_management.py User management + PAT + CLI distribution + HTML auth redirect (#9 #10 #11 #12) (#28) 2026-04-22 14:24:28 +02:00
test_web_ui.py feat(auth): unified role management — UI + REST API + CLI + schema v9 (v0.11.4) (#73) 2026-04-27 02:23:01 +02:00
test_ws_gateway.py test: add Block C services tests (68 tests across 6 files) 2026-04-12 11:11:48 +02:00