* fix(jira): harden _remote_links fetch — transient API failure no longer wipes parquet rows
Pre-fix, all three fetch_remote_links call sites (service.py,
scripts/backfill.py, scripts/backfill_remote_links.py) silently
returned [] on 401/403/429/5xx or httpx.RequestError. Callers overlaid
that [] onto cached issue JSON, and transform_remote_links interpreted
the empty list as 'issue legitimately has no remote links — delete
existing rows', so a transient Jira auth blip permanently wiped
remote-link history.
Now:
- Every fetch site raises JiraFetchError on non-200/non-404 status,
on httpx.RequestError, and on the 'service not configured' path.
- Overlay sites skip the _remote_links key when fetch raises, leaving
it ABSENT (not present-but-empty).
- transform_remote_links returns None for absent/null keys (preserve
existing rows) vs [] (legitimate empty — wipe).
- Both consumers (batch transform_all, incremental
transform_single_issue) honor the new contract.
- End-to-end tests
test_incremental_preserves_remote_links_when_overlay_absent and
test_incremental_wipes_remote_links_when_overlay_present_but_empty
lock both halves.
Adversarial-review fixes bundled:
- service.py: unconfigured-service path now raises JiraFetchError
instead of returning [] (a webhook can arrive while API creds are
missing — HMAC verification uses a separate JIRA_WEBHOOK_SECRET).
Regression guard test_raises_when_unconfigured added.
- consistency_check.py: AUTO_FIX_THRESHOLD bumped 10 -> 20 to cover
typical SLA-poller hiccups before escalating to ERROR.
- CLAUDE.md: connectors/jira/transform.py removed from 'Files NOT to
modify' (overlay-contract change required touching it; module
remains sensitive but is no longer off-limits).
* release: 0.54.19 — jira remote_links hardening (transient API failure no longer wipes parquet rows)
CLAUDE.md rewritten (708 -> ~320 lines): four overlapping release
sections collapsed to one, stale v1->v35 schema history dropped (it
lives in CHANGELOG), marketplace endpoint internals and verbose
process sections moved out or tightened.
New focused docs:
- docs/RELEASING.md - release process, deploy workflows, CI quirks
(RELEASE_TEMPLATE.md folded in as an appendix)
- docs/marketplace.md - marketplace ingestion + re-serving internals
- docs/README.md - documentation index by audience, linked from
README.md and CLAUDE.md
Archived under docs/archive/: docs/superpowers/ (52 historical
planning artifacts), HACKATHON.md, pd-ps-comments.md,
security-audit-2026-04.md, future/NOTIFICATIONS.md.
Removed the docs/auto-install.md stub. Fixed dangling links in
connectors/jira/README.md and dev_docs/README.md, repointed
code/doc references to archived paths.
Sweep operator runbooks (docs/QUICKSTART, docs/HEADLESS_USAGE,
docs/architecture, docs/sample-data, docs/agent-workspace-prompt,
docs/metrics/metrics.yml, dev_docs/server, dev_docs/disaster-recovery),
the corporate-memory service README, the jira connector README + backfill
scripts, the deploy skill, and test docstrings. Replaces `da sync` →
`agnes pull`, `da analyst setup` → `agnes init`, `da metrics ...` →
`agnes catalog --metrics` / `agnes admin metrics ...`, `da fetch` →
`agnes snapshot create`, plus the matching docker-compose admin
invocations.
Vendor-specific `/opt/data-analyst/` install paths in jira backfill /
consistency scripts and operator docs are replaced with the
placeholder `<install-dir>` and a new `AGNES_ENV_FILE` env-var override
that lets a deployment inject its actual install path without a code
change. Aligns with the OSS vendor-agnostic policy in CLAUDE.md.
CHANGELOG `### Internal` entry summarizes the audit and reaffirms the
intentional stale-marker tuples (`_LEGACY_STRINGS`, `_OUR_COMMAND_MARKERS`)
that must keep referencing `da sync` / `da fetch` / etc. for hook upgrade
and override-detection logic.
* fix(security): close Jira webhook fail-open + path traversal (#83)
Two related vulnerabilities:
1. Fail-open signature check: when JIRA_WEBHOOK_SECRET was unset,
_verify_signature returned True and any unauthenticated POST to
/webhooks/jira would run the full ingest pipeline. Now fail-closed —
the handler short-circuits with 503 (operator-misconfiguration signal,
distinct from 401 wrong-signature) when the secret is missing.
2. Path traversal via attacker-controlled issue_key: webhook payloads
carry issue.key, which flowed unsanitized into save_issue (issues_dir /
"{issue_key}.json"), download_attachment (attachments_dir / issue_key),
and incremental_transform (raw_dir / "issues" / "{issue_key}.json"). A
crafted webhook with issue.key="../../etc/passwd" could write outside
the Jira data dir.
Defense-in-depth: new connectors/jira/validation.py exposes
is_valid_issue_key (whitelist regex ^[A-Z][A-Z0-9_]{0,31}-\d{1,12}$) and
safe_join_under (Path.resolve() containment check). Both are enforced at
the webhook entry point AND at every filesystem boundary in the connector.
Tests:
- New tests/test_jira_validation.py — unit tests for both helpers
(parametrized invalid keys, traversal/symlink/absolute-path cases).
- Webhook tests: test_unconfigured_secret_returns_503,
test_path_traversal_in_issue_key_rejected (parametrized over 10 bad keys),
test_valid_issue_key_accepted.
CHANGELOG: two CRITICAL Fixed bullets under Unreleased.
Closes#83.
* fix(security): close remaining #83 review findings — webhookEvent traversal, _handle_deletion guard, regex tightening
Reviewer of PR #93 flagged four MUST-FIXes:
1. _log_webhook_event used the attacker-controlled `webhookEvent` field
as a filename component without sanitization. Payload with
`webhookEvent: "../../tmp/pwn"` could escape WEBHOOK_LOG_DIR. Now:
- non-`[A-Za-z0-9_-]` runs are replaced with `_` (dot excluded so
`..` cannot survive sanitization as a directory component)
- length capped at 64 chars
- final path routed through safe_join_under
New regression test `test_webhook_event_path_traversal_sanitized`.
2. _handle_deletion (connectors/jira/service.py:530) and
process_webhook_event (line 487) still used raw issue_key in path
builds. Even though the webhook handler validates upstream, the
"defense-in-depth at every filesystem boundary" claim required these
too. Both now run is_valid_issue_key and safe_join_under guards.
3. Regex `^[A-Z][A-Z0-9_]{0,31}-\d{1,12}$` permitted underscores in
project keys. Atlassian's project-key validator does not — `A_B-1`
is rejected by Jira itself. Tightened to `[A-Z0-9]` and updated
tests: `ABC_DEF-1` is now invalid, added Cyrillic А-1 (lookalike),
CRLF, and oversize cases to the bad-key parametrization.
4. Existing test test_deletion_of_nonexistent_issue_returns_true used
`PROJ-NOEXIST` which is not a real Jira key shape. Updated to
`PROJ-99999`. The test still exercises the same intent (deletion of
issue with no local file is idempotent).
73/73 jira tests pass locally (test_jira_webhooks + test_jira_validation
+ test_jira_service + test_jira_service_full + test_jira_incremental).
CHANGELOG updated to document the regex tightening and the new
webhookEvent sanitization.
Refs review of #93.
* fix(tests): test_journey_jira tests assumed fail-open before #83 fix
CI failure on PR #93 caught two journey tests that pinned the OLD
fail-open contract:
- test_webhook_with_no_secret_configured_accepted asserted 200 when
JIRA_WEBHOOK_SECRET was unset. After the #83 fix that's a 503
(operator misconfig). Renamed to _refused and flipped the assertion.
- test_webhook_empty_payload_rejected didn't set the secret, so the
503 short-circuit fired before the empty-payload 400 could. Set
JIRA_WEBHOOK_SECRET in the patched Config so the test exercises the
intended path.
56/56 jira journey + webhook + validation tests now pass.
* fix(security): #93 round-3 — webhook fallback format + save_issue early validation
Devin Review caught two real findings:
1. Webhook handler regression: the round-2 fix extracted issue_key only
from event_data['issue']['key'], but process_webhook_event has long
supported a fallback 'issue_key' top-level field for certain Jira
event formats (e.g. delete events historically). The handler now
blocks those events with 400 before they reach the service layer.
Fix: mirror process_webhook_event's fallback in the handler — try
issue.key first, fall through to event_data.get('issue_key') when
empty. is_valid_issue_key still validates whichever source provided
the key.
2. save_issue defense-in-depth was incomplete: is_valid_issue_key ran
AFTER fetch_remote_links and fetch_sla_fields had already used the
unvalidated issue_key in HTTP URL construction
({base_url}/issue/{issue_key}/remotelink etc.). A future internal
caller invoking save_issue directly with attacker-controlled input
could trigger outbound requests with a malicious path component
(limited SSRF / URL-path manipulation against the Jira API server).
Fix: move the is_valid_issue_key check to immediately after the
null guard, before any HTTP request or filesystem op. Webhook layer
still validates upstream, this is the second layer.
66 jira tests pass.
Refs Devin Review of #93.
* fix(changelog): #93 round-4 — add BREAKING marker to fail-closed bullet
Devin Review caught: the JIRA_WEBHOOK_SECRET fail-closed change is a
behavior change for operators (response code 503 vs old 200) that
existing alerting may treat differently. Per CLAUDE.md changelog
discipline rule, operators grep for **BREAKING** before bumping the
pin. Added the marker + a short note on what action operators need
to take (set the env var if they haven't).
Refs Devin Review of #93.
* fix: #93 round-5 — null-issue crash + comment drift
Devin Review caught two findings on the round-4 commit:
1. Pre-existing crash on null issue field: a webhook payload with
{"issue": null} (rather than omitting the key) caused
event_data.get("issue", {}) to return None, then issue.get("key")
raised AttributeError → unhandled 500. Pre-existing but reachable.
Fix: 'event_data.get("issue") or {}' normalises None to {}, then
the existing fallback / validation path returns 400 cleanly.
New regression test test_null_issue_field_does_not_crash.
2. Inline comment drift: the comment at line 77 documented the allowed
character class as [A-Za-z0-9._-] (with dot) but the regex at line 27
excludes dot deliberately (so '..' cannot survive sanitization).
Fixed the comment to match.
52 jira tests pass.
Refs Devin Review of #93 round 5.
* fix: #93 round-6 — process_webhook_event also normalises null issue field
Devin Review caught: the webhook handler at app/api/jira_webhooks.py
correctly handles {"issue": null} via 'event_data.get("issue") or {}',
but process_webhook_event at connectors/jira/service.py:509 still
used the bare 'event_data.get("issue", {})' which returns None on
explicit null. Internal callers (anything that invokes
process_webhook_event without going through the HTTP handler) would
hit the same AttributeError the round-5 fix closed at the handler
layer. Same one-line fix.
32 jira tests pass.
Refs Devin Review of #93 round 5.
* fix: #93 round-7 — issue-key regex uses [0-9] not \d
Devin Review caught: Python 3's \d matches any Unicode decimal digit
(Arabic-Indic ٣, Bengali ৩, Devanagari ३, …). A key like TEST-٣ would
pass the regex even though it's not a valid Jira input. Tightened to
[0-9] (ASCII only).
Added three Unicode-digit cases to the bad-key parametrization in
test_jira_validation.py to lock in the contract.
Refs Devin Review of #93 round 6.
* fix: #93 round-8 — use \\Z anchor not $ in issue-key regex
Devin Review caught: Python's $ anchor matches before a trailing \\n,
so re.match('…$', 'TEST-1\\n') returns a match. is_valid_issue_key
returned True for CRLF-injected keys. \\Z is hard end-of-string and
closes that bypass.
Manual verification:
is_valid_issue_key('TEST-1\\n') → False (was True before fix)
is_valid_issue_key('TEST-1\\r\\n') → False
is_valid_issue_key('TEST-1') → True
Refs Devin Review of #93 round 7.
* docs: #93 round-9 — CHANGELOG regex matches implementation
The Flask-based webhook endpoint at connectors/jira/webhook.py is no longer used.
FastAPI handles webhooks via app/api/jira_webhooks.py which is already integrated
into the application. This removes the redundant Flask code.
DuckDB has used WAL by default since v0.8, so this pragma is not
valid DuckDB syntax. Removed obsolete try-except block that attempted
to enable WAL on system database initialization.
- Replace padak/tmp_oss → keboola/agnes-the-ai-analyst in all docs, infra, CLI
- Replace your-org/ai-data-analyst → keboola/agnes-the-ai-analyst in README, Jira docs
- Remove real GCP project ID from terraform.tfvars.example
- Delete internal draft documents (dev_docs/draft/)
- Update infra/main.tf to clone from main branch
- New extract_init.py: creates extract.duckdb with _meta + views for 6 entity types
- Update default paths to /data/extracts/jira/data/ and /data/extracts/jira/raw/
- After parquet writes, update _meta table in extract.duckdb
- Trigger SyncOrchestrator.rebuild_source("jira") after successful transform
Move all Jira-specific code into a self-contained connector module:
- 22 files moved via git mv (transform, service, webhook, scripts,
systemd units, tests, docs, bin helper)
- All imports updated to use connectors.jira.* paths
- Jira is now conditional: auto-detected via JIRA_DOMAIN env var
- Webapp registers Jira blueprint only when available
- Health service monitors Jira timers only when enabled
- Profiler loads Jira tables dynamically from filesystem
- Sync settings uses config-driven dependency validation
- Renamed keboola_platform_url -> custom_url in transform
- Updated deploy.sh, sudoers-deploy, backfill_gap.sh paths
- Fixed pytest.ini to skip live tests by default
