agnes-the-ai-analyst/tests/test_jira_service_full.py
ZdenekSrotyr 2f783c5c0a
fix(security): close Jira webhook fail-open + path traversal (#83) (#93)
* fix(security): close Jira webhook fail-open + path traversal (#83)

Two related vulnerabilities:

1. Fail-open signature check: when JIRA_WEBHOOK_SECRET was unset,
   _verify_signature returned True and any unauthenticated POST to
   /webhooks/jira would run the full ingest pipeline. Now fail-closed —
   the handler short-circuits with 503 (operator-misconfiguration signal,
   distinct from 401 wrong-signature) when the secret is missing.

2. Path traversal via attacker-controlled issue_key: webhook payloads
   carry issue.key, which flowed unsanitized into save_issue (issues_dir /
   "{issue_key}.json"), download_attachment (attachments_dir / issue_key),
   and incremental_transform (raw_dir / "issues" / "{issue_key}.json"). A
   crafted webhook with issue.key="../../etc/passwd" could write outside
   the Jira data dir.

Defense-in-depth: new connectors/jira/validation.py exposes
is_valid_issue_key (whitelist regex ^[A-Z][A-Z0-9_]{0,31}-\d{1,12}$) and
safe_join_under (Path.resolve() containment check). Both are enforced at
the webhook entry point AND at every filesystem boundary in the connector.

Tests:
- New tests/test_jira_validation.py — unit tests for both helpers
  (parametrized invalid keys, traversal/symlink/absolute-path cases).
- Webhook tests: test_unconfigured_secret_returns_503,
  test_path_traversal_in_issue_key_rejected (parametrized over 10 bad keys),
  test_valid_issue_key_accepted.

CHANGELOG: two CRITICAL Fixed bullets under Unreleased.

Closes #83.

* fix(security): close remaining #83 review findings — webhookEvent traversal, _handle_deletion guard, regex tightening

Reviewer of PR #93 flagged four MUST-FIXes:

1. _log_webhook_event used the attacker-controlled `webhookEvent` field
   as a filename component without sanitization. Payload with
   `webhookEvent: "../../tmp/pwn"` could escape WEBHOOK_LOG_DIR. Now:
   - non-`[A-Za-z0-9_-]` runs are replaced with `_` (dot excluded so
     `..` cannot survive sanitization as a directory component)
   - length capped at 64 chars
   - final path routed through safe_join_under
   New regression test `test_webhook_event_path_traversal_sanitized`.

2. _handle_deletion (connectors/jira/service.py:530) and
   process_webhook_event (line 487) still used raw issue_key in path
   builds. Even though the webhook handler validates upstream, the
   "defense-in-depth at every filesystem boundary" claim required these
   too. Both now run is_valid_issue_key and safe_join_under guards.

3. Regex `^[A-Z][A-Z0-9_]{0,31}-\d{1,12}$` permitted underscores in
   project keys. Atlassian's project-key validator does not — `A_B-1`
   is rejected by Jira itself. Tightened to `[A-Z0-9]` and updated
   tests: `ABC_DEF-1` is now invalid, added Cyrillic А-1 (lookalike),
   CRLF, and oversize cases to the bad-key parametrization.

4. Existing test test_deletion_of_nonexistent_issue_returns_true used
   `PROJ-NOEXIST` which is not a real Jira key shape. Updated to
   `PROJ-99999`. The test still exercises the same intent (deletion of
   issue with no local file is idempotent).

73/73 jira tests pass locally (test_jira_webhooks + test_jira_validation
+ test_jira_service + test_jira_service_full + test_jira_incremental).

CHANGELOG updated to document the regex tightening and the new
webhookEvent sanitization.

Refs review of #93.

* fix(tests): test_journey_jira tests assumed fail-open before #83 fix

CI failure on PR #93 caught two journey tests that pinned the OLD
fail-open contract:

- test_webhook_with_no_secret_configured_accepted asserted 200 when
  JIRA_WEBHOOK_SECRET was unset. After the #83 fix that's a 503
  (operator misconfig). Renamed to _refused and flipped the assertion.

- test_webhook_empty_payload_rejected didn't set the secret, so the
  503 short-circuit fired before the empty-payload 400 could. Set
  JIRA_WEBHOOK_SECRET in the patched Config so the test exercises the
  intended path.

56/56 jira journey + webhook + validation tests now pass.

* fix(security): #93 round-3 — webhook fallback format + save_issue early validation

Devin Review caught two real findings:

1. Webhook handler regression: the round-2 fix extracted issue_key only
   from event_data['issue']['key'], but process_webhook_event has long
   supported a fallback 'issue_key' top-level field for certain Jira
   event formats (e.g. delete events historically). The handler now
   blocks those events with 400 before they reach the service layer.
   Fix: mirror process_webhook_event's fallback in the handler — try
   issue.key first, fall through to event_data.get('issue_key') when
   empty. is_valid_issue_key still validates whichever source provided
   the key.

2. save_issue defense-in-depth was incomplete: is_valid_issue_key ran
   AFTER fetch_remote_links and fetch_sla_fields had already used the
   unvalidated issue_key in HTTP URL construction
   ({base_url}/issue/{issue_key}/remotelink etc.). A future internal
   caller invoking save_issue directly with attacker-controlled input
   could trigger outbound requests with a malicious path component
   (limited SSRF / URL-path manipulation against the Jira API server).
   Fix: move the is_valid_issue_key check to immediately after the
   null guard, before any HTTP request or filesystem op. Webhook layer
   still validates upstream, this is the second layer.

66 jira tests pass.

Refs Devin Review of #93.

* fix(changelog): #93 round-4 — add BREAKING marker to fail-closed bullet

Devin Review caught: the JIRA_WEBHOOK_SECRET fail-closed change is a
behavior change for operators (response code 503 vs old 200) that
existing alerting may treat differently. Per CLAUDE.md changelog
discipline rule, operators grep for **BREAKING** before bumping the
pin. Added the marker + a short note on what action operators need
to take (set the env var if they haven't).

Refs Devin Review of #93.

* fix: #93 round-5 — null-issue crash + comment drift

Devin Review caught two findings on the round-4 commit:

1. Pre-existing crash on null issue field: a webhook payload with
   {"issue": null} (rather than omitting the key) caused
   event_data.get("issue", {}) to return None, then issue.get("key")
   raised AttributeError → unhandled 500. Pre-existing but reachable.
   Fix: 'event_data.get("issue") or {}' normalises None to {}, then
   the existing fallback / validation path returns 400 cleanly.
   New regression test test_null_issue_field_does_not_crash.

2. Inline comment drift: the comment at line 77 documented the allowed
   character class as [A-Za-z0-9._-] (with dot) but the regex at line 27
   excludes dot deliberately (so '..' cannot survive sanitization).
   Fixed the comment to match.

52 jira tests pass.

Refs Devin Review of #93 round 5.

* fix: #93 round-6 — process_webhook_event also normalises null issue field

Devin Review caught: the webhook handler at app/api/jira_webhooks.py
correctly handles {"issue": null} via 'event_data.get("issue") or {}',
but process_webhook_event at connectors/jira/service.py:509 still
used the bare 'event_data.get("issue", {})' which returns None on
explicit null. Internal callers (anything that invokes
process_webhook_event without going through the HTTP handler) would
hit the same AttributeError the round-5 fix closed at the handler
layer. Same one-line fix.

32 jira tests pass.

Refs Devin Review of #93 round 5.

* fix: #93 round-7 — issue-key regex uses [0-9] not \d

Devin Review caught: Python 3's \d matches any Unicode decimal digit
(Arabic-Indic ٣, Bengali ৩, Devanagari ३, …). A key like TEST-٣ would
pass the regex even though it's not a valid Jira input. Tightened to
[0-9] (ASCII only).

Added three Unicode-digit cases to the bad-key parametrization in
test_jira_validation.py to lock in the contract.

Refs Devin Review of #93 round 6.

* fix: #93 round-8 — use \\Z anchor not $ in issue-key regex

Devin Review caught: Python's $ anchor matches before a trailing \\n,
so re.match('…$', 'TEST-1\\n') returns a match. is_valid_issue_key
returned True for CRLF-injected keys. \\Z is hard end-of-string and
closes that bypass.

Manual verification:
  is_valid_issue_key('TEST-1\\n') → False (was True before fix)
  is_valid_issue_key('TEST-1\\r\\n') → False
  is_valid_issue_key('TEST-1') → True

Refs Devin Review of #93 round 7.

* docs: #93 round-9 — CHANGELOG regex matches implementation
2026-04-27 19:53:55 +02:00

182 lines
7 KiB
Python

"""Full tests for the Jira service (JiraService.process_webhook_event and friends)."""
import json
from pathlib import Path
from unittest.mock import MagicMock, patch
import pytest
from tests.helpers.factories import WebhookEventFactory
@pytest.fixture
def jira_env(tmp_path, monkeypatch):
"""Set up a Jira environment with required dirs and env vars."""
data_dir = tmp_path / "jira_data"
data_dir.mkdir()
(data_dir / "issues").mkdir()
monkeypatch.setenv("JIRA_DOMAIN", "mycompany.atlassian.net")
monkeypatch.setenv("JIRA_EMAIL", "bot@mycompany.com")
monkeypatch.setenv("JIRA_API_TOKEN", "test-token-xyz")
monkeypatch.setenv("JIRA_DATA_DIR", str(data_dir))
monkeypatch.setenv("JIRA_WEBHOOK_SECRET", "webhook-secret-123")
return data_dir
def _make_jira_service(jira_env):
"""Create a fresh JiraService with test configuration."""
from connectors.jira import service as svc
svc.Config.JIRA_DOMAIN = "mycompany.atlassian.net"
svc.Config.JIRA_EMAIL = "bot@mycompany.com"
svc.Config.JIRA_API_TOKEN = "test-token-xyz"
svc.Config.JIRA_DATA_DIR = jira_env
svc.Config.JIRA_WEBHOOK_SECRET = "webhook-secret-123"
svc._jira_service = None
return svc.JiraService()
def _fake_issue_data(issue_key: str = "TEST-1") -> dict:
return {
"key": issue_key,
"id": "10001",
"fields": {
"summary": "Test issue summary",
"status": {"name": "Open"},
"issuetype": {"name": "Bug"},
"attachment": [],
"comment": {"comments": []},
},
}
class TestJiraServiceWebhookProcessing:
def test_process_issue_updated_calls_fetch_and_save(self, jira_env):
"""process_webhook_event for issue_updated fetches fresh data from API."""
service = _make_jira_service(jira_env)
event_data, _, _ = WebhookEventFactory.issue_updated("PROJ-100")
issue_data = _fake_issue_data("PROJ-100")
with patch.object(service, "fetch_issue", return_value=issue_data), \
patch.object(service, "fetch_remote_links", return_value=[]), \
patch.object(service, "fetch_sla_fields", return_value=None), \
patch.object(service, "download_all_attachments", return_value=[]), \
patch("connectors.jira.service.trigger_incremental_transform", return_value=True):
result = service.process_webhook_event(event_data)
assert result is True
saved_file = jira_env / "issues" / "PROJ-100.json"
assert saved_file.exists()
with open(saved_file) as f:
saved = json.load(f)
assert saved["key"] == "PROJ-100"
def test_process_issue_deleted_marks_file(self, jira_env):
"""process_webhook_event for issue_deleted marks existing JSON with _deleted_at."""
service = _make_jira_service(jira_env)
# Pre-create the issue JSON
issue_file = jira_env / "issues" / "PROJ-200.json"
issue_file.write_text(json.dumps({"key": "PROJ-200", "fields": {}}))
event_data, _, _ = WebhookEventFactory.issue_deleted("PROJ-200")
with patch("connectors.jira.service.trigger_incremental_transform", return_value=True):
result = service.process_webhook_event(event_data)
assert result is True
with open(issue_file) as f:
saved = json.load(f)
assert "_deleted_at" in saved
def test_process_missing_issue_key_returns_false(self, jira_env):
"""Webhook event without issue key returns False."""
service = _make_jira_service(jira_env)
result = service.process_webhook_event({"webhookEvent": "jira:issue_updated"})
assert result is False
def test_process_uses_embedded_data_when_fetch_fails(self, jira_env):
"""Falls back to embedded issue data in webhook payload when API fetch fails."""
service = _make_jira_service(jira_env)
event_data = {
"webhookEvent": "jira:issue_updated",
"issue": {
"key": "PROJ-300",
"id": "10003",
"fields": {
"summary": "Embedded issue",
"attachment": [],
"comment": {"comments": []},
},
},
}
with patch.object(service, "fetch_issue", return_value=None), \
patch.object(service, "fetch_remote_links", return_value=[]), \
patch.object(service, "fetch_sla_fields", return_value=None), \
patch.object(service, "download_all_attachments", return_value=[]), \
patch("connectors.jira.service.trigger_incremental_transform", return_value=True):
result = service.process_webhook_event(event_data)
assert result is True
saved_file = jira_env / "issues" / "PROJ-300.json"
assert saved_file.exists()
def test_deletion_of_nonexistent_issue_returns_true(self, jira_env):
"""Deleting an issue that has no local file returns True (idempotent)."""
service = _make_jira_service(jira_env)
event_data, _, _ = WebhookEventFactory.issue_deleted("PROJ-99999")
result = service.process_webhook_event(event_data)
assert result is True
def test_fetch_issue_returns_none_on_404(self, jira_env):
"""fetch_issue returns None when Jira returns 404."""
import httpx
service = _make_jira_service(jira_env)
mock_response = MagicMock()
mock_response.status_code = 404
mock_client = MagicMock()
mock_client.get.return_value = mock_response
mock_client.__enter__ = lambda s: mock_client
mock_client.__exit__ = MagicMock(return_value=False)
with patch("connectors.jira.service.httpx.Client", return_value=mock_client):
result = service.fetch_issue("PROJ-MISSING")
assert result is None
def test_fetch_issue_returns_data_on_200(self, jira_env):
"""fetch_issue returns parsed JSON on HTTP 200."""
service = _make_jira_service(jira_env)
issue_data = _fake_issue_data("PROJ-42")
mock_response = MagicMock()
mock_response.status_code = 200
mock_response.json.return_value = issue_data
mock_client = MagicMock()
mock_client.get.return_value = mock_response
mock_client.__enter__ = lambda s: mock_client
mock_client.__exit__ = MagicMock(return_value=False)
with patch("connectors.jira.service.httpx.Client", return_value=mock_client):
result = service.fetch_issue("PROJ-42")
assert result is not None
assert result["key"] == "PROJ-42"
def test_webhook_event_factory_signature_verification(self):
"""WebhookEventFactory produces correct HMAC-SHA256 signatures."""
import hashlib
import hmac
secret = "test-secret"
event_data, payload, sig = WebhookEventFactory.issue_updated("TEST-1", secret)
expected_mac = hmac.new(
secret.encode("utf-8"), payload, hashlib.sha256
).hexdigest()
assert sig == f"sha256={expected_mac}"