agnes-the-ai-analyst/dev_docs/draft/services-integration.md

Service Connector - Integration of Internal APIs into Data Analyst Platform

Origin

This plan was derived from two independent proposals (Claude and Codex), both reviewed by 3 AI models (6 reviews total). The reviews identified real issues but also pushed the design toward over-engineering. This final version applies KISS and YAGNI to keep only what matters.

Previous drafts (kept for reference):

• services-integration-claude.md - Claude plan (good patterns, missing encryption)
• services-integrations-codex.md - Codex plan (SSH runtime injection, encrypted store - overkill for pilot)

What we cut and why:

• Fernet encryption of connections.json - encryption key lives on the same server as the data, security theater
• fcntl file locking - 5-10 users clicking Connect once a month, race condition probability ~0%
• Transactional connect with rollback - if token exchange fails, user clicks again
• Audit log + HMAC + logrotate - Flask access log is enough for 5 users
• Auto-rotation timer + systemd units - set long TTL, user reconnects if expired
• Feature flag rollout - just deploy when ready
• Security tests (CSRF, injection) - internal tool behind Google OAuth, all users are employees

Context

The data analyst platform supports data analysis (parquet + DuckDB). We want analysts to also interact with internal services (Purchase Orders, Invoicing, CRM) through Claude Code.

What the analyst needs:

1. API key in their local .env file
2. Skill file in .claude/rules/ teaching Claude Code how to use the API

What we already have:

• sync_data.sh that syncs files from server to analyst's machine
• .claude_rules/ sync for corporate memory (skills already flow through this)
• sudo install pattern for writing to user home dirs
• Dashboard with AJAX cards (Data Settings, Telegram)

Trust model: Employee laptops are trusted (corporate-managed, encrypted disks). If a laptop is compromised, we have bigger problems than API keys.

Architecture Overview

User clicks "Connect" on your-instance.example.com
    |
    v
Webapp calls service's token-exchange endpoint (shared secret)
    |
    v
API key stored in /data/service-connectors/connections.json (plaintext, 660)
    |
    v
Webapp writes /home/{user}/.service_env (sudo install, mode 600)
Webapp writes /home/{user}/.claude_rules/sc_{service}.md (skill file)
    |
    v
Analyst runs sync_data.sh (existing)
    |
    v
.service_env -> merged into local .env
sc_*.md -> already synced via existing .claude_rules/ sync

That's it. No encryption layer, no file locking, no audit log, no rotation timer.

Implementation

1. Service Registry

File: docs/setup/service_connectors.json

[
  {
    "id": "purchase_orders",
    "name": "Purchase Order System",
    "description": "Create and query purchase orders",
    "token_exchange_url": "https://po.internal.example.com/api/internal/token-exchange",
    "token_revoke_url": "https://po.internal.example.com/api/internal/token-revoke",
    "env_var_name": "PO_API_KEY",
    "skill_file": "sc_purchase_orders.md",
    "enabled": true
  }
]

Deployed to /data/docs/setup/ by deploy.sh (same as other config files).

2. Backend Service

File: webapp/service_connector_service.py

Follows webapp/sync_settings_service.py pattern exactly.

"""
Service connector - manages API key provisioning for internal services.

Reads service registry from /data/docs/setup/service_connectors.json.
Stores user connections in /data/service-connectors/connections.json.
Writes .service_env and skill files to user home dirs via sudo install.
"""

import json
import logging
import os
import subprocess
import tempfile
from datetime import datetime
from pathlib import Path
from typing import Any

import httpx

logger = logging.getLogger(__name__)

CONNECTORS_DIR = Path(os.environ.get("CONNECTORS_DIR", "/data/service-connectors"))
CONNECTIONS_FILE = CONNECTORS_DIR / "connections.json"
REGISTRY_FILE = Path(os.environ.get("REGISTRY_FILE", "/data/docs/setup/service_connectors.json"))
SKILLS_DIR = Path(os.environ.get("SC_SKILLS_DIR", "/data/docs/service_connector_skills"))

# Username mapping (reuse existing pattern)
WEBAPP_TO_SERVER_USERNAME = {
    # Add overrides here if webapp username != server username
    # "jane.smith": "jane",
}


def get_available_services() -> list[dict]:
    """Load service registry."""
    if not REGISTRY_FILE.exists():
        return []
    with open(REGISTRY_FILE) as f:
        services = json.load(f)
    return [s for s in services if s.get("enabled", True)]


def get_user_connections(username: str) -> dict:
    """Get user's active connections (without API keys)."""
    connections = _load_connections()
    user_conns = connections.get(username, {})
    # Strip API keys from response
    safe = {}
    for service_id, conn in user_conns.items():
        safe[service_id] = {
            "connected": conn.get("connected", False),
            "connected_at": conn.get("connected_at"),
            "expires_at": conn.get("expires_at"),
        }
    return safe


def connect_service(username: str, service_id: str, user_email: str) -> tuple[bool, str]:
    """Exchange token with service, store key, install to user home."""
    service = _get_service_config(service_id)
    if not service:
        return False, "Unknown service"

    # Get shared secret for this service
    secret_env = f"SC_SECRET_{service_id.upper()}"
    shared_secret = os.environ.get(secret_env)
    if not shared_secret:
        logger.error(f"Missing {secret_env} environment variable")
        return False, "Service not configured"

    # Token exchange
    try:
        resp = httpx.post(
            service["token_exchange_url"],
            headers={"Authorization": f"Bearer {shared_secret}"},
            json={"user_email": user_email, "ttl_days": 365},
            timeout=30,
        )
        resp.raise_for_status()
        token_data = resp.json()
    except Exception as e:
        logger.error(f"Token exchange failed for {service_id}: {e}")
        return False, f"Token exchange failed: {e}"

    # Store in connections.json
    connections = _load_connections()
    connections.setdefault(username, {})[service_id] = {
        "connected": True,
        "api_key": token_data["api_key"],
        "token_id": token_data.get("token_id"),
        "connected_at": datetime.utcnow().isoformat() + "Z",
        "expires_at": token_data.get("expires_at"),
    }
    _save_connections(connections)

    # Write .service_env and skills to user home
    server_username = _get_server_username(username)
    _regenerate_user_env(server_username, connections.get(username, {}))
    _install_service_skills(server_username, connections.get(username, {}))

    return True, "Connected successfully"


def disconnect_service(username: str, service_id: str) -> tuple[bool, str]:
    """Revoke token and remove credentials."""
    connections = _load_connections()
    conn = connections.get(username, {}).get(service_id)
    if not conn:
        return False, "Not connected"

    # Try to revoke remotely (best-effort)
    service = _get_service_config(service_id)
    token_id = conn.get("token_id")
    if service and token_id:
        try:
            secret_env = f"SC_SECRET_{service_id.upper()}"
            shared_secret = os.environ.get(secret_env, "")
            httpx.post(
                service["token_revoke_url"],
                headers={"Authorization": f"Bearer {shared_secret}"},
                json={"token_id": token_id},
                timeout=30,
            )
        except Exception as e:
            logger.warning(f"Remote revoke failed for {service_id}/{token_id}: {e}")

    # Always clean up locally
    connections.get(username, {}).pop(service_id, None)
    if username in connections and not connections[username]:
        del connections[username]
    _save_connections(connections)

    # Regenerate user files
    server_username = _get_server_username(username)
    _regenerate_user_env(server_username, connections.get(username, {}))
    _install_service_skills(server_username, connections.get(username, {}))

    return True, "Disconnected"


# --- Internal helpers ---

def _get_service_config(service_id: str) -> dict | None:
    """Find service in registry by ID."""
    for s in get_available_services():
        if s["id"] == service_id:
            return s
    return None


def _get_server_username(webapp_username: str) -> str:
    """Map webapp username to server Linux username."""
    return WEBAPP_TO_SERVER_USERNAME.get(webapp_username, webapp_username)


def _load_connections() -> dict:
    """Load connections.json."""
    if not CONNECTIONS_FILE.exists():
        return {}
    with open(CONNECTIONS_FILE) as f:
        return json.load(f)


def _save_connections(data: dict) -> None:
    """Atomic write to connections.json (same pattern as sync_settings_service)."""
    fd, temp_path = tempfile.mkstemp(dir=str(CONNECTORS_DIR), suffix=".json")
    try:
        os.write(fd, json.dumps(data, indent=2).encode())
        os.close(fd)
        os.chmod(temp_path, 0o660)
        os.replace(temp_path, str(CONNECTIONS_FILE))
    except Exception:
        os.close(fd) if not os.get_inheritable(fd) else None
        os.unlink(temp_path)
        raise


def _regenerate_user_env(server_username: str, user_connections: dict) -> None:
    """Write .service_env to user home via sudo install."""
    # Build env file content
    lines = []
    for service_id, conn in user_connections.items():
        if not conn.get("connected"):
            continue
        service = _get_service_config(service_id)
        if service:
            lines.append(f"{service['env_var_name']}={conn['api_key']}")

    # Write to temp file, then sudo install
    fd, temp_path = tempfile.mkstemp(suffix=".env")
    try:
        os.write(fd, "\n".join(lines).encode() if lines else b"")
        os.close(fd)

        target = f"/home/{server_username}/.service_env"
        if lines:
            subprocess.run(
                ["sudo", "/usr/bin/install", "-o", server_username, "-g", server_username,
                 "-m", "600", temp_path, target],
                check=True, capture_output=True,
            )
        else:
            # No connections - remove .service_env if it exists
            subprocess.run(
                ["sudo", "rm", "-f", target],
                check=True, capture_output=True,
            )
    finally:
        os.unlink(temp_path)


def _install_service_skills(server_username: str, user_connections: dict) -> None:
    """Install sc_*.md skill files to user's .claude_rules/ via sudo helper."""
    connected_services = [
        sid for sid, conn in user_connections.items() if conn.get("connected")
    ]

    # Copy relevant skill files to temp dir
    temp_dir = tempfile.mkdtemp()
    try:
        for service_id in connected_services:
            service = _get_service_config(service_id)
            if service and service.get("skill_file"):
                src = SKILLS_DIR / service["skill_file"]
                if src.exists():
                    dest = Path(temp_dir) / service["skill_file"]
                    dest.write_bytes(src.read_bytes())

        subprocess.run(
            ["sudo", "/usr/local/bin/install-service-env",
             server_username, temp_dir],
            check=True, capture_output=True,
        )
    finally:
        import shutil
        shutil.rmtree(temp_dir, ignore_errors=True)

3. API Routes

Add to webapp/app.py in register_routes():

from webapp import service_connector_service

@app.route("/api/service-connectors")
@login_required
def api_service_connectors():
    username = get_username_from_email(session["user"]["email"])
    services = service_connector_service.get_available_services()
    connections = service_connector_service.get_user_connections(username)
    return jsonify({"services": services, "connections": connections})

@app.route("/api/service-connectors/connect", methods=["POST"])
@login_required
def api_service_connect():
    username = get_username_from_email(session["user"]["email"])
    email = session["user"]["email"]
    service_id = request.json.get("service_id")
    ok, msg = service_connector_service.connect_service(username, service_id, email)
    return jsonify({"success": ok, "message": msg})

@app.route("/api/service-connectors/disconnect", methods=["POST"])
@login_required
def api_service_disconnect():
    username = get_username_from_email(session["user"]["email"])
    service_id = request.json.get("service_id")
    ok, msg = service_connector_service.disconnect_service(username, service_id)
    return jsonify({"success": ok, "message": msg})

4. Sudo Helper

File: server/bin/install-service-env (copy of install-user-rules, modified for sc_* prefix)

#!/bin/bash
# Install service connector skill files to user's .claude_rules/.
# Called by webapp (www-data) via sudo.
#
# Usage: sudo install-service-env USERNAME SKILLS_SOURCE_DIR

set -euo pipefail

if [[ $EUID -ne 0 ]]; then
    echo "Must be run as root (via sudo)" >&2
    exit 1
fi

if [[ $# -lt 2 ]]; then
    echo "Usage: sudo install-service-env USERNAME SKILLS_SOURCE_DIR" >&2
    exit 1
fi

USERNAME="$1"
SOURCE_DIR="$2"

if ! id "$USERNAME" &>/dev/null; then
    echo "User '$USERNAME' does not exist" >&2
    exit 1
fi

if [[ ! -d "$SOURCE_DIR" ]]; then
    echo "Source directory '$SOURCE_DIR' does not exist" >&2
    exit 1
fi

USER_HOME=$(eval echo "~${USERNAME}")
RULES_DIR="${USER_HOME}/.claude_rules"

mkdir -p "$RULES_DIR"
chown "${USERNAME}:${USERNAME}" "$RULES_DIR"
chmod 700 "$RULES_DIR"

# Remove old service connector skills only (sc_*.md), preserve km_*.md
rm -f "${RULES_DIR}"/sc_*.md

# Install new skill files
COUNT=0
for src_file in "${SOURCE_DIR}"/*.md; do
    if [[ -f "$src_file" ]]; then
        /usr/bin/install -o "$USERNAME" -g "$USERNAME" -m 600 "$src_file" "$RULES_DIR/"
        COUNT=$((COUNT + 1))
    fi
done

echo "Installed ${COUNT} service skills for ${USERNAME} in ${RULES_DIR}"

5. Dashboard UI Card

Add to webapp/templates/dashboard.html - new card in existing grid, same AJAX pattern as Data Settings toggles:

• Grid of service cards (name + description from registry)
• Green "Connected" badge or grey "Not connected"
• Connect / Disconnect button
• Shows expires_at if connected

6. Sync Extension

Add to scripts/sync_data.sh after existing corporate memory sync block:

# --- Sync service connector credentials ---
if scp -q data-analyst:~/.service_env /tmp/.service_env_$$ 2>/dev/null; then
    # Remove old service connector block
    if [ -f ./.env ]; then
        awk '
            /^# --- SERVICE CONNECTOR START ---$/ { skip=1; next }
            /^# --- SERVICE CONNECTOR END ---$/ { skip=0; next }
            !skip { print }
        ' ./.env > ./.env.tmp && mv ./.env.tmp ./.env
    fi
    # Append new block
    {
        echo "# --- SERVICE CONNECTOR START ---"
        cat /tmp/.service_env_$$
        echo "# --- SERVICE CONNECTOR END ---"
    } >> ./.env
    rm -f /tmp/.service_env_$$
    echo "Service connector credentials synced"
else
    # No active connections - clean old block if present
    if [ -f ./.env ] && grep -q "^# --- SERVICE CONNECTOR START ---$" ./.env 2>/dev/null; then
        awk '
            /^# --- SERVICE CONNECTOR START ---$/ { skip=1; next }
            /^# --- SERVICE CONNECTOR END ---$/ { skip=0; next }
            !skip { print }
        ' ./.env > ./.env.tmp && mv ./.env.tmp ./.env
        echo "Service connector credentials removed"
    fi
fi

Skill files (sc_*.md) are already synced by the existing .claude_rules/ sync block.

7. Deploy Changes

Add to server/deploy.sh:

# Service connectors directory
mkdir -p /data/service-connectors
chown www-data:data-ops /data/service-connectors
chmod 2770 /data/service-connectors

# Deploy skill files
mkdir -p /data/docs/service_connector_skills
cp -r docs/service_connector_skills/* /data/docs/service_connector_skills/ 2>/dev/null || true

# Deploy service registry
cp docs/setup/service_connectors.json /data/docs/setup/ 2>/dev/null || true

# Install sudo helper
install -m 755 server/bin/install-service-env /usr/local/bin/

Add to server/sudoers-webapp:

www-data ALL=(ALL) NOPASSWD: /usr/local/bin/install-service-env

Add to .github/workflows/deploy.yml env block:

SC_SECRET_PURCHASE_ORDERS: ${{ secrets.SC_SECRET_PURCHASE_ORDERS }}

Token Exchange Protocol

What each internal service needs to implement (simple Bearer + JSON):

POST /api/internal/token-exchange
Authorization: Bearer <shared_secret>
Content-Type: application/json
Body: {"user_email": "john@your-domain.com", "ttl_days": 365}
Response: {"status": "ok", "api_key": "...", "token_id": "...", "expires_at": "..."}

POST /api/internal/token-revoke
Authorization: Bearer <shared_secret>
Content-Type: application/json
Body: {"token_id": "tok_xyz789"}
Response: {"status": "ok"}

TTL is 365 days. If a key expires, user clicks Reconnect. No auto-rotation needed.

Files to Create

File	Purpose	Size estimate
webapp/service_connector_service.py	Connect, disconnect, env generation	~150 lines
docs/setup/service_connectors.json	Service registry	~20 lines
docs/service_connector_skills/sc_purchase_orders.md	PO API skill for Claude Code	~50 lines
server/bin/install-service-env	Sudo helper (copy of install-user-rules)	~30 lines
tests/test_service_connector_service.py	Unit tests	~100 lines

Files to Modify

File	Change	Size estimate
webapp/app.py	Add 3 API routes	~20 lines
webapp/templates/dashboard.html	Service connectors card	~60 lines
server/sudoers-webapp	Add install-service-env entry	1 line
server/deploy.sh	Create dirs, deploy skills, add env vars	~10 lines
scripts/sync_data.sh	.service_env merge block	~20 lines
.github/workflows/deploy.yml	Add SC_SECRET_* to env	~3 lines

Total new code: ~350 lines (vs ~800+ in the hybrid plan, ~1200+ in the Codex plan)

Security Model

Stage	Protection
Token exchange	HTTPS + per-service shared secret
Server storage (connections.json)	File permissions 660, dir 2770 (www-data:data-ops)
User home (.service_env)	Mode 600, sudo install
Transit	SCP over SSH
Client (.env)	Local filesystem, Claude Code denies Read(.env)
Trust model	Employee laptops trusted (corporate-managed, encrypted disks)

Key Patterns Reused

• Sudo install: sync_settings_service.py:_regenerate_user_config()
• Atomic JSON write: sync_settings_service.py:_write_json() (tempfile + os.replace)
• Username mapping: corporate_memory_service.py:_get_server_username()
• Sudo helper: server/bin/install-user-rules (same structure)
• Dashboard AJAX: Sync settings toggles in dashboard.html

Verification

1. pytest tests/test_service_connector_service.py
2. Deploy, click Connect on PO, verify .service_env in /home/{user}/
3. Run sync_data.sh, verify .env contains PO_API_KEY
4. Verify .claude/rules/sc_purchase_orders.md exists
5. In Claude Code: python -c "from dotenv import load_dotenv; load_dotenv(); import os; print(os.environ.get('PO_API_KEY', 'NOT SET'))"
6. Click Disconnect, sync, verify key removed

What We Might Add Later (only if needed)

Feature	When to add
Encryption of connections.json	If we get a compliance requirement
Auto-rotation	If services start issuing short-lived tokens
Audit log	If we need forensics capability
File locking	If we ever have concurrent connect/disconnect issues
SSH runtime injection	If laptop trust model changes