p24-infra Docs

plan-2427-rotate-secret-system

32 min read

Plan: Per-Service Autonomous Secret Rotation System

Issue: #2427 Type: Code-change-design Status: Draft — iteration 2 Last updated: 2026-07-01 Author: Claude Sonnet 4.6 (AI-Dev) via plan branch

0. Why This, Why Now

Problem: The current scripts/rotate-credentials.py (1668 lines) is a sequential monolith. One SSH connection failure or one provider API error stops all subsequent rotations in the same run. The script rotates 30+ credentials in a single GH Actions job — if it fails at credential #15, credentials #16–30 stay stale until the next Monday. There is no per-service failure isolation, no structured audit trail written by the scripts themselves, and no way to trigger a single-service rotation via the worker queue (which handles everything else autonomously).

Why this approach: Decomposing into per-service bash scripts in a worker queue achieves: (a) fault isolation — one service failure cannot block another; (b) per-service scheduling — high-frequency internal keys can rotate daily while low-frequency vendor tokens rotate monthly; (c) queue integration — the same dispatcher that handles code issues and PR reviews can handle credential rotation with priority=1; (d) no Claude agent spawn — rotation is deterministic bash, not LLM-driven, so it uses ~8 MB RAM instead of ~1 GB.

Why now: The ATRAX_AUTH_STRING rotation (issue #1569) and the OVH Valkey rotation (#2415) both required multi-step manual intervention that the current system could not automate. The upcoming quarterly rotation window (W3/W4 2026) is the natural point to establish the new system before the next rotation cycle begins.

1. Current State

What exists today

Central monolithic rotator (scripts/rotate-credentials.py, 1668 lines)

All rotation logic lives in a single Python script executed weekly by the credential-rotation.yml GH Actions workflow (Monday 06:00 UTC). The script:

  • Opens paramiko SSH connections to vps-i1, vps-h1, and bms-4
  • Rotates up to 30+ credentials, each with a bespoke rotate_<service>() function
  • Calls sops_update_key() after each rotation to update the SOPS encrypted file
  • Commits the SOPS change to the repo and pushes via git push
  • Updates dev_r_services.last_rotated and next_due in Supabase
  • Creates or comments on GH issues for manual (Tier 3) credentials

Distribution chain (current)

  1. Rotate at provider (API call or SSH command)
  2. sed -i on the live .env file on the affected server via paramiko
  3. docker compose up -d --no-deps <service> via paramiko
  4. gh secret set via subprocess
  5. sops_update_key(): decrypt to memory, modify, re-encrypt in-place, commit, push

OVH Redis rotator (scripts/ovh_redis_rotate.py)

A standalone script for the OVH private database credential. It writes the new password to /tmp/new_redis_pass on the remote server (chmod 600) and never prints it. Consumer update is done manually following the playbook.

Worker queue (dev_r_worker_queue in Supabase)

Current weight classes:

  • light — triage, plan review, continue-issue jobs; ~64 MB RAM; no Claude agent spawn for classification work
  • heavy — full dev-issue implementation; spawns a Claude Code agent; ~1–2 GB RAM
  • OOM-killed light jobs are re-dispatched as heavy automatically

Current job types: dev-issue, continue-issue, review-pr, review-plan

What is manual today

  • All Tier 3 credentials (Mezmo dashboard, Sentry UI token, DISCORD_BOT_TOKEN, n8n MCP token, WAHA_CONTROL_TOKEN)
  • OVH Valkey consumer update (bms-1 Pinbox24 containers) — Part B requires multi-hop SCP workflow
  • Distribution to .env.local on developer workstation
  • Vercel env updates when VERCEL_TOKEN is unavailable in the workflow
  • n8n credential vault updates — n8n community edition has no programmatic API for updating stored credential values

Key gaps in the current design

  1. Single point of failure — one broken SSH connection or one failing rotator blocks all subsequent rotations in the same run
  2. Monolithic coupling — all rotation logic in one file; per-service iteration is slow and risky
  3. No per-service scheduling — all rotations fire on the same Monday 06:00 cron regardless of optimal cadence per service
  4. No structured audit trail from scripts — docs/secrets-rotation-log.md is maintained manually; Supabase dev_r_rotation_log is populated only by bms-4 Claude workers
  5. SOPS age key lives only in GH Actions — the AGE_KEY_GHA secret is available only to GH Actions runners, not VPS servers; this is correct but means VPS-side rotation scripts cannot update SOPS directly
  6. No per-service verification step — no standardised confirmation that the new credential is accepted by the service before the rotation is marked done

2. Target Architecture

The design decomposes rotation into four independent components that compose through a standard interface.

┌─────────────────────────────────────────────────────────────────────────────┐
│  Trigger layer                                                               │
│  ┌──────────────────┐  ┌─────────────────────┐  ┌────────────────────────┐ │
│  │ rotate-schedule  │  │ dispatch-to-queue    │  │  Manual /role-secret-  │ │
│  │ .yml (per-service│  │ (infra-task-request  │  │  manager session       │ │
│  │  cron triggers)  │  │  label)              │  │                        │ │
│  └────────┬─────────┘  └──────────┬──────────┘  └───────────┬────────────┘ │
└───────────┼──────────────────────┼─────────────────────────┼──────────────┘
            │                      │                          │
            └──────────────────────┴──────────────────────────┘
                                   │ INSERT dev_r_worker_queue row
                                   ▼
┌───────────────────────────────────────────────────────────────────────────┐
│  Component 1: Worker Queue (dev_r_worker_queue)                            │
│  job_type = "rotate-secret", weight = "super-light"                        │
│  metadata = { "service": "WAHA_API_KEY", "sops_file": "vps-h1" }          │
└────────────────────────────────────┬──────────────────────────────────────┘
                                     │ Dispatcher claims row on bms-4
                                     ▼
┌───────────────────────────────────────────────────────────────────────────┐
│  Component 2: Per-service rotation scripts                                 │
│  /opt/p24-infra/scripts/rotate/<credential-name>.sh                        │
│  - Runs on bms-4 (SSH-executes on target server when needed)              │
│  - Generates new credential via openssl/secrets module                     │
│  - Applies to service (sed + docker restart, API call, etc.)              │
│  - Verifies the new credential works                                       │
│  - Emits [ROTATE] structured log lines to stdout (no secret values)       │
│  - Age-encrypts new value into a payload file for Component 3             │
│  - Calls gh workflow run sops-sync-receiver.yml with base64 payload       │
└────────────────────────────────────┬──────────────────────────────────────┘
                                     │ gh workflow run (base64 ciphertext only)
                                     ▼
┌───────────────────────────────────────────────────────────────────────────┐
│  Component 3: SOPS Sync Authority                                          │
│  GH Actions workflow "sops-sync-receiver.yml"                              │
│  - Triggered by: rotation script via `gh workflow run`                     │
│  - Receives: age-encrypted payload (base64) + log_id as workflow inputs    │
│  - Decrypts with GH Secret AGE_KEY_SOPS_SYNC_RECEIVER                    │
│  - Runs sops_update_key() on the correct SOPS file                        │
│  - Commits + pushes to dev → triggers secrets-sync.yml                    │
│  - PATCH dev_r_rotation_log row with SOPS commit SHA + gh_secret status   │
└────────────────────────────────────┬──────────────────────────────────────┘
                                     │ secrets-sync.yml fires on merge
                                     ▼
┌───────────────────────────────────────────────────────────────────────────┐
│  Component 4: Audit Trail                                                  │
│  dev_r_rotation_log (Supabase) + docs/secrets-rotation-log.md (git)       │
│  - Rotation script opens a "pending" row at start (via Supabase REST)    │
│  - Closes it with "completed" + sops_commit + verify_result               │
│  - Grafana alert fires if any row stays pending > 30 min                  │
│  - Monthly summary appended to docs/secrets-rotation-log.md by bot       │
└───────────────────────────────────────────────────────────────────────────┘

Key design decisions

  • No Claude agent spawn for rotation — rotation scripts are purpose-built bash, not Claude Code sessions. This cuts RAM from ~1 GB to ~8 MB and eliminates the “Claude re-derives a broken approach” risk.
  • SOPS authority stays in GH Actions — the age key never leaves GHA secrets. Rotation scripts on VPS servers never hold the age private key. They produce an age-encrypted payload that only the GHA receiver can decrypt.
  • SSH only — the dispatcher on bms-4 SSH-executes rotation scripts. No new open ports, no HTTP endpoints that receive or return credential values.
  • Idempotent scripts — each script can be re-run safely if interrupted mid-way. Generate step checks whether the service already uses the pending new value (by testing it first).
  • Separation of concerns — each service script owns: generation, application, verification, and payload encoding. The receiver owns: SOPS update, GH Secret update, audit close.

3. Script Template

The canonical bash structure for any per-service rotation script. No secret values appear in stdout or logs; all values travel through age-encrypted files or closed file descriptors.

#!/usr/bin/env bash
# rotate/CREDENTIAL_NAME.sh — autonomous rotation for <SERVICE>
#
# Called by: queue dispatcher on bms-4 (via SSH) or directly by a secret-manager session
# Called as: bash rotate/CREDENTIAL_NAME.sh [--dry-run]
#
# Outputs (never to stdout):
#   /tmp/rotate-CREDENTIAL_NAME-payload.age — age-encrypted JSON payload for SOPS authority
#   /tmp/rotate-CREDENTIAL_NAME-verify.txt  — verification result (non-sensitive)
#
# [ROTATE] log lines: written to stdout, consumed by the dispatcher for audit trail.
# Format: [ROTATE] <timestamp> <CREDENTIAL_NAME> <step> <status> [<non-sensitive detail>]
# Example: [ROTATE] 2026-07-01T06:12:45Z WAHA_API_KEY apply ok container_id=abc123
# NEVER include credential values in [ROTATE] lines.
#
# Exit codes:
#   0  — rotation complete, payload written, sops-sync-receiver triggered
#   1  — generation failed (no change applied, safe to retry)
#   2  — apply failed (new credential generated but not applied — retry possible)
#   3  — verification failed (applied but unconfirmed — human review required)
#   4  — payload encrypt or dispatch failed (applied+verified but SOPS update blocked)
 
set -euo pipefail
DRY_RUN="${1:-}"
 
# ── Configuration ─────────────────────────────────────────────────────────────
CREDENTIAL_NAME="WAHA_API_KEY"
SOPS_FILE="vps-h1"
GH_SECRET_NAME="WAHA_API_KEY"
GH_REPO="radieu/p24-infra"
TARGET_SERVER="72.60.32.61"
TARGET_ENV_PATH="/root/.env"
TARGET_SERVICE_RESTART="docker compose -f /root/docker-compose.yml up -d --no-deps waha"
VERIFY_ENDPOINT="https://waha2.vps-h1.infra.zintegrowana.online/api/health"
VERIFY_HEADER_NAME="X-Api-Key"
SUPABASE_URL="${SUPABASE_URL:-}"
SUPABASE_SERVICE_KEY="${SUPABASE_SERVICE_KEY:-}"
# Public key of the GHA sops-sync-receiver — safe to hardcode; not a private key
SOPS_AGE_RECIPIENT="age1p24infraghareceiverXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
PAYLOAD_FILE="/tmp/rotate-${CREDENTIAL_NAME}-payload.age"
VERIFY_FILE="/tmp/rotate-${CREDENTIAL_NAME}-verify.txt"
# ─────────────────────────────────────────────────────────────────────────────
 
log() {
    echo "[ROTATE] $(date -u +%Y-%m-%dT%H:%M:%SZ) ${CREDENTIAL_NAME} $*"
}
 
fail() {
    local code="$1"; shift
    log "FAILED exit=${code} $*"
    exit "$code"
}
 
patch_log() {
    local log_id="$1"; shift
    local body="$1"
    if [[ -n "$log_id" && -n "$SUPABASE_URL" && -n "$SUPABASE_SERVICE_KEY" ]]; then
        curl -sf -X PATCH "${SUPABASE_URL}/rest/v1/dev_r_rotation_log?id=eq.${log_id}" \
            -H "apikey: ${SUPABASE_SERVICE_KEY}" \
            -H "Authorization: Bearer ${SUPABASE_SERVICE_KEY}" \
            -H "Content-Type: application/json" \
            -d "$body" 2>/dev/null || true
    fi
}
 
# ── Step 0: Open audit log row ──────────────────────────────────────────────
LOG_ID=""
if [[ -n "$SUPABASE_URL" && -n "$SUPABASE_SERVICE_KEY" ]]; then
    LOG_ID=$(curl -sf -X POST "${SUPABASE_URL}/rest/v1/dev_r_rotation_log" \
        -H "apikey: ${SUPABASE_SERVICE_KEY}" \
        -H "Authorization: Bearer ${SUPABASE_SERVICE_KEY}" \
        -H "Content-Type: application/json" \
        -H "Prefer: return=representation" \
        -d "{\"secret_name\":\"${CREDENTIAL_NAME}\",\"reason\":\"scheduled\",\"rotator\":\"rotate-script-bms4\",\"status\":\"pending\"}" \
        2>/dev/null | python3 -c "import sys,json; rows=json.load(sys.stdin); print(rows[0]['id'] if rows else '')" \
        2>/dev/null || true)
fi
log "start log_id=${LOG_ID:-none}"
 
# ── Step 1: Generate new credential ─────────────────────────────────────────
log "generate start"
# openssl rand -hex 24 = 48 hex chars = 192 bits. No special chars that break sed.
NEW_KEY=$(openssl rand -hex 24)
if [[ -z "$NEW_KEY" ]]; then
    fail 1 "openssl rand produced empty string"
fi
log "generate ok length=${#NEW_KEY}"
 
if [[ "$DRY_RUN" == "--dry-run" ]]; then
    log "DRY_RUN skip apply and verify"
    NEW_KEY=""
    exit 0
fi
 
# ── Step 2: Apply to service ─────────────────────────────────────────────────
log "apply start target=${TARGET_SERVER}"
# NEW_KEY is hex-only so no sed delimiter conflicts with | or &
ssh -o BatchMode=yes -o StrictHostKeyChecking=no -o ConnectTimeout=10 \
    root@"${TARGET_SERVER}" \
    "sed -i \"s|^${CREDENTIAL_NAME}=.*|${CREDENTIAL_NAME}=${NEW_KEY}|\" ${TARGET_ENV_PATH}" \
    2>/dev/null \
    || fail 2 "sed replace failed on ${TARGET_SERVER}:${TARGET_ENV_PATH}"
 
ssh -o BatchMode=yes -o StrictHostKeyChecking=no -o ConnectTimeout=10 \
    root@"${TARGET_SERVER}" \
    "${TARGET_SERVICE_RESTART}" 2>/dev/null \
    || fail 2 "service restart failed on ${TARGET_SERVER}"
log "apply ok"
 
# Give the container time to start before verifying
sleep 5
 
# ── Step 3: Verify new credential works ──────────────────────────────────────
log "verify start endpoint=${VERIFY_ENDPOINT}"
HTTP_CODE=$(ssh -o BatchMode=yes -o StrictHostKeyChecking=no root@"${TARGET_SERVER}" \
    "curl -sf -o /dev/null -w '%{http_code}' \
     -H '${VERIFY_HEADER_NAME}: ${NEW_KEY}' \
     '${VERIFY_ENDPOINT}' 2>/dev/null" \
    2>/dev/null || echo "000")
 
if [[ "$HTTP_CODE" != "200" ]]; then
    echo "http_code=${HTTP_CODE}" > "${VERIFY_FILE}"
    patch_log "$LOG_ID" "{\"status\":\"failed\",\"error_message\":\"verify http_code=${HTTP_CODE}\"}"
    fail 3 "verify http_code=${HTTP_CODE} — new key rejected by service"
fi
echo "http_code=200 timestamp=$(date -u +%Y-%m-%dT%H:%M:%SZ)" > "${VERIFY_FILE}"
log "verify ok http_code=${HTTP_CODE}"
 
# ── Step 4: Age-encrypt payload for SOPS authority ───────────────────────────
log "encrypt-payload start"
# Construct JSON in a temp variable, pipe immediately to age, clear variable.
# The payload JSON is never written to disk in plaintext.
VERIFY_RESULT=$(cat "${VERIFY_FILE}")
PAYLOAD_JSON="{\"credential_name\":\"${CREDENTIAL_NAME}\",\"sops_file\":\"${SOPS_FILE}\",\"new_value\":\"${NEW_KEY}\",\"gh_secret_name\":\"${GH_SECRET_NAME}\",\"gh_repo\":\"${GH_REPO}\",\"verify_result\":\"${VERIFY_RESULT}\",\"log_id\":\"${LOG_ID}\"}"
 
echo "$PAYLOAD_JSON" | age --recipient "${SOPS_AGE_RECIPIENT}" \
    --output "${PAYLOAD_FILE}" 2>/dev/null \
    || fail 4 "age encrypt failed"
 
# Wipe plaintext from memory immediately
PAYLOAD_JSON=""
NEW_KEY=""
log "encrypt-payload ok file=${PAYLOAD_FILE}"
 
# ── Step 5: Dispatch to SOPS sync receiver ───────────────────────────────────
log "dispatch start workflow=sops-sync-receiver.yml"
PAYLOAD_B64=$(base64 -w0 < "${PAYLOAD_FILE}")
 
gh workflow run sops-sync-receiver.yml \
    --repo "${GH_REPO}" \
    --field "payload_b64=${PAYLOAD_B64}" \
    --field "log_id=${LOG_ID}" \
    2>/dev/null \
    || fail 4 "gh workflow run sops-sync-receiver.yml failed"
 
PAYLOAD_B64=""
log "dispatch ok"
log "complete rotation dispatched to sops-sync-receiver"
rm -f "${PAYLOAD_FILE}" "${VERIFY_FILE}"

Script contract rules

  1. All credential values flow through variables only — never as function arguments visible in ps aux, command substitution results printed to terminal, or log messages.
  2. [ROTATE] lines are the only stdout output. They are consumed by the dispatcher and written to dev_r_rotation_log.log_lines. They contain only: credential name, step name, status, and non-sensitive metadata (HTTP code, elapsed time, host). Never a credential value.
  3. Exit code determines dispatcher behavior: 0 = success, 1 = safe retry, 2 = retry-after-human, 3 = human-action issue, 4 = human-action issue (SOPS drift state).
  4. Scripts are stored in scripts/rotate/ and named <credential_name_lowercase>.sh.
  5. Each script has a --dry-run flag that generates a credential but does not apply it, does not write a payload file, and exits 0.
  6. Multi-key credentials (e.g., Wasabi access key + secret key pair) are handled by a single script that packages both in the age payload as new_values: { KEY1: val1, KEY2: val2 }.

4. SOPS Sync Protocol

Problem

Per-service rotation scripts run on bms-4 or target servers. They must update secrets/<file>.env.sops in the git repo. The age private key (AGE_KEY_GHA) is a GH Secret available only to GH Actions runners, never to VPS servers. VPS servers cannot push to the GH Actions runner.

Solution: Age-encrypted payload + GHA receiver workflow

bms-4 rotation script
    │
    │  age-encrypted JSON (ciphertext; safe to transmit over any channel)
    │  { credential_name, sops_file, new_value, gh_secret_name, verify_result, log_id }
    │
    │  gh workflow run sops-sync-receiver.yml \
    │      --field payload_b64=<base64-of-ciphertext> \
    │      --field log_id=<uuid>
    │
    ▼ GH Actions runner (has AGE_KEY_SOPS_SYNC_RECEIVER private key)
    │
    │ 1. base64-decode payload_b64
    │ 2. age --decrypt (using AGE_KEY_SOPS_SYNC_RECEIVER) → plaintext JSON
    │ 3. Validate JSON structure (schema check; reject unexpected keys)
    │ 4. sops_update_key(sops_file, credential_name, new_value)
    │    - decrypt SOPS file with AGE_KEY_GHA
    │    - replace or insert the credential line
    │    - re-encrypt in-place (LF-only; no BOM)
    │    - canary decrypt (exit nonzero = abort + revert)
    │    - git commit: "chore: auto-rotate CREDENTIAL_NAME [skip ci]"
    │    - git push to dev (or bot branch with auto-merge label)
    │ 5. gh secret set GH_SECRET_NAME --body new_value --repo GH_REPO
    │ 6. PATCH dev_r_rotation_log?id=eq.LOG_ID
    │    { status: "completed", sops_commit: SHA, gh_secret: "updated",
    │      completed_at: now() }
    │
    ▼ secrets-sync.yml fires on push to dev
    │
    ▼ Deploys updated .env to vps-i1 (/opt/p24-infra/monitoring/.env)
      and bms-4 (/opt/p24-infra/bms-4/.env)

Age recipient setup

Two age key pairs are used — one existing, one new:

  1. Existing SOPS key set — secrets/*.env.sops encrypted for: developer workstation, AGE_KEY_GHA (GH Actions), and vps-i1 claude-runner (PR #1470). The GHA receiver uses AGE_KEY_GHA for SOPS operations.

  2. New receiver key pair — Used only for rotation payload transport. Public key is hardcoded in every rotation script (it is a recipient, not a secret). Private key is stored in GH Secrets as AGE_KEY_SOPS_SYNC_RECEIVER. This key is NOT added as a SOPS file recipient — it only decrypts the inter-component JSON payload.

This separation preserves least privilege: a compromised rotation script on a VPS server can only produce encrypted payloads destined for the GHA receiver. It cannot decrypt existing SOPS files. The receiver validates payload structure before writing any values to SOPS.

Payload format

{
  "schema_version": "1",
  "credential_name": "WAHA_API_KEY",
  "sops_file": "vps-h1",
  "new_value": "<credential value — encrypted in transit>",
  "gh_secret_name": "WAHA_API_KEY",
  "gh_repo": "radieu/p24-infra",
  "verify_result": "http_code=200 timestamp=2026-07-01T06:12:45Z",
  "log_id": "uuid-of-supabase-rotation-log-row"
}

For multi-key credentials (Wasabi IAM pair, etc.):

{
  "schema_version": "1",
  "credential_name": "P24_INFRA_WASABI_KEYS",
  "sops_file": "monitoring",
  "new_values": {
    "P24_INFRA_WASABI_ACCESS_KEY": "<access-key>",
    "P24_INFRA_WASABI_SECRET_KEY": "<secret-key>"
  },
  "gh_secrets": {
    "P24_INFRA_WASABI_ACCESS_KEY": "<access-key>",
    "P24_INFRA_WASABI_SECRET_KEY": "<secret-key>"
  },
  "gh_repo": "radieu/p24-infra",
  "verify_result": "iam_list_keys=ok",
  "log_id": "uuid"
}

GHA receiver workflow (.github/workflows/sops-sync-receiver.yml)

name: SOPS Sync Receiver
 
on:
  workflow_dispatch:
    inputs:
      payload_b64:
        description: "Age-encrypted rotation payload (base64)"
        required: true
        type: string
      log_id:
        description: "Supabase dev_r_rotation_log row UUID"
        required: false
        type: string
 
jobs:
  sync:
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683  # v4.2.2
        with:
          token: ${{ secrets.GH_PAT }}
 
      - name: Install sops and age
        run: |
          curl -Lo /usr/local/bin/sops \
            https://github.com/getsops/sops/releases/download/v3.9.1/sops-v3.9.1.linux.amd64
          chmod +x /usr/local/bin/sops
          curl -Lo /tmp/age.tar.gz \
            https://github.com/FiloSottile/age/releases/download/v1.2.0/age-v1.2.0-linux-amd64.tar.gz
          tar -xz -C /usr/local/bin --strip-components=1 \
            -f /tmp/age.tar.gz age/age age/age-keygen
 
      - name: Decrypt payload and update SOPS
        env:
          AGE_KEY_SOPS_SYNC_RECEIVER: ${{ secrets.AGE_KEY_SOPS_SYNC_RECEIVER }}
          SOPS_AGE_KEY: ${{ secrets.AGE_KEY_GHA }}
          SUPABASE_URL: ${{ secrets.SUPABASE_URL }}
          SUPABASE_SERVICE_KEY: ${{ secrets.SUPABASE_SERVICE_KEY }}
          GH_TOKEN: ${{ secrets.GH_PAT }}
          PAYLOAD_B64: ${{ inputs.payload_b64 }}
          LOG_ID: ${{ inputs.log_id }}
        run: python3 scripts/sops-sync-receiver.py

Git commit flow in the receiver

branch: dev (direct push using GH_PAT with repo write scope)
commit: "chore: auto-rotate CREDENTIAL_NAME in SOPS_FILE.env.sops [skip ci]"
author: credential-rotation-bot <noreply@github.com>

The [skip ci] annotation prevents dispatch-to-queue.yml from treating the commit as a developer issue. The secrets-sync.yml is configured to fire on push to dev even with [skip ci] commits (GH Actions schedule/push triggers ignore [skip ci] for push events unless explicitly filtered — needs validation).

5. Queue Extension

New weight class: super-light

Rotation scripts are pure bash + SSH + one gh workflow run call. They do NOT spawn a Claude Code agent. RAM footprint is the bash process (~4 MB) plus one SSH connection (~4 MB). Total: ~8 MB per concurrent rotation.

Changes to dev_r_worker_queue schema:

-- Migration: add super-light weight and rotate-secret job type
 
-- Step 1: Drop existing constraints
ALTER TABLE dev_r_worker_queue
  DROP CONSTRAINT IF EXISTS dev_r_worker_queue_weight_check;
ALTER TABLE dev_r_worker_queue
  DROP CONSTRAINT IF EXISTS dev_r_worker_queue_job_type_check;
 
-- Step 2: Re-add with new values
ALTER TABLE dev_r_worker_queue
  ADD CONSTRAINT dev_r_worker_queue_weight_check
  CHECK (weight IN ('super-light', 'light', 'heavy'));
 
ALTER TABLE dev_r_worker_queue
  ADD CONSTRAINT dev_r_worker_queue_job_type_check
  CHECK (job_type IN ('dev-issue', 'continue-issue', 'review-pr', 'review-plan',
                      'rotate-secret', 'infra-task'));
 
-- Step 3: Super-light capacity column on bms-4
ALTER TABLE dev_r_server_capacity
  ADD COLUMN IF NOT EXISTS max_super_light integer DEFAULT 8;
UPDATE dev_r_server_capacity
  SET max_super_light = 8 WHERE server_label = 'bms-4';

New job type: rotate-secret

Dispatcher changes (scripts/queue-dispatcher-loop.py on bms-4):

elif job['job_type'] == 'rotate-secret' and job['weight'] == 'super-light':
    service = job['metadata'].get('service')      # e.g. "WAHA_API_KEY"
    dry_run = job['metadata'].get('dry_run', False)
    script_name = service.lower().replace('-', '_') + '.sh'
    script_path = f"/opt/p24-infra/scripts/rotate/{script_name}"
 
    if not os.path.exists(script_path):
        fail_job(job['id'], f"No rotation script for {service} at {script_path}")
        continue
 
    dry_run_flag = "--dry-run" if dry_run else ""
    cmd = ["bash", script_path] + ([dry_run_flag] if dry_run_flag else [])
 
    # CRITICAL: all stdout from the script is collected but NOT forwarded to
    # journald or any other log sink that may persist credential values.
    # Only [ROTATE]-prefixed lines are safe to store.
    proc = subprocess.Popen(
        cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
        env={**os.environ, "SUPABASE_URL": SUPABASE_URL,
             "SUPABASE_SERVICE_KEY": SUPABASE_SERVICE_KEY}
    )
    rotate_lines = []
    for line in proc.stdout:
        if line.startswith('[ROTATE]'):
            rotate_lines.append(line.rstrip())
        # Non-[ROTATE] lines are silently discarded to prevent any accidental
        # credential value leakage into the dispatcher log or Supabase.
 
    proc.wait()
    stderr_output = proc.stderr.read()[:500] if proc.returncode != 0 else ""
 
    update_queue_row(
        job['id'],
        status='done' if proc.returncode == 0 else 'failed',
        error_message=(None if proc.returncode == 0
                       else f"exit {proc.returncode}: {stderr_output}"),
        log_lines='\n'.join(rotate_lines)
    )

Queue insert for rotate-secret jobs (via rotate-schedule.yml GHA workflow):

# Inserted into dev_r_worker_queue when a per-service cron fires:
{
    "github_issue_number": None,  # Open question: see Q1
    "repo": "radieu/p24-infra",
    "job_type": "rotate-secret",
    "weight": "super-light",
    "priority": 1,
    "metadata": {
        "service": "WAHA_API_KEY",
        "sops_file": "vps-h1",
        "dry_run": False
    },
    "status": "queued"
}

p24-meta-dispatcher CF Worker extension

The CF Worker at p24-meta-dispatcher.radieu.workers.dev handles dev-issue via POST /queue-issue. Add a new route for rotation jobs triggered from external project sessions:

POST /queue-rotation
Authorization: Bearer <QUEUE_API_KEY>
Content-Type: application/json

{
  "service": "WAHA_API_KEY",
  "reason": "scheduled|exposure|preventive",
  "priority": 1
}

Response 201: { "id": N, "status": "queued", "service": "WAHA_API_KEY" }

The CF Worker validates service against a hardcoded allowlist derived from the scripts/rotate/ directory listing (no dynamic injection), maps it to sops_file via a static lookup table, and inserts into dev_r_worker_queue. No credential values flow through or are stored in the CF Worker at any point.

5b. File Inventory

All new and modified files for the complete implementation. Listed in dependency order.

New files

FilePurpose
scripts/rotate/ (directory)Per-service rotation scripts directory
scripts/rotate/README.mdContract documentation for rotation scripts
scripts/rotate/template.shCanonical bash template (from §3)
scripts/rotate/pdf_service_api_key.shPhase 2 — PDF service rotation
scripts/rotate/gf_rendering_renderer_token.shPhase 2 — Grafana renderer token
scripts/rotate/waha_api_key.shPhase 2 — WAHA API key
scripts/rotate/waha_hmac_secret.shPhase 3 — WAHA HMAC
scripts/rotate/redis_password.shPhase 3 — Redis on bms-4
scripts/rotate/bms4_n8n_api_key.shPhase 3 — n8n API key
scripts/rotate/p24_infra_wasabi_keys.shPhase 3 — Wasabi IAM key pair
scripts/rotate/traccar_password.shPhase 3 — Traccar admin
scripts/rotate/cloudflare_token_zintegrowana.shPhase 3 — CF scoped token
scripts/rotate/mongodb_rs0_prometheus_password.shPhase 3 — MongoDB Prometheus user
scripts/sops-sync-receiver.pyGHA receiver: decrypt payload → update SOPS → gh secret set
.github/workflows/sops-sync-receiver.ymlGHA receiver workflow triggered by rotation scripts
.github/workflows/rotate-schedule.ymlPer-service cron triggers → insert queue rows
docs/plans/plan-2427-rotate-secret-system.mdThis document (design plan)

Modified files

FileChangeWhy
scripts/queue-dispatcher-loop.pyAdd rotate-secret + super-light handler branchDispatcher must execute rotation scripts as a first-class job type
infra-src/meta-dispatcher/src/index.ts (CF Worker)Add POST /queue-rotation routeAllow external project sessions to trigger rotations

Supabase migrations (new files under migrations/)

Migration fileChange
migrations/NNNN_add_super_light_weight.sqlALTER TABLE dev_r_worker_queue — add super-light to weight check constraint
migrations/NNNN_add_rotate_secret_job_type.sqlALTER TABLE dev_r_worker_queue — add rotate-secret to job_type check constraint
migrations/NNNN_add_max_super_light_capacity.sqlALTER TABLE dev_r_server_capacity ADD COLUMN max_super_light

Not modified (left as-is in Phase 1)

FileReason
scripts/rotate-credentials.pyContinues to run for unmigrated credentials during transition; entries for migrated services are progressively commented out (feature-flag via scripts/rotate/<name>.sh existence check)
secrets/*.env.sopsNo structural change; rotation scripts update individual keys via receiver
.github/workflows/credential-rotation.ymlContinues to run Monday 06:00 UTC for unmigrated credentials

6. Rollout Plan

Priority: rotation frequency x blast radius. Low-blast-radius, high-frequency credentials ship first to validate the end-to-end pipeline.

PriorityCredentialSOPS fileKey name(s)Running onScript complexityNotes
1PDF service API keymonitoringPDF_SERVICE_API_KEYvps-i1Low — openssl rand + sed + docker compose up + curl healthBest first: self-contained, internal service, no external API dependency
2Grafana renderer tokenmonitoringGF_RENDERING_RENDERER_TOKENvps-i1Low — same pattern; shared token between grafana + renderer containersTests atomicity of two-container restart
3WAHA API keyvps-h1WAHA_API_KEYvps-h1 (SSH from bms-4)Low — openssl rand + SSH + docker compose upTests cross-server SSH execution path
4WAHA HMAC secretvps-h1WAHA_HMAC_SECRETvps-h1 + CF WorkerMedium — CF Worker secret must be updated BEFORE vps-h1 restart to prevent HMAC rejection gapTests CF Worker API call from within script
5Redis passwordn8n-bms4REDIS_PASSWORDbms-4Medium — coordinated restart of redis + all n8n workers (BullMQ reconnect)Tests multi-container restart ordering
6n8n API key (bms-4)n8n-bms4BMS4_N8N_API_KEYbms-4Medium — n8n REST POST /api/v1/user/api-key → delete old keyTests external service REST API in script
7Wasabi p24-infra IAM pairmonitoringP24_INFRA_WASABI_ACCESS_KEY + P24_INFRA_WASABI_SECRET_KEYbms-4 (boto3)High — Wasabi IAM API, key pair, multiple consumers in .envTests multi-key payload format
8Traccar admin passwordmonitoringTRACCAR_PASSWORDvps-i1Medium — Traccar REST API PUT /api/users/{id} + verify loginTests REST API auth verification pattern
9Cloudflare scoped tokenmonitoringCLOUDFLARE_TOKEN_ZINTEGROWANACF APIHigh — requires CF_GLOBAL_API_KEY to be valid; policy preservation logicTests CF token self-rotation (needs CF_GLOBAL_API_KEY in rotation script env)
10MongoDB Prometheus usern8n-bms4MONGODB_RS0_PROMETHEUS_PASSWORDbms-4 → bms-2 (SSH)High — MongoDB db.changeUserPassword() via SSH; PRIMARY detectionTests multi-hop SSH: bms-4 → bms-2

Credentials NOT migrated in Phase 1 (remain in rotate-credentials.py or manual flow):

  • GRAFANA_ADMIN_PASSWORD — Grafana CLI grafana-cli admin reset-admin-password has SQLite side effects; migrate Phase 2
  • SUPABASE_GRAFANA_PASSWORD — Management API; migrate Phase 2 after SOPS sync proven
  • MONGODB_RS0_ADMIN_PASSWORD — blast radius: entire Pinbox24 production stack; migrate only after Phase 1 fully validated
  • All Tier 3 (manual) credentials — remain on human-action GH issue flow
  • ATRAX_AUTH_STRING — requires vendor portal access; remains manual
  • SENTRY_AUTH_TOKEN — UI-only; remains manual

7. Security Model

What is exposed vs protected

TransportWhat flowsProtected byResidual threat
SSH stdout (bms-4 dispatcher ← script)[ROTATE] log lines only — no valuesScript contract: values in variables only; dispatcher suppresses non-[ROTATE] linesScript bug writes value to stdout — mitigated by dispatcher silently dropping non-[ROTATE] lines
SSH command arguments (bms-4 → target)sed commands use ${NEW_KEY} variable substitution — value is in shell process env, not in the command string passed via SSH execBatchMode=yes prevents interactive prompts; `sed -i “s^KEY=.*
age-encrypted payload in GH Actions inputbase64(age_encrypt(json_with_new_value)) — ciphertextage X25519 encryption; only runner with AGE_KEY_SOPS_SYNC_RECEIVER can decryptGH Actions run logs expose ciphertext — useless without private key
GH Actions runner environmentAGE_KEY_SOPS_SYNC_RECEIVER, SOPS_AGE_KEY, SUPABASE_SERVICE_KEYGH Actions secret masking; runner isolationSupply chain attack on pinned actions/checkout — mitigated by SHA pinning
SOPS encrypted file in gitAll credential values as ciphertextage X25519 encryption; multiple recipients neededRepository breach exposes ciphertext — safe without age key
dev_r_rotation_log rowscredential_name, reason, rotator, verify_result (HTTP code), log_lines ([ROTATE] lines)Row-level security; SUPABASE_SERVICE_KEY authSupabase breach exposes operational metadata only — no credential values by contract
GH issue commentsRotation outcomes: service name, success/failRepository visibilityCredential names in issue comments — acceptable; values never appear

Threat model by layer

Layer 1 — Rotation script on VPS server (bms-4 or target via SSH)

A compromised script could attempt to exfiltrate new credential values.

Mitigations:

  • Scripts are version-controlled in git — any change requires PR + review
  • The script only passes the public key to age --recipient — the private key (held only in GHA) is never on the VPS server
  • Dispatcher on bms-4 suppresses all stdout lines not starting with [ROTATE], preventing accidental prints from reaching journald
  • set -euo pipefail ensures the script aborts on any unexpected error rather than continuing in an undefined state

Layer 2 — Age payload in gh workflow run call

The gh workflow run call includes the base64 ciphertext as a command-line argument. This appears briefly in ps aux on bms-4 and in the GH Actions run trigger log.

Mitigations:

  • Ciphertext is not a secret — useless without AGE_KEY_SOPS_SYNC_RECEIVER
  • For future hardening: upload payload to a pre-signed Wasabi URL and pass only the URL as the input (removes the ciphertext from ps aux and from the gh workflow run argument entirely)

Layer 3 — GH Actions runner

The AGE_KEY_SOPS_SYNC_RECEIVER private key and SOPS_AGE_KEY (alias for AGE_KEY_GHA) are in runner memory during execution.

Mitigations:

  • All GH Actions steps pinned to SHA (not tag) to prevent tag-mutable supply chain attacks
  • Receiver key is separate from SOPS key — a receiver key compromise does not enable arbitrary SOPS file decryption; it only allows decrypting rotation payloads
  • The receiver validates credential_name against an allowlist before touching SOPS

Layer 4 — dev_r_rotation_log in Supabase

By design, new_value is NEVER stored in this table. Only the SOPS commit SHA and HTTP verify result are stored.

Layer 5 — SOPS files in git

Unchanged from current design. New rotation system does not add additional recipients or weaken encryption.

7b. Error Notification Design

Per CLAUDE.md §Error Notification Standard, every new script, GH Action, cron job, and automation MUST send a Discord embed + create a GH issue on error. The rotation system follows this standard at two levels:

Script-level (per-service rotation scripts)

Every rotation script includes a fail() function that triggers both notifications on any non-zero exit:

fail() {
    local code="$1"; shift
    local msg="$*"
    log "FAILED exit=${code} ${msg}"
 
    # Discord notification (credential name only — no values)
    if [[ -n "${P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL:-}" ]]; then
        curl -sf -X POST "$P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL" \
            -H "Content-Type: application/json" \
            -d "{\"embeds\":[{\"title\":\"RED ERROR — rotate/${CREDENTIAL_NAME}.sh\",\"color\":15158332,\"description\":\"Step: ${msg}\\nExit: ${code}\"}]}" \
            2>/dev/null || true
    fi
 
    # GH issue (only for exit codes 3 and 4 — requires human review)
    if [[ "$code" -ge 3 && -n "${GH_TOKEN:-}" ]]; then
        gh issue create --repo radieu/p24-infra \
            --title "[Infra] rotate/${CREDENTIAL_NAME} — exit ${code}: ${msg}" \
            --label "bug" \
            --body "## Rotation failed
 
Credential: \`${CREDENTIAL_NAME}\`
SOPS file: \`${SOPS_FILE}\`
Exit code: ${code}
Step: ${msg}
Log ID: ${LOG_ID:-none}
 
Rotation script exited non-zero. Exit 3 = verified but new credential NOT accepted by service.
Exit 4 = credential applied+verified but SOPS sync receiver NOT triggered.
 
Review \`dev_r_rotation_log\` row \`${LOG_ID:-none}\` for full [ROTATE] log.
" 2>/dev/null || true
    fi
 
    exit "$code"
}

For exit codes 1–2 (retriable errors), only Discord is notified. For exit codes 3–4 (human review required), both Discord and a GH issue are created.

GHA receiver-level (sops-sync-receiver.yml)

      - name: Notify on failure
        if: failure()
        env:
          P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL: ${{ secrets.P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL }}
          GH_TOKEN: ${{ secrets.GH_PAT }}
          LOG_ID: ${{ inputs.log_id }}
        run: |
          curl -sf -X POST "$P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL" \
            -H "Content-Type: application/json" \
            -d '{"embeds":[{"title":"RED ERROR — sops-sync-receiver","color":15158332,"description":"SOPS sync receiver failed. LOG_ID='"$LOG_ID"'"}]}' || true
 
          gh issue create --repo radieu/p24-infra \
            --title "[Infra] sops-sync-receiver — SOPS update failed" \
            --label "bug" \
            --body "SOPS sync receiver workflow failed. Log ID: $LOG_ID. Check GH Actions run for details." || true

Dispatcher-level (queue-dispatcher-loop.py)

When the dispatcher records a rotate-secret job as failed, it already posts a Prometheus metric increment for worker_queue_failed_total{job_type="rotate-secret"}. This triggers the existing Grafana alert WorkerJobFailed within 5 minutes, which routes to Discord via Alertmanager.

No additional dispatcher-level error notification is required — the existing monitoring stack covers it.

7c. Dangerous Operations and Rollbacks

Schema migration: ALTER TABLE with constraint changes

Risk: Dropping and re-adding a CHECK constraint is DDL that may lock the table momentarily.

Safeguard: Supabase Postgres supports ADD CONSTRAINT IF NOT EXISTS which is a no-op if the constraint already exists. The migration runs in the Supabase management UI or via psql on bms-4 (not via the rotation system itself).

Rollback command (if migration needs to be undone):

-- Rollback: remove super-light from weight constraint (restore to light|heavy only)
ALTER TABLE dev_r_worker_queue
  DROP CONSTRAINT IF EXISTS dev_r_worker_queue_weight_check;
ALTER TABLE dev_r_worker_queue
  ADD CONSTRAINT dev_r_worker_queue_weight_check
  CHECK (weight IN ('light', 'heavy'));
 
-- Rollback: remove rotate-secret from job_type constraint
ALTER TABLE dev_r_worker_queue
  DROP CONSTRAINT IF EXISTS dev_r_worker_queue_job_type_check;
ALTER TABLE dev_r_worker_queue
  ADD CONSTRAINT dev_r_worker_queue_job_type_check
  CHECK (job_type IN ('dev-issue', 'continue-issue', 'review-pr', 'review-plan', 'infra-task'));

Direct push to dev from GHA receiver

Risk: The sops-sync-receiver.yml pushes directly to dev, bypassing PR review. A malformed payload could corrupt a SOPS file and push the corruption to dev.

Safeguard: The receiver runs a canary decrypt (sops --decrypt ... | Out-Null) before git commit. If canary fails, it aborts and reverts with git checkout -- secrets/<file>.env.sops. The commit is never reached.

Rollback command (if a bad SOPS file slips through):

# On any machine with the age key:
git log --oneline secrets/<file>.env.sops | head -5
# Find the last good commit SHA, then:
git revert <bad-commit-sha>
# Or force-reset that file:
git checkout <good-sha> -- secrets/<file>.env.sops
sops --decrypt --input-type dotenv --output-type dotenv secrets/<file>.env.sops | Out-Null
# Verify exit 0, then push

Live credential rotation (apply step in rotation scripts)

Risk: A rotation script applies the new credential to the service, then fails at verification. The service is now running with the new credential but SOPS has the old value (drift). Old value may be revoked at provider already.

Safeguard: The script does NOT revoke the old credential at the provider. Old credential revocation is performed only by the SOPS sync receiver AFTER confirming the SOPS update succeeded. If the rotation script fails at verification (exit 3), the old credential is still valid at the provider.

Recovery path for exit 3:

  1. Check Discord/GH issue for the specific service and step
  2. SSH to the target server; test the new credential manually
  3. If new credential works: trigger gh workflow run sops-sync-receiver.yml manually with the same payload
  4. If new credential is broken: revert the .env on the target server (sed -i with the old value from SOPS) and restart the service

Recovery path for exit 4 (applied + verified, SOPS sync failed):

  1. New credential is live and working
  2. SOPS file still has the old value
  3. Trigger gh workflow run sops-sync-receiver.yml manually — the rotation script’s payload is stored in /tmp/rotate-<CREDENTIAL_NAME>-payload.age on bms-4 for 24 hours before cleanup
  4. If the payload file is gone: a secret-manager session must manually update SOPS with the value known to be live (requires SSH to target server to read the .env file — safe, not a chat exposure)

7d. Verification Commands

After Phase 1 infrastructure deployment (before any credential migration):

# 1. Confirm schema migration applied
# Run on Supabase SQL editor or via psql on bms-4:
SELECT conname, consrc FROM pg_constraint
WHERE conrelid = 'dev_r_worker_queue'::regclass
  AND conname LIKE '%weight%' OR conname LIKE '%job_type%';
-- Should show super-light and rotate-secret in the allowed values
 
# 2. Confirm max_super_light column exists
SELECT server_label, max_super_light FROM dev_r_server_capacity
WHERE server_label = 'bms-4';
-- Should return: bms-4 | 8
 
# 3. Dry-run rotation script for PDF_SERVICE_API_KEY
ssh root@54.36.123.110 "bash /opt/p24-infra/scripts/rotate/pdf_service_api_key.sh --dry-run"
# Expected output: [ROTATE] ... generate ok  [ROTATE] ... DRY_RUN skip apply and verify
# Exit code: 0
 
# 4. Insert a test super-light rotate-secret queue row and confirm dispatcher picks it up
# (dry_run=true to avoid live rotation)
curl -sf -X POST "${SUPABASE_URL}/rest/v1/dev_r_worker_queue" \
  -H "apikey: ${SUPABASE_SERVICE_KEY}" \
  -H "Authorization: Bearer ${SUPABASE_SERVICE_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"repo":"radieu/p24-infra","job_type":"rotate-secret","weight":"super-light","priority":1,"metadata":{"service":"PDF_SERVICE_API_KEY","sops_file":"monitoring","dry_run":true},"status":"queued"}'
 
# 5. Check the queue row transitions to done within 5 minutes
SELECT id, job_type, status, log_lines, started_at, completed_at
FROM dev_r_worker_queue
WHERE job_type = 'rotate-secret'
ORDER BY id DESC LIMIT 1;
-- status should be: done
-- log_lines should contain [ROTATE] lines
 
# 6. First live rotation: PDF_SERVICE_API_KEY
# Confirm current value works:
ssh root@217.154.82.162 "curl -sf -o /dev/null -w '%{http_code}' \
  -H 'X-Api-Key: $(grep PDF_SERVICE_API_KEY /opt/p24-infra/monitoring/.env | cut -d= -f2-)' \
  http://localhost:8080/health"
# Should return: 200
 
# 7. Trigger live rotation:
curl -sf -X POST "${SUPABASE_URL}/rest/v1/dev_r_worker_queue" \
  -H "apikey: ${SUPABASE_SERVICE_KEY}" \
  -H "Authorization: Bearer ${SUPABASE_SERVICE_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"repo":"radieu/p24-infra","job_type":"rotate-secret","weight":"super-light","priority":1,"metadata":{"service":"PDF_SERVICE_API_KEY","sops_file":"monitoring","dry_run":false},"status":"queued"}'
 
# 8. After rotation completes: verify SOPS was updated
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops |
  ForEach-Object { if ($_ -match "^PDF_SERVICE_API_KEY=") { Write-Host "Key present: YES" } }
# Should output: Key present: YES
 
# 9. Verify the new value is live (by testing the endpoint, not by reading the value):
ssh root@217.154.82.162 "PKEY=\$(grep PDF_SERVICE_API_KEY /opt/p24-infra/monitoring/.env | cut -d= -f2-);
  HTTP=\$(curl -sf -o /dev/null -w '%{http_code}' -H \"X-Api-Key: \$PKEY\" http://localhost:8080/health);
  echo \"http_code=\$HTTP\""
# Should output: http_code=200
 
# 10. Verify rotation log row was closed:
SELECT id, secret_name, status, sops_commit, completed_at
FROM dev_r_rotation_log
WHERE secret_name = 'PDF_SERVICE_API_KEY'
ORDER BY id DESC LIMIT 1;
-- status: completed, sops_commit: <non-null SHA>, completed_at: <timestamp>

8. Open Questions

The following decisions require human input before implementation begins. Each states the implication of leaving it unanswered.

Q1: Should rotate-secret queue rows use github_issue_number = NULL, or should a stub GH issue be created per rotation run?

The current dev_r_worker_queue schema and some Postgres functions (e.g., reset_stale_workers()) may assume github_issue_number IS NOT NULL. Options: (a) relax the constraint + update reset_stale_workers() to handle NULL; (b) create a reusable “rotation-sentinel” issue per service that is referenced repeatedly; (c) create a new GH issue per rotation run (audit trail but issue spam).

Implication: if left as NULL without schema change, queue functions may fail when processing rotate-secret rows.

Q2: Should the SOPS sync receiver push directly to dev or open a PR from a bot/rotate-<credential> branch?

Direct push to dev is faster and matches what sops_update_key() in rotate-credentials.py does today. Opening a PR respects the branching policy in CLAUDE.md (“all changes via PR, never direct to main/dev”) but adds latency and PR noise.

Option: allow direct push to dev only for commits with [auto-rotate] in the message, treating it as an exception similar to secrets-sync.yml deployments.

Implication: without a decision, the receiver will need a placeholder approach that may need to be changed.

Q3: Failure mode when sops-sync-receiver.yml cannot be triggered (GHA outage, quota, gh workflow run fails)?

At this point, the new credential is live on the service but SOPS has the old value — a drift state. What is the recovery path?

Options: (a) rotation script retries gh workflow run up to 3 times with backoff before returning exit 4; (b) a sops-drift-monitor.yml daily workflow detects drift (live service credential test vs SOPS decrypted value) and alerts; (c) Grafana alert on dev_r_rotation_log rows with status=completed but sops_commit=NULL after 1 hour.

Implication: without (b) or (c), SOPS drift could persist undetected.

Q4: How should scripts authenticate to services for the verify step?

For HTTP-based verifies (WAHA, n8n, Traccar, PDF service), curl -H "Authorization: Bearer ${NEW_KEY}" from within the SSH session is safe (curl does not log headers by default). For database-based verifies (MySQL ALTER USER, Postgres ALTER ROLE), a docker exec or psql call inside SSH is needed.

Should verification be part of the rotation script or a separate verify/<service>.sh script that can be tested independently without triggering a rotation?

Implication: independent verify scripts improve testability but add complexity.

Q5: Should AGE_KEY_SOPS_SYNC_RECEIVER be added as a SOPS recipient to any SOPS files?

Current design: the receiver uses AGE_KEY_GHA (already a SOPS recipient) for SOPS operations, and AGE_KEY_SOPS_SYNC_RECEIVER only for decrypting the rotation payload. This avoids adding a third key to every SOPS file.

If the receiver key is added as a SOPS recipient, the SOPS update step is simpler (receiver decrypts SOPS directly), but a receiver key compromise would expose all SOPS-encrypted secrets.

Implication: architectural choice with security model consequences.

Q6: What is the rotate-schedule.yml trigger strategy — one workflow per service with its own cron, or a single matrix workflow?

Option A: one workflow per service, each with its own schedule: cron. Simple, independent, easy to disable one service. Downside: 10–30 workflow files.

Option B: one rotate-schedule.yml with a matrix strategy over a service list defined in a config file. Cleaner but harder to give each service a custom cron schedule.

Implication: the number of GH workflow files in .github/workflows/ and the flexibility of per-service schedule management.

Q7: What is the transition strategy for rotate-credentials.py?

(a) Keep rotate-credentials.py as-is for unmigrated credentials; migrated credentials are removed from it service by service (dual-mode during transition). (b) Immediately refactor rotate-credentials.py into a thin dispatcher that calls scripts/rotate/<name>.sh for all services. (c) Feature-flag inside rotate-credentials.py — if a scripts/rotate/<name>.sh exists, skip the Python rotator for that credential.

Implication: option (c) requires the least rework and provides the safest migration path, but rotate-credentials.py must be aware of the scripts/rotate/ directory.

Q8: How should gh workflow run sops-sync-receiver.yml wait for the receiver to complete?

The gh workflow run call returns immediately after queueing the workflow. The rotation script exits 0 without knowing whether SOPS was actually updated. If the receiver fails (GHA flake, Python error), the queue row shows done but SOPS is stale.

Options: (a) poll gh run list after triggering until the run completes; (b) the receiver POSTs a callback to a Supabase endpoint that the rotation script polls; (c) accept eventual consistency — the rotation is “done” when the new credential is live, and SOPS sync is a separate concern tracked via dev_r_rotation_log.sops_commit.

Implication: option (a) is simplest but adds 1–5 minutes to script runtime. Option (c) is the fastest but requires the drift monitor (Q3) to be reliable.

9. Implementation Checklist

Milestone: Design (this plan) → architect approval → moves to In Progress

Phase 1 — Infrastructure (no credential migration)

  • Supabase migration: add super-light to weight check, add rotate-secret to job_type check
  • Add max_super_light column to dev_r_server_capacity
  • Generate age key pair for AGE_KEY_SOPS_SYNC_RECEIVER; add GH Secret; hardcode public key in template
  • Create scripts/sops-sync-receiver.py
  • Create .github/workflows/sops-sync-receiver.yml
  • Update scripts/queue-dispatcher-loop.py to handle rotate-secret + super-light
  • Create scripts/rotate/ directory with README.md and template.sh
  • Create rotate-schedule.yml GHA workflow structure

Phase 2 — First 3 services (proof of concept)

  • scripts/rotate/pdf_service_api_key.sh — dry-run + live
  • scripts/rotate/gf_rendering_renderer_token.sh — dry-run + live
  • scripts/rotate/waha_api_key.sh — dry-run + live
  • Verify SOPS sync receiver produced correct commit for each

Phase 3 — Services 4–10 from §6

Phase 4 — Decommission migrated entries from rotate-credentials.py

10. Compliance

Per CLAUDE.md §Compliance, new services and scheduled automations require dev_r_services registration.

New automations to register in dev_r_services:

service_namedescriptioncompliance_workbookrotation_type
sops-sync-receiverGHA workflow receiving age-encrypted rotation payloads and updating SOPSyesN/A
rotate-scheduleGHA cron trigger workflow dispatching per-service rotation jobs to queueyesN/A
rotate-script-*Per-service bash rotation scripts (one row per service as they go live)yesauto

Not AI-powered — no entry in dev_r_ai_systems required. No EU AI Act high-risk obligations apply (CLAUDE.md deadline 2026-08-02).

Ops doc required (docs/rotate-secret-operations.md) — to be created in Phase 1 alongside the implementation. Contents: how to trigger a manual rotation, how to monitor queue status, how to recover from each exit code, how to add a new rotation script.

  • Issue: #2427
  • Existing rotator: scripts/rotate-credentials.py (1668 lines — current monolith)
  • OVH Redis standalone: scripts/ovh_redis_rotate.py
  • SOPS operations playbook: docs/playbooks/secret-manager.md
  • Rotation tier matrix: docs/playbooks/secret-rotation-access-matrix.md
  • Worker queue operations: docs/playbooks/worker-queue-operations.md
  • Infra task executor: docs/playbooks/infra-task-executor.md
  • Audit trail pattern: docs/playbooks/static-api-key-incident-rotation.md §Step 1
  • Credential rotation GHA: .github/workflows/credential-rotation.yml
  • Dispatch to queue GHA: .github/workflows/dispatch-to-queue.yml
  • SOPS secrets sync: .github/workflows/secrets-sync.yml

Graph View