disaster-recovery

P24-Infra Disaster Recovery Runbook

RTO/RPO Targets

Project type	RTO	RPO	Recovery path
p24-infra monitoring stack	4h	2h	Ansible re-provision + git pull + docker compose up
n8n on BMS-4	1h	30min	Docker restart; workflow restore from git
et-operational-platform	2h	0	Vercel rollback to previous deployment
brandpilot	2h	0	Vercel rollback
brandingpilot-client content	24h	24h	Wasabi restore → Supabase import
p24-branding Supabase	4h	24h	Supabase point-in-time restore
SOPS age key	8h	last commit	KeePass offline backup + git history
BMS-4 entire server	24h	24h	New OVH server + Ansible provision + git pull

Scenario A — IONOS vps-i1 Complete Loss

Symptoms: All monitoring URLs unreachable, Prometheus gone.

1. Provision new IONOS VPS (use /provision-vps or Ansible):

   ansible-playbook ansible/playbooks/provision-new-vps.yml -e "target_ip=NEW_IP"

2. Update DNS A record in Cloudflare:

   python3 scripts/dns-manager.py upsert "*.vps-i1.infra.zintegrowana.online" "NEW_IP"

3. Deploy monitoring stack:

   ssh root@NEW_IP "cd /opt/p24-infra && docker compose up -d"

4. Verify: Prometheus healthy, Grafana loads, Alertmanager test email succeeds.
5. Update dev_r_services with new IP.

ETA: ~4 hours

Scenario B — p24-branding Supabase Data Loss

Symptoms: n8n cannot read client tokens; brand_scheduler queries fail.

1. Supabase dashboard → Database → Backups → Restore to point-in-time
2. Select recovery time (before incident)
3. After restore: verify tables intact:

   SELECT COUNT(*) FROM brand_scheduler;
   SELECT COUNT(*) FROM brand_client_config;

4. Re-run failed scheduled content:

   UPDATE brand_scheduler
   SET status='pending'
   WHERE status='failed' AND scheduled_at > NOW() - INTERVAL '24 hours';

5. Verify n8n connects via pool (port 6543).

ETA: ~4 hours

Scenario C — SOPS Age Key Lost (Primary Key Only)

Symptoms: Cannot decrypt secrets/*.env.sops; sops commands fail.

1. Retrieve from KeePass offline backup (Radek only — physical access required)
2. Write recovered key to C:\Users\konar\.age\p24-infra-keys.txt
3. Verify decryption: sops -d secrets/monitoring.env.sops > /dev/null
4. Rotate key pair (generate new key to prevent future same incident):
   • age-keygen -o C:\tmp\new-age-key.txt
   • Add new public key to .sops.yaml
   • Re-key every file — not sops updatekeys, which is broken on dotenv *.env.sops in SOPS 3.9.1 with no working flag combination (#4601): Get-ChildItem secrets\*.env.sops | ForEach-Object { .\scripts\sops-set.ps1 -SopsFile $_.FullName -RekeyOnly }
   • Distribute new key to all VPSes
   • Remove old public key from .sops.yaml
   • Commit and push
5. Document in docs/secrets-rotation-log.md.

ETA: 8 hours

Scenario D — BMS-4 Complete Loss

Symptoms: n8n unreachable, AI-Dev-BMS4-1 offline, MongoDB arbiter lost.

1. Provision new OVH Kimsufi server (OVH control panel — Radek only)
2. Update ~/.ssh/config with new IP
3. Run Ansible: ansible-playbook ansible/playbooks/provision-new-vps.yml
4. MongoDB: rs.add({host: "NEW_IP:27017", arbiterOnly: true})
5. Restore n8n workflows from git:

   for f in n8n-workflows/*.json; do
     curl -s -X POST "https://n8n.NEW_DOMAIN/api/v1/workflows" \
       -H "X-N8N-API-KEY: $N8N_API_KEY" -d @"$f"
   done

6. Update DNS: n8n.bms-4.infra.zintegrowana.online → new IP

ETA: 24 hours

Scenario E — GitHub Actions Runner Offline

Symptoms: CI workflows queued; self-hosted runner shows offline in GH Settings.

1. Check runner process on relevant VPS:

   ssh root@217.154.82.162 "systemctl status actions.runner.* || pgrep -a Runner"

2. Restart runner service:

   ssh root@217.154.82.162 "systemctl restart actions.runner.*"

3. If token expired (runner shows “no token”): re-register from GH Settings → Actions → Runners.
4. Verify CI queue clears within 5 minutes.

ETA: 30 minutes

Post-Incident Checklist (All Scenarios)

• Root cause documented in GitHub Issue
• Secrets rotated if compromise suspected
• Monitoring confirmed healthy
• Backup verified post-recovery
• docs/secrets-rotation-log.md updated if any credentials changed
• Weekly self-assessment report will auto-capture the incident metrics

Related Documentation

• disaster-recovery-runbook.md — Detailed scenario runbook (MongoDB failover, vps-i1, vps-h1, bms-1, Supabase, Claude auth)
• secrets-management.md — SOPS+age credential management playbook
• infrastructure-overview.md — Full infrastructure map