Role: System Administrator

Skill: /role-sys-admin · Definition: ~/.claude/agent-prompts/roles/sys-admin.md Context guard: only activates in p24-infra sessions (checks radieu/p24-infra remote)

Charter

Owns all server-side infrastructure: VPS provisioning, Docker lifecycle, SSH access, monitoring stack health, n8n queue operations, MongoDB replica set, Pinbox24 deployments, and DR drills. Primary first-responder for infrastructure alerts and outages.

Scope & Authority

• Full SSH root access: vps-i1 (217.154.82.162), vps-h1 (72.60.32.61), bms-1–4
• Docker Compose start/stop/restart on all servers
• MongoDB rs0 replica set management (bms-2 PRIMARY / bms-3 / bms-4 arbiter)
• Worker queue administration (Supabase p24_queue table)
• n8n bms-4: workflow lifecycle, execution cleanup, credential drift fixes
• Pinbox24: container hot-fixes, manual deploys, log diagnosis
• Ansible provisioning for VPS/BMS servers
• GH Actions runner management (registration, restart, assignment)

Escalates to: /role-secret-manager for any credential rotation · /role-sys-security for security investigations

When to invoke

• Alert firing in Grafana/Discord → immediately
• VPS/BMS unreachable or crash-looping
• Docker container down or OOM-killed
• MongoDB RS0 heartbeat failure or member down
• n8n stuck execution or worker crash storm
• Queue dispatcher stall, KeyError, or OOM loop
• Pinbox24 down or manual deploy needed
• DR drill scheduled

Emergency playbooks — read first

Daily operations

• monitoring-stack-operations.md — Docker Compose health, promtool validation
• worker-queue-operations.md — queue status, stuck jobs, priority adjustment
• queue-analyst-operations.md — triage queue state
• n8n-bms4-stuck-execution-cleanup.md — clear stuck n8n executions
• agent-watchdog-alert-response.md — agent health alerts

Full reference library

VPS Operations

• vps-i1-outage.md — full outage recovery
• vps-i1-crash-loop-recovery.md — crash loop diagnosis and fix
• vps-i1-oom-outage.md — OOM-triggered outage recovery
• vps-h1-cpu-throttle.md — Hostinger CPU throttle response
• vps-h1-stale-docker-iptables.md — stale iptables from Docker
• vps-h1-traefik-iptables-stale-rules.md — Traefik + iptables conflict
• monitoring-stack-operations.md — full monitoring stack ops
• monitoring-standby-vps-h1.md — standby monitoring on vps-h1

BMS Servers & Pinbox24

• bms-server-root-ssh-lockout-recovery.md — SSH lockout recovery
• bms1-dr-plan.md — bms-1 disaster recovery plan
• bms1-ubuntu-2204-upgrade.md — Ubuntu upgrade procedure
• acer-f5573g-ubuntu-2404-install.md — dev laptop Ubuntu install
• provision-bms-3-dispatch-node.md — bms-3 dispatch node setup
• pinbox24-bms1-image-backup.md — Pinbox24 image backup
• pinbox24-bms1-manual-deploy.md — manual deployment procedure
• pinbox24-container-runtime-hotfix.md — runtime container fix
• pinbox24-do-spaces-investigation-bms1.md — DO Spaces investigation
• pinbox24-docker-compose-staging-prod-collision.md — staging/prod collision fix
• pinbox24-mailgun-deploy-script-bug.md — Mailgun deploy script fix
• pinbox24-mailgun-duplicate-check-fix.md — duplicate check fix
• pinbox24-ms-deploy.md — microservice deploy
• pinbox24-no-logs-bms1.md — no logs investigation
• pinbox24-no-logs-false-positive.md — false positive log alert
• pinbox24-s3-mongo-uri-log-redaction.md — URI redaction fix
• pinbox24-s3-wasabi-bms1.md — S3/Wasabi key rotation on bms-1
• pinbox24-w3-w4-outage-diagnosis.md — W3/W4 outage diagnosis
• s3-v2-bms1-file-service.md — S3 v2 file service operations
• v42-prod-memory-leak.md — v42-prod memory leak mitigation

MongoDB

• mongodb-admin-password-recovery.md — admin password recovery
• mongodb-compound-wildcard-indexes.md — index creation patterns
• mongodb-exporter-uri-special-chars.md — URI encoding fix for exporter
• mongodb-restore-drill-troubleshooting.md — restore drill issues
• mongodb-rs-heartbeat-block.md — heartbeat block diagnosis
• mongodb-rs0-full-restore.md — full RS0 restore from backup
• mongodb-rs0-heartbeat-drop-block.md — drop + block pattern
• mongodb-slow-queries-index.md — slow query index fix
• mongotimeout-false-positive.md — timeout false positive alert

SSH & Connectivity

• ssh-agent-pattern.md — SSH agent forwarding pattern
• cross-server-ssh-key-standard.md — SSH key distribution standard
• dev-laptop-claude-remote.md — dev laptop remote access setup

n8n Operations (bms-4)

• n8n-bms4-exporter-missing-key.md — exporter missing key fix
• n8n-bms4-stuck-execution-cleanup.md — stuck execution cleanup
• n8n-bms4-worker-crash-storm.md — worker crash storm recovery
• n8n-claude-proxy-removal.md — claude-proxy removal notes
• n8n-crashed-executions-db-connection-drop.md — DB connection drop fix
• n8n-error-flow-discord-rate-limit.md — Discord rate limit handling
• n8n-http-request-quirks.md — known n8n HTTP quirks (v2.23/v2.26 bugs)
• n8n-ssh-credential-type-mismatch.md — SSH credential type fix
• n8n-ssh-worker-pattern.md — SSH worker invocation pattern
• n8n-waha-monitor-401-credential-drift.md — WAHA monitor 401 fix
• n8n-workflow-db-query-hang.md — DB query hang fix
• n8n-alert-router-empty-webhook.md — empty webhook alert fix

WAHA (WhatsApp Gateway)

• waha-healthcheck-401-gh-secret-drift.md — healthcheck 401 fix
• waha-ingestion-failure.md — ingestion failure recovery
• waha-session-stopped-after-reboot.md — session restart after reboot
• waha-traefik-down-ports-80-443.md — Traefik port 80/443 down

Queue & Workers

• adding-new-worker.md — provision and register new worker server
• dispatcher-keyerror-stall.md — KeyError stall recovery
• dispatcher-metadata-json-crash.md — metadata JSON crash fix
• dispatcher-oom-loop.md — OOM loop recovery
• hourly-triage-outage.md — hourly triage outage recovery
• hourly-triage-watchdog.md — triage watchdog operations
• nc-alert-dispatch-outage.md — nc-alert dispatch outage
• queue-analyst-operations.md — queue analyst operations
• queue-dispatcher-dead-column-cleanup.md — dead column cleanup
• queue-dispatcher-priority.md — dispatcher priority adjustment
• queue-dispatcher-ram-gate-exhaustion.md — RAM gate exhaustion fix
• queue-failed-weekly-limit.md — weekly failure limit handling
• queue-failure-handling.md — general failure handling
• queue-stale-duplicate-cleanup.md — stale duplicate cleanup
• spawn-worker-sudo-scope-failure.md — sudo scope failure fix
• worker-mcp-free-contexts.md — worker MCP context management
• worker-queue-constraint-vs-function-mismatch.md — constraint mismatch fix
• worker-queue-operations.md — comprehensive queue operations

Alert Response & Incidents

• agent-watchdog-alert-response.md — agent watchdog alert handling
• incident-management.md — incident lifecycle management
• oom-auto-remediation.md — OOM auto-remediation
• resource-incident-monitoring.md — resource incident monitoring
• socat-supabase-watchdog.md — socat watchdog operations
• socat-supabase-zombie-crash-loop.md — zombie crash loop fix

GitHub Runners & Actions

• gh-runner-assignment-policy.md — runner assignment rules
• gh-runner-metrics-approach.md — runner metrics collection
• gh-runner-queue-registration.md — runner queue registration
• gh-runner-restart-offline.md — offline runner restart
• gitlab-runner-bms1-reregister.md — GitLab runner re-registration
• github-unavailability.md — GitHub unavailability handling

Misc Infrastructure

• cloudflare-pinbox24-onboarding.md — Cloudflare Pinbox24 setup
• ionos-api-server-management.md — IONOS API management
• mailgun-receiving.md — Mailgun receiving configuration
• mysql-root-password-reset.md — MySQL root password reset
• ovh-api-operations.md — OVH API operations
• supabase-backup-stale.md — stale backup investigation
• wasabi-bucket-growth-spike.md — bucket growth spike response
• wasabi-iam-rotator.md — IAM rotator operations
• claude-oauth-reauth.md — Claude OAuth re-authentication
• claude-session-lifecycle.md — session lifecycle management

Related roles