p24-infra Docs

github-unavailability

7 min read

Playbook — Surviving GitHub Unavailability

Trigger: GitHub is unreachable, degraded, or access to radieu/p24-infra / radieu/et-operational-platform is lost — for any reason:

  • GitHub-wide outage (git push/gh/Actions failing globally — check githubstatus.com)
  • Account suspension / lockout / 2FA-device loss on the radieu org owner
  • Accidental or malicious repo deletion / force-push / history rewrite
  • Compromised PAT used to tamper with repos or Actions
  • Billing lapse disabling private repos or Actions minutes

This is the prevention + response plan for that scenario. It is the GitHub counterpart to ../disaster-recovery.md (which covers VPS/Supabase loss) and complements its Scenario E — GitHub Actions Runner Offline (a runner being down ≠ GitHub being down).

1. Why this matters — what depends on GitHub

GitHub is a single point of failure for far more than source hosting. If it disappears, the following stop working until restored or routed around:

DependencyWhat breaks when GitHub is down
Git hosting (origin)No push/pull/clone/PR for either repo
GitHub Actions (~45 workflows)All CI/CD, deploys, backups-via-Actions, scheduled jobs (hourly-devops-triage, supabase-backup, credential-rotation, secrets-sync, …) stop firing
Self-hosted runners (8 workflows use self-hosted)Jobs queue forever; deploy-on-merge halts
GitHub IssuesEntire autonomous pipeline halts — triage, worker queue dispatch, gh issue create error-reporting, SLA watchdog
GitHub SecretsCI-only secrets (VPS_SSH_PRIVATE_KEY, AGE_KEY_GHA, VPS IPs) become unreadable to any new workflow run
gh CLI automationEvery script/skill that shells out to gh issue/gh pr/gh api errors
Error Notification StandardThe “create a GitHub Issue on error” half of the standard fails (Discord half still works)
Deployssecrets-sync.yml, deploy-monitoring-config.yml, portal-deploy.yml, Vercel-via-Actions all stall

Key insight on secrets: real infra secrets do not live only in GitHub — they live in secrets/*.env.sops (age-encrypted, inside the git repo) and the age private key is backed up offline (KeePass + the DR runbook). So as long as we have an off-GitHub copy of the repo, we still have every secret. GitHub Secrets only hold CI-only values, which are re-creatable. This is why the git repo itself is the crown jewel to protect — protect the repo and you protect the secrets.

2. Prevention — controls that must be in place before an outage

2.1 Off-GitHub git mirror to Wasabi (primary control)

A nightly job mirrors both repos (full history + all refs) into an age-encrypted bundle on Wasabi S3 — storage we already pay for and which is fully independent of GitHub.

  • Script: scripts/backup-git-mirror.sh
  • Destination: s3://ecotrans-backups/git-mirror/{daily,weekly,monthly}/…/backup.tar.zst.age
  • Retention: 7 daily / 4 weekly / 3 monthly (same rotation as the other backups)
  • Runs from VPS cron, NOT GitHub Actions — running it on Actions would make the resilience control depend on the very thing it protects against.

Install (run on bms-4, which has the repo, creds, and 32 GB headroom):

# 1. Ensure backup env is present (same vars as scripts/backup-ionos.sh):
#    AGE_PUBKEY, WASABI_BACKUP_ACCESS_KEY, WASABI_BACKUP_SECRET_KEY,
#    P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL
#    plus a clone token: GIT_MIRROR_GH_TOKEN (read-only PAT) or GH_TOKEN
grep -E 'AGE_PUBKEY|WASABI_BACKUP|GIT_MIRROR_GH_TOKEN' /root/.backup-env
 
# 2. Add cron (03:30 UTC daily, after the other nightly backups):
( crontab -l 2>/dev/null; \
  echo '30 3 * * * /opt/p24-infra/scripts/backup-git-mirror.sh >> /var/log/p24-backup.log 2>&1' \
) | crontab -
 
# 3. Smoke-test once by hand:
/opt/p24-infra/scripts/backup-git-mirror.sh

The script emits a Prometheus freshness metric (git_mirror_last_success_timestamp) via the node_exporter textfile collector — alert if it goes stale (see §2.4).

2.2 Live working clones on every node (already true — keep it that way)

Every VPS/bms node already has a full clone at /opt/p24-infra. A full git clone is a complete history backup. Do not shallow-clone the canonical checkouts. During an outage these local clones are read-write and can serve as the temporary source of truth (§3).

Keep a second git remote that can become origin within minutes. Two already-available options:

  • GitLab — we already run an n8n workflow (save-workflows-to-gitlab, lShfAU645lP0rcR6) that pushes to a GitLab repo, so a GitLab account/namespace exists. Create p24-infra + et-operational-platform mirror projects there.
  • Codeberg / self-hosted Gitea on a bms node — fully under our control.

Configure a push mirror so it stays current without manual effort:

cd /opt/p24-infra
git remote add mirror git@gitlab.com:<ns>/p24-infra.git
# Push all branches + tags on demand (or wire into backup-git-mirror.sh):
git push --mirror mirror

A warm standby turns “GitHub is down for hours” from a work-stoppage into a one-command git remote set-url origin … failover.

2.4 Monitoring & detection

  • Mirror freshness alert — Prometheus rule on time() - git_mirror_last_success_timestamp > 129600 (36 h) → warning.
  • GitHub reachability — a lightweight Blackbox/HTTP probe of https://api.github.com/zen (or status page) so dashboards show GitHub up/down rather than guessing from failed jobs.
  • Status page — bookmark https://www.githubstatus.com/; subscribe radieu@gmail.com to incident emails for instant external confirmation.

3. Response — what to do during an outage

Step 0 — Confirm scope (2 min)

curl -fsS https://api.github.com/zen           # global GitHub up?
curl -fsS https://www.githubstatus.com/api/v2/status.json | jq .status.description
git ls-remote https://github.com/radieu/p24-infra HEAD   # repo-level access?

Classify:

  • A — GitHub-wide outage (status page red): wait it out; switch to local workflow below; do not thrash.
  • B — Account/repo access lost (status page green, we get 404/403): escalate to human immediately (§4) — likely auth/billing/suspension, not an outage.
  • C — Repo tampered (force-push, deletion, bad history): freeze, then restore from mirror (§3.3).

3.1 Keep working locally (scenarios A & B)

GitHub down does not stop development — git is distributed.

# Work as normal on the local clone; commit freely.
cd /opt/p24-infra
git checkout -b patch/<issue>-<slug>
# … edit, commit …
 
# If a warm standby exists (§2.3), push there instead of GitHub:
git push mirror patch/<issue>-<slug>

Queue the PRs mentally / in a local note; open them on GitHub once it returns. Autonomous AI-Dev workers should pause new dispatch while GitHub Issues are unreachable (the queue claim + gh issue view calls will fail fast) — this is expected; do not retry-loop.

3.2 Deploys during an outage

Deploys are normally merge → Actions → server. With Actions down, deploy manually from a node’s local clone (the same commands the workflows run):

# Monitoring stack (what deploy-monitoring-config.yml does):
ssh root@217.154.82.162 "cd /opt/p24-infra && git pull <local-or-mirror> && cd monitoring && docker compose up -d"
# Secrets (what secrets-sync.yml does): decrypt SOPS locally and write the .env, then restart the service.

Because secrets live in SOPS in the repo (not only in GitHub Secrets), manual deploys work fully offline as long as you hold the age key.

3.3 Recover from repo tampering / loss (scenario C)

Restore the canonical repo from the Wasabi mirror:

# 1. Pull the latest encrypted mirror bundle:
AWS_ACCESS_KEY_ID=$WASABI_BACKUP_ACCESS_KEY AWS_SECRET_ACCESS_KEY=$WASABI_BACKUP_SECRET_KEY \
  aws --endpoint-url https://s3.eu-central-1.wasabisys.com \
  s3 cp s3://ecotrans-backups/git-mirror/daily/$(date -u +%F)/backup.tar.zst.age /tmp/gm.age
 
# 2. Decrypt + unpack (needs the age private key — /home/<user>/.age/p24-infra-keys.txt):
age -d -i ~/.age/p24-infra-keys.txt /tmp/gm.age | zstd -d | tar -C /tmp -xf -
# → /tmp/git-mirror/p24-infra.git  (a --mirror clone: full history + all refs)
 
# 3. Re-create or repoint the GitHub repo, then push the recovered history:
cd /tmp/git-mirror/p24-infra.git
git push --mirror https://github.com/radieu/p24-infra.git

If GitHub is permanently unavailable, push the mirror to the warm standby (§2.3) and make it the new origin across all nodes:

git push --mirror git@gitlab.com:<ns>/p24-infra.git
# On each node: git remote set-url origin git@gitlab.com:<ns>/p24-infra.git

4. Escalation

Follow the Error Notification Standard, but note its GitHub-Issue half may be unavailable — so Discord is the primary channel during a GitHub outage:

  1. Discord → P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL (red embed), title 🔴 GitHub unavailable — <scope A/B/C>.
  2. Email radieu@gmail.com directly (do not rely on Issue-created email).
  3. For scenario B/C (access lost or tampering) escalate to the human owner immediately — these need GitHub-account-level action (re-auth, billing, support ticket, repo restore) that automation cannot perform.
  4. File the GitHub Issue retroactively once GitHub returns, for the audit trail.

5. Prevention checklist (verify quarterly)

  • backup-git-mirror.sh cron installed on bms-4 and last run succeeded (git_mirror_last_success_timestamp fresh)
  • A restore drill performed: decrypt the latest mirror bundle and confirm git fsck passes on the unpacked --mirror clone
  • Warm standby remote (GitLab/Gitea) exists and is < 7 days stale
  • age private key backed up offline (KeePass) — without it the mirror is useless (cross-ref ../disaster-recovery-runbook.md)
  • Mirror-freshness Prometheus alert active
  • radieu@gmail.com subscribed to githubstatus.com incident emails

Prevention summary

ControlProtects againstStatus field to watch
Wasabi git mirror (backup-git-mirror.sh)Repo loss, tampering, GitHub-wide lossgit_mirror_last_success_timestamp
Live local clones on every nodeShort outages, manual deploysn/a (always present)
Warm standby remoteProlonged outage / GitHub abandonmentmirror push timestamp
SOPS-in-repo secrets + offline age keyLoss of GitHub SecretsDR runbook §SOPS key
Discord-first escalationIssue-creation being unavailablen/a

Auto-generated by worker agent for issue #1236.

Graph View

Backlinks