A credential stored in SOPS stops authenticating against the live service it is meant to
unlock — most commonly because:
The live password was rotated/reset directly on the server (or via a break-glass recovery
procedure) without SOPS being updated to match.
A partial multi-file rotation updated some SOPS mirrors of a credential but missed one
(this credential class often has 4–5 mirror copies — see
docs/playbooks/mongodb-credential-rotation.md §SOPS File Map for the MongoDB rs0 case).
The SOPS value itself got corrupted or truncated during an edit (CRLF/BOM issues — see
docs/playbooks/sops-windows-crlf.md).
Why this playbook exists (#3506):MONGODB_RS0_ADMIN_PASSWORD drifted out of sync with the
live bms-2 rs0 PRIMARY value and was only discovered mid-task, when issue #3468 blocked on a
failing db.auth(). Nothing in the monitoring stack caught it proactively before a worker hit
it. scripts/check-sops-credential-liveness.sh + the SopsCredentialAuthFailed alert close that
gap — a dead credential now pages instead of silently blocking the next task that needs it.
How to confirm it
Check which credential fired. The alert’s key label names the probe (e.g. rs0-admin,
rs0-prometheus) — the probe table is defined at the top of
scripts/check-sops-credential-liveness.sh (DEFAULT_PROBES), which also records which SOPS
file/key each probe checks.
Distinguish “credential dead” from “detector dead”. Two alerts cover this, so the alert name
already tells you which case you are in: SopsCredentialAuthFailed fires on == 0 (a probe ran,
auth failed — the key label names the credential); SopsCredentialLivenessStopped fires on
absent() (the metric is missing entirely — cron stopped running, or the textfile collector
isn’t being scraped). Check whether the metric exists at all on
https://prometheus.vps-i1.infra.zintegrowana.online/graph?g0.expr=p24_sops_credential_valid:
Present, value 0 → the credential itself is failing live auth. Continue to step 3.
Absent entirely → the cron job or textfile collector broke. Check
/var/log/p24-sops-credential-liveness.log (or the cron’s own log target) on vps-i1 and the
node_exporter textfile collector directory (/var/lib/node_exporter/textfile_collector on
vps-i1) for a sops_credential_liveness_<key>.prom file. Fix the cron/collector, not the
credential.
Re-run the read-only probe manually (never rotate before confirming). Run it from the
repo root — the built-in probes name their SOPS files relative to the repo root
(secrets/mongodb-bms.env.sops), and the vps-i1 cron cds into {{ repo_dir }} before invoking
the script for exactly this reason (#3506). The script also self-anchors its probe paths to its
own repo root, so it works from any CWD, but running from the root keeps the manual step matching
the cron:
# On vps-i1 (or any host with the age key) — safe, read-only, prints only pass/fail# (#5066: the vps-i1 SOPS recipient key lives at /home/claude-runner/.age/p24-infra-keys.txt,# not /root/.age/p24-infra-keys.txt — that path is never provisioned by any ansible role or# .sops.yaml recipient entry. root can read the claude-runner key fine.)cd /opt/p24-infraSOPS_AGE_KEY_FILE=/home/claude-runner/.age/p24-infra-keys.txt scripts/check-sops-credential-liveness.sh
# Windows dev machine — run via Git Bash / WSL (the script is POSIX shell)cd C:\code_2026\p24-infra$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"bash scripts/check-sops-credential-liveness.sh
Confirms the failure is real and current, not a stale metric.
Step-by-step fix
This playbook does not perform the rotation itself — it only gets you to the right
credential-specific runbook quickly, per the read-only/no-write safety rule the detector itself
follows (see the script’s own SAFETY comments).
Identify the exact credential and its rotation playbook. For the two built-in probes:
rs0-admin / rs0-prometheus → docs/playbooks/mongodb-credential-rotation.md
(§SOPS File Map lists all mirror files; §Step 0b requires verifying admin_backup works
BEFORE touching the main admin password — this is the exact class of self-lockout that
caused incident #3494).
Determine which side is stale — SOPS or the live server (this is usually the real
question, not “what should the new password be”):
If the live server was rotated out-of-band (recovery, manual reset), re-sync SOPS to the
live value.
If SOPS was rotated but the live change never landed (partial rollout, a mirror file
missed), re-apply the live change or re-distribute SOPS to the missed file(s).
Dispatch the actual credential operation per role boundaries — this playbook’s role
(sys-admin/whoever is triaging the alert) does not write SOPS. Follow
docs/playbooks/secret-manager-request.md / spawn a secret-manager agent per
CLAUDE.md §Role Enforcement.
After the fix lands, re-run the probe to confirm p24_sops_credential_valid{key="..."}
returns to 1 before considering the incident closed. Do not rely on the next scheduled cron
run alone — verify immediately.
Log the incident in docs/secrets-rotation-log.md, same as any other rotation.
Escalation
Symptom
Action
Alert fires but the manual re-run (step 3 above) shows AUTH_OK
Stale alert — check Alertmanager for: 2h window and Prometheus scrape freshness; the underlying credential is fine, no rotation needed.
SopsCredentialLivenessStopped firing (metric absent) and the cron job is confirmed running
Check the node_exporter textfile collector directory exists and is being scraped — same three-directory fleet variance documented in scripts/check-env-file-perms.sh.
Rotation attempted but self-locks the credential (no working fallback)
This is exactly incident #3494 — follow docs/playbooks/mongodb-credential-rotation.md’s Double-Lockout Recovery procedure (MongoDB) or the credential’s own rotation playbook’s equivalent.
Same key alerts repeatedly right after a rotation
A partial multi-file SOPS update — re-check every mirror file listed for that credential (see the credential’s own rotation playbook’s file map) hash-matches the live value.
SopsCredentialLivenessStopped firing after a vps-i1 reboot/OOM-recovery incident and crontab -l | grep sops-credential-liveness on vps-i1 comes back empty (no #Ansible: p24-check-sops-credential-liveness marker line at all)
This is #5066’s root cause, not a one-off — a reboot/reset can silently drop ansible-managed crontab entries without dropping the whole crontab (other #Ansible: entries can survive fine), and nothing re-applies the vps-i1 playbook automatically after a host recovery. Confirm with ssh root@217.154.82.162 crontab -l, then either re-run ansible-playbook ansible/playbooks/vps-i1.yml --tags sops-credential-liveness or manually re-add the line from ansible/playbooks/vps-i1.yml’s job: field (see its cron task for the exact command — includes the correct SOPS_AGE_KEY_FILE path). Also check test -f /home/claude-runner/.age/p24-infra-keys.txt on the same host — a host recovery/reset is exactly the kind of event that can also lose an out-of-band-provisioned file. Add a step to every vps-i1 recovery runbook (vps-i1-outage.md, vps-i1-oom-outage.md): after docker compose up -d, also verify this cron and the age key file, not just the containers.
Prevention
Never rotate a credential across multiple SOPS mirror files in separate sessions/PRs — do
it in one pass, per the credential’s own rotation playbook, so a partial update can’t
reintroduce this alert immediately after “fixing” it.
Add a probe for every new Tier-1 credential that guards live-service access (cluster-admin
equivalents, monitoring-scrape users) by appending one line to DEFAULT_PROBES in
scripts/check-sops-credential-liveness.sh — do not hand-roll a one-off check elsewhere.
Keep the probe interval conservative (hourly at most, per the script’s own SAFETY notes) —
this detector holds credentials that can trip auth-rate protections if probed too aggressively.
Run the detector from one host only (vps-i1) — do not deploy it fleet-wide; a monitoring
script holding cluster-admin-adjacent credentials is itself a risk surface.