A credential stored in SOPS stops authenticating against the live service it is meant to
unlock — most commonly because:
The live password was rotated/reset directly on the server (or via a break-glass recovery
procedure) without SOPS being updated to match.
A partial multi-file rotation updated some SOPS mirrors of a credential but missed one
(this credential class often has 4–5 mirror copies — see
docs/playbooks/mongodb-credential-rotation.md §SOPS File Map for the MongoDB rs0 case).
The SOPS value itself got corrupted or truncated during an edit (CRLF/BOM issues — see
docs/playbooks/sops-windows-crlf.md).
Why this playbook exists (#3506):MONGODB_RS0_ADMIN_PASSWORD drifted out of sync with the
live bms-2 rs0 PRIMARY value and was only discovered mid-task, when issue #3468 blocked on a
failing db.auth(). Nothing in the monitoring stack caught it proactively before a worker hit
it. scripts/check-sops-credential-liveness.sh + the SopsCredentialAuthFailed alert close that
gap — a dead credential now pages instead of silently blocking the next task that needs it.
How to confirm it
Check which credential fired. The alert’s key label names the probe (e.g. rs0-admin,
rs0-prometheus) — the probe table is defined at the top of
scripts/check-sops-credential-liveness.sh (DEFAULT_PROBES), which also records which SOPS
file/key each probe checks.
Distinguish “credential dead” from “detector dead”. Two alerts cover this, so the alert name
already tells you which case you are in: SopsCredentialAuthFailed fires on == 0 (a probe ran,
auth failed — the key label names the credential); SopsCredentialLivenessStopped fires on
absent() (the metric is missing entirely — cron stopped running, or the textfile collector
isn’t being scraped). Check whether the metric exists at all on
https://prometheus.vps-i1.infra.zintegrowana.online/graph?g0.expr=p24_sops_credential_valid:
Present, value 0 → the credential itself is failing live auth. Continue to step 3.
Absent entirely → the cron job or textfile collector broke. Check
/var/log/p24-sops-credential-liveness.log (or the cron’s own log target) on vps-i1 and the
node_exporter textfile collector directory (/var/lib/node_exporter/textfile_collector on
vps-i1) for a sops_credential_liveness_<key>.prom file. Fix the cron/collector, not the
credential.
Re-run the read-only probe manually (never rotate before confirming). Run it from the
repo root — the built-in probes name their SOPS files relative to the repo root
(secrets/mongodb-bms.env.sops), and the vps-i1 cron cds into {{ repo_dir }} before invoking
the script for exactly this reason (#3506). The script also self-anchors its probe paths to its
own repo root, so it works from any CWD, but running from the root keeps the manual step matching
the cron:
# On vps-i1 (or any host with the age key) — safe, read-only, prints only pass/fail# (#5066: the vps-i1 SOPS recipient key lives at /home/claude-runner/.age/p24-infra-keys.txt,# not /root/.age/p24-infra-keys.txt — that path is never provisioned by any ansible role or# .sops.yaml recipient entry. root can read the claude-runner key fine.)cd /opt/p24-infraSOPS_AGE_KEY_FILE=/home/claude-runner/.age/p24-infra-keys.txt scripts/check-sops-credential-liveness.sh
# Windows dev machine — run via Git Bash / WSL (the script is POSIX shell)cd C:\code_2026\p24-infra$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"bash scripts/check-sops-credential-liveness.sh
Confirms the failure is real and current, not a stale metric.
Step-by-step fix
This playbook does not perform the rotation itself — it only gets you to the right
credential-specific runbook quickly, per the read-only/no-write safety rule the detector itself
follows (see the script’s own SAFETY comments).
Identify the exact credential and its rotation playbook. For the two built-in probes:
rs0-admin / rs0-prometheus → docs/playbooks/mongodb-credential-rotation.md
(§SOPS File Map lists all mirror files; §Step 0b requires verifying admin_backup works
BEFORE touching the main admin password — this is the exact class of self-lockout that
caused incident #3494).
Determine which side is stale — SOPS or the live server (this is usually the real
question, not “what should the new password be”):
If the live server was rotated out-of-band (recovery, manual reset), re-sync SOPS to the
live value.
If SOPS was rotated but the live change never landed (partial rollout, a mirror file
missed), re-apply the live change or re-distribute SOPS to the missed file(s).
Dispatch the actual credential operation per role boundaries — this playbook’s role
(sys-admin/whoever is triaging the alert) does not write SOPS. Follow
docs/playbooks/secret-manager-request.md / spawn a secret-manager agent per
CLAUDE.md §Role Enforcement.
After the fix lands, re-run the probe to confirm p24_sops_credential_valid{key="..."}
returns to 1 before considering the incident closed. Do not rely on the next scheduled cron
run alone — verify immediately.
Log the incident in docs/secrets-rotation-log.md, same as any other rotation.
Escalation
Symptom
Action
Alert fires but the manual re-run (step 3 above) shows AUTH_OK
Stale alert — check Alertmanager for: 2h window and Prometheus scrape freshness; the underlying credential is fine, no rotation needed.
SopsCredentialLivenessStopped firing (metric absent) and the cron job is confirmed running
Check the node_exporter textfile collector directory exists and is being scraped — same three-directory fleet variance documented in scripts/check-env-file-perms.sh.
Rotation attempted but self-locks the credential (no working fallback)
This is exactly incident #3494 — follow docs/playbooks/mongodb-credential-rotation.md’s Double-Lockout Recovery procedure (MongoDB) or the credential’s own rotation playbook’s equivalent.
Same key alerts repeatedly right after a rotation
A partial multi-file SOPS update — re-check every mirror file listed for that credential (see the credential’s own rotation playbook’s file map) hash-matches the live value.
SopsCredentialLivenessStopped firing after a vps-i1 reboot/OOM-recovery incident and the cron is missing entirely
This is #5066’s root cause, not a one-off — a reboot/reset can silently drop ansible-managed cron entries, and nothing re-applies the vps-i1 playbook automatically after a host recovery. Since #5857 (v2), this cron lives in /etc/cron.d/p24-check-sops-credential-liveness, not root’s crontab (ssh root@217.154.82.162 cat /etc/cron.d/p24-check-sops-credential-liveness — a crontab -l check will show nothing even when healthy). Confirm with the cat command above, then either re-run ansible-playbook ansible/playbooks/vps-i1.yml --tags sops-credential-liveness or manually recreate the file from ansible/playbooks/vps-i1.yml’s cron tasks (PATH line + job line). Also check test -f /home/claude-runner/.age/p24-infra-keys.txt on the same host — a host recovery/reset is exactly the kind of event that can also lose an out-of-band-provisioned file. Add a step to every vps-i1 recovery runbook (vps-i1-outage.md, vps-i1-oom-outage.md): after docker compose up -d, also verify this cron and the age key file, not just the containers.
SopsCredentialAuthFailed firing continuously for hours/days (not a single blip), cron log (/var/log/p24-sops-credential-liveness.log on vps-i1) shows could not decrypt <KEY> from secrets/... on every run, but a manual re-run of the script from an interactive shell shows AUTH_OK
This is #5857’s root cause — a third variant of the 5066 detector-environment bug class, not a real credential failure. Cron’s default PATH (/usr/bin:/bin — confirm by capturing env from a one-shot cron job) does not include /usr/local/bin, where sops lives; decrypt_key()’s command -v sops guard then fails silently on every scheduled run. Pitfall from the first fix attempt (PR #6390, rejected on review): setting job: "PATH=... cd {{ repo_dir }} && ... script.sh" does NOT work — a VAR=val prefix attaches only to the single command it prefixes (here, cd, a builtin that ignores PATH entirely), not to whatever follows &&. The script kept failing with cron’s unmodified PATH even after that “fix” landed on the live host; it looked fixed only because the verification method (env -i PATH=... sh -c '...') implicitly set PATH for the whole subshell instead of faithfully reproducing VAR=val cmd1 && cmd2 scoping. Always verify a cron-job fix by running the exact job string through sh -c with cron’s actual minimal env, not a wrapper that accidentally fixes the scoping bug for you. The durable fix (v2) uses a cron_file:-scoped env: yes PATH entry (own top-of-file line, applies to every job in that file) instead of an inline prefix — matches the existing pattern in bms-4.yml for n8n-bms4-execution-cleanup/p24-check-env-file-perms.
Prevention
Never rotate a credential across multiple SOPS mirror files in separate sessions/PRs — do
it in one pass, per the credential’s own rotation playbook, so a partial update can’t
reintroduce this alert immediately after “fixing” it.
Add a probe for every new Tier-1 credential that guards live-service access (cluster-admin
equivalents, monitoring-scrape users) by appending one line to DEFAULT_PROBES in
scripts/check-sops-credential-liveness.sh — do not hand-roll a one-off check elsewhere.
Keep the probe interval conservative (hourly at most, per the script’s own SAFETY notes) —
this detector holds credentials that can trip auth-rate protections if probed too aggressively.
Run the detector from one host only (vps-i1) — do not deploy it fleet-wide; a monitoring
script holding cluster-admin-adjacent credentials is itself a risk surface.