p24-infra Docs

stale-env-local-after-sops-rotation

3 min read

Playbook: Stale .env.local After a SOPS Rotation

Trigger

A local dev server (any repo using the secrets/<name>.env.sops → .env.local pattern) throws auth errors against a service whose credential was recently rotated in SOPS — e.g. MongoDB MongoServerError: Authentication failed., a Supabase 401, or an n8n webhook signature mismatch. The credential value in the workstation’s .env.local no longer matches the live server-side value because SOPS was updated (rotation, exposure incident, new key added) but .env.local was never regenerated.

First observed: 2026-08-04, et-operational-platform, PINBOX24_MONGODB_URI — rotated 2026-08-02 via issue #5079 (credential exposed in a worker chat transcript, see #5078). The rotation and its live-server “Phase B” write both completed and the issue closed same day, but this workstation’s .env.local was generated before that and never re-synced — the dev session had no way to know a rotation had happened.

Confirm

  1. Identify which SOPS file backs the failing credential (see the project’s CLAUDE.md → Secrets section for the secrets/<name>.env.sops → key map).
  2. Check whether that file has a commit newer than .env.local’s mtime: 
    # SOPS file's last relevant commit
cd C:\code_2026\p24-infra
git log -1 --format="%h %ad" --date=iso -- secrets/<name>.env.sops
 
# .env.local's last write time
(Get-Item "C:\code_2026\<repo>\.env.local").LastWriteTime
    If the commit date is newer than the file’s LastWriteTime, .env.local is stale.
  3. Confirm the specific credential is actually the cause (don’t guess): reproduce the failure and match the error to the connection using that credential — e.g. a Mongo driver “Authentication failed” implicates *_MONGODB_URI, a Supabase 401 implicates the anon/service-role key, an n8n webhook 401 implicates the shared HMAC secret.

Fix

Regenerate .env.local from the current SOPS value and restart the dev process — module-level process.env.X reads (e.g. a MongoDB client singleton built at import time) do not pick up a live env reload; the Node process must restart.

powershell scripts/decrypt-et-op-env.ps1   # project-specific decrypt script — see repo CLAUDE.md

Then kill and restart the dev server (/restart-dev in et-operational-platform, or the project’s equivalent). If a port’s listener PID has already detached from the harness’s tracked background task (task shows “completed” but the port is still listening), find and kill it directly:

netstat -ano | Select-String ":3000\s"
Stop-Process -Id <pid> -Force

Escalation

  • If the regenerated .env.local still fails auth, the live server-side credential may not match SOPS yet (an in-progress rotation’s “Phase B” not applied) — check the rotation-log / the relevant [SECRET-REQUEST] issue for phase status before assuming the local fix is wrong.
  • If there’s no local p24-infra clone, or sops/age key access is unavailable, escalate via /request-sm (ad-hoc) or a [SECRET-REQUEST] issue per docs/playbooks/secret-manager-request.md.

Prevention

et-operational-platform now runs a predev npm hook (scripts/check-env-staleness.js) that compares .env.local’s mtime against the last commit touching secrets/et-operational-platform.env.sops in the local p24-infra clone, and prints a non-blocking warning + fix command when stale. It fails open (silently skips) if the p24-infra clone, git, or .env.local aren’t present — safe in CI and on other machines.

To roll the same pattern out to another project:

  1. Copy scripts/check-env-staleness.js, updating SOPS_RELATIVE_PATH to that project’s secrets/<name>.env.sops.
  2. Add "predev": "node scripts/check-env-staleness.js" to package.json scripts.
  3. Note the mechanism in that repo’s CLAUDE.md Secrets section.

This is best-effort, not a guarantee — it only compares against the local clone’s git history, so a workstation that hasn’t git pulled p24-infra recently won’t see the newest rotation either. It catches the common case (dev forgot to re-sync after being told a rotation landed) rather than every possible staleness path.

Graph View

Backlinks