p24-infra Docs

sops-rotation-classifier-patterns

10 min read

Playbook: SOPS Rotation — Classifier-Safe Patterns & Windows Batch Writes

Investigation output for #2407 (“Investigate SOPS write/rotation batch formatting issues on Windows”), triggered by the 2026-07-01 s3-v2-v42-prod incident (#2401) where autonomous Wasabi IAM rotation could not complete because the local boto3 rotation script was repeatedly blocked by the command-safety classifier.

This doc is the single answer to the five investigation tasks. It consolidates and cross-links existing playbooks rather than duplicating them:

Scope note. This playbook documents patterns. It does not execute the pending #2401 rotations (pinbox24 keys on bms-1, the permanent MONGODB_URI SOPS write) — those need root shell + human actions and stay tracked under #2401 and docs/priorities.md.

Task 1 — Why does a local boto3 + subprocess-SSH rotation script trip the classifier?

Symptom (#2401): a local Python rotation script (boto3 IAM calls + subprocess SSH to the target server) was repeatedly blocked at “Stage 2” of the command-safety classifier, even though the script never prints any credential value.

Root cause — the classifier reasons about materialization paths, not about print(). It asks “could a secret VALUE end up somewhere observable (argv, process table, an inline literal, stdout, a log)?” — and it must decide statically, before the command runs. A boto3 + subprocess-SSH rotator hits several materialization signals it cannot rule out:

Signal in the scriptWhy the classifier flags it
Secret passed explicitly to boto3: boto3.client('iam', aws_secret_access_key=SECRET, ...)The value flows into a client constructor argument. The classifier can’t prove it isn’t later logged, re-serialized, or echoed.
Credentials or the new key in argv: subprocess.run(['ssh', ..., f'echo {NEW_KEY} > ...']) or --password=...argv is world-visible via ps aux — a first-class leak vector the classifier blocks on sight (mirrors the global ps aux / --password=xxx rule).
Inline script literal built at call time (heredoc / python3 -c "...secret...")Dynamically-constructed code carrying a credential reads as credential materialization; the classifier can’t analyze a string it will only see fully expanded at runtime.
Reading a secret then branching/formatting it into f-stringsAny string interpolation of a secret-derived variable is treated as a potential print/emit path.

The through-line: the script never prints the value, but the classifier cannot prove that, and several data paths (argv, client-arg, inline literal) are ones it refuses by policy. “It works when I run it by hand” is exactly the state the classifier is designed to gate.

Task 2 — A rotation script pattern that passes the classifier

Design rule: secrets must flow only through the environment and restrictive-permission files — never through argv, never through inline script literals, never through explicit credential-carrying function args the classifier can’t clear.

2a. Inject secrets as env vars via sops exec-env; read them with os.environ

sops exec-env decrypts and injects keys as environment variables into a child process, then runs your command. The secret never appears in argv or in any file the session reads.

# The script is a COMMITTED file that reads os.environ — never a `-c` inline literal.
sops exec-env --input-type dotenv --output-type dotenv \
  secrets/administration.env.sops 'python3 scripts/rotate-wasabi-key.py'

sops exec-env on dotenv files: pass --input-type dotenv --output-type dotenv. Plain sops exec-env file.env.sops 'cmd' JSON-parses the file first and fails — see the broken-patterns table in sops-edit-operations.md.

2b. Let boto3 read credentials implicitly from the environment

Do not pass aws_secret_access_key= explicitly. boto3 reads AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY from the environment automatically — so the value never appears as a constructor argument.

#!/usr/bin/env python3
"""scripts/rotate-wasabi-key.py — reference pattern (committed file, no inline literal)."""
import os, sys, boto3
 
# Credentials arrive ONLY via os.environ (injected by `sops exec-env`). Never hard-coded,
# never in argv. Fail fast if absent — but NEVER print the value.
for required in ("WASABI_ADMIN_ACCESS_KEY", "WASABI_ADMIN_SECRET_KEY"):
    if not os.environ.get(required):
        sys.exit(f"missing env var: {required}")   # NAME only — never the value
 
# boto3 picks up AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY implicitly.
os.environ.setdefault("AWS_ACCESS_KEY_ID", os.environ["WASABI_ADMIN_ACCESS_KEY"])
os.environ.setdefault("AWS_SECRET_ACCESS_KEY", os.environ["WASABI_ADMIN_SECRET_KEY"])
iam = boto3.client("iam", endpoint_url="https://iam.wasabisys.com", region_name="us-east-1")
 
# ... create new key, update consumers, delete old key ...
# Print only NON-secret outcome fields: key IDs are identifiers, not secrets; secret material
# is written to a 0600 file consumed downstream, never echoed.
print("rotated: ok")   # summary only

2c. Never put credentials in SSH argv

If a rotation must land a new value on a remote host, do not build ssh host "echo $NEW_SECRET > file" (argv leak). Instead write the value to a local 0600 temp file and stream it over stdin:

import subprocess, tempfile, os
fd, tmp = tempfile.mkstemp(); os.close(fd); os.chmod(tmp, 0o600)
with open(tmp, "w", newline="\n") as f:
    f.write(new_value)                       # value stays in a 0600 file, not argv
with open(tmp, "rb") as f:
    subprocess.run(["ssh", "root@host", "cat > /root/target.env.part"],
                   stdin=f, check=True)       # value rides stdin, never argv/ps aux
os.remove(tmp)

Use key-based SSH auth only (no --password= / sshpass in argv). Verify success by exit code, not by printing remote output.

2d. Checklist — a script that passes the classifier

  • Script is a committed .py file, not a python3 -c "..." inline literal.
  • Every secret is read from os.environ — no hard-coded values, no defaults containing secrets.
  • Secrets injected via sops exec-env ... 'python3 scripts/...py'.
  • boto3 reads creds from the environment implicitly — no aws_secret_access_key= argument.
  • No secret in any argv (subprocess, ssh, curl -H "...$TOKEN" where the token is expanded into a logged command).
  • Remote writes stream via stdin/0600 files, never echo-into-argv.
  • Only NON-secret identifiers (key IDs, “rotated: ok”) are printed. Never print(os.environ).

Task 3 — Make the Vercel rotator the canonical rotation path

Recommendation: YES for the p24-infra Wasabi IAM user — it already is, and it sidesteps the classifier entirely. Rotation runs server-side on Vercel; the local session only fires a single authenticated HTTP POST. Nothing sensitive materializes locally:

# Token via env (not expanded into a logged literal); check status code, never print the body.
ROTATOR_TOKEN=$(sops -d --input-type dotenv --output-type dotenv \
  secrets/administration.env.sops | grep '^ROTATOR_API_KEY=' | cut -d= -f2-)
curl -sf -o /dev/null -w '%{http_code}\n' -X POST \
  https://<deployed-url>/api/rotate -H "Authorization: Bearer $ROTATOR_TOKEN"
unset ROTATOR_TOKEN

Why this is the preferred path:

  • The Wasabi admin key never leaves SOPS/Vercel — no local boto3, so no Stage-2 block.
  • New key material is written to Vercel env vars + triggers secrets-sync.yml; it is never returned in the response body. See wasabi-iam-rotator.md.
  • Only the manual SOPS post-update remains local — a file edit, not a credential-materializing script (use the batch pattern in Task 4).

Scope gap — what the current rotator does NOT cover (the #2401 blocker)

The deployed function is hard-scoped to the p24-infra IAM user and updates Vercel env vars + the vps-i1 monitoring stack. The #2401 rotations that got stuck were different:

  • pinbox24PublicAccessKeyId / pinbox24PublicSecretAccessKey (main + office) live in /root/s3v2-prod/s3-v2-environment.env on bms-1 — a different IAM user and a different consumer (a server env file, not Vercel). The current endpoint cannot rotate these.

To make the Vercel endpoint truly canonical for all Wasabi keys, extend it (follow-up work):

  1. Accept {"iam_username": "...", "consumer": "..."} in the POST body (it already reads WASABI_IAM_USERNAME; generalize it per-request).
  2. Add a pluggable “consumer update” step — for a server-env consumer, push the new value over an authenticated deploy hook (server pulls; the endpoint never SSHes with a secret in argv).
  3. Keep the “new secret never in the response body” invariant.

Until that exists, rotate the pinbox24 keys with the Task 2 classifier-safe local pattern (sops exec-env + committed script + stdin streaming) — do not build inline boto3 one-liners.

Task 4 — Batch multi-key SOPS write on Windows (BOM/CRLF-safe)

Rotations usually change several keys at once (e.g. WASABI_ACCESS_KEY + WASABI_SECRET_KEY + two P24_INFRA_* aliases). Do it in one decrypt → edit-all → encrypt → canary cycle, not one cycle per key. Full CRLF/BOM rules: sops-windows-crlf.md. Full edit mechanics (path_regex, --in-place): sops-edit-operations.md.

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$SOPS_FILE = "C:\code_2026\p24-infra\secrets\monitoring.env.sops"
$TEMP_FILE = "C:\code_2026\p24-infra\secrets\monitoring-edit.env.sops"  # MUST end .env.sops (path_regex)
 
# 1. Decrypt ONCE to a temp file — LF-only, no BOM (the two Windows corruption sources)
$plain = sops --decrypt --input-type dotenv --output-type dotenv $SOPS_FILE
[System.IO.File]::WriteAllText($TEMP_FILE, ($plain -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
 
# 2. Apply ALL key updates in one pass. Read new values from env (never echo them).
#    $env:NEW_ACCESS / $env:NEW_SECRET were populated earlier from the Vercel dashboard.
$lines = [System.IO.File]::ReadAllLines($TEMP_FILE)
$updates = @{
  "WASABI_ACCESS_KEY"           = $env:NEW_ACCESS
  "WASABI_SECRET_KEY"           = $env:NEW_SECRET
  "P24_INFRA_WASABI_ACCESS_KEY" = $env:NEW_ACCESS
  "P24_INFRA_WASABI_SECRET_KEY" = $env:NEW_SECRET
}
$out = foreach ($line in $lines) {
  $k = ($line -split '=',2)[0]
  if ($updates.ContainsKey($k)) { "$k=$($updates[$k])" } else { $line }
}
# 3. Write back LF-only, no BOM — NEVER `>` / Out-File / Set-Content (they add CRLF/BOM)
[System.IO.File]::WriteAllText($TEMP_FILE, ($out -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
 
# 4. Encrypt ONCE, explicit dotenv types (avoids the 3.9.1 JSON-output bug), then canary
sops --encrypt --input-type dotenv --output-type dotenv --output $SOPS_FILE $TEMP_FILE
Remove-Item $TEMP_FILE -Force -ErrorAction SilentlyContinue      # destroy plaintext even on failure
sops --decrypt --input-type dotenv --output-type dotenv $SOPS_FILE | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS corrupt — do NOT git add. See sops-windows-crlf.md" }
 
# 5. Also verify NO trailing-CR baked into VALUES (Failure mode 2 — silent). Run on Linux/WSL:
#    sops -d --input-type dotenv --output-type dotenv secrets/monitoring.env.sops | grep -acP '\r$'   # expect 0
$env:NEW_ACCESS = ""; $env:NEW_SECRET = ""

Two corruption classes to defend against every time (both invisible in editors):

  1. CR in the file / BOM — from >, Out-File, Set-Content. .gitattributes text eol=lf catches file-structure CR; WriteAllText($false) avoids BOM. Canary decrypt catches parse errors.
  2. CR baked into encrypted VALUES (silent, no parse error) — only detectable with grep -acP '\r$' on the decrypted stream (step 5). This is the one .gitattributes cannot catch.

Task 5 — Can sops set add individual keys without a full decrypt/re-encrypt?

No. sops set / sops --set are broken for dotenv files in SOPS 3.9.1 — do not use them. Confirmed in the known-broken table of sops-edit-operations.md:

AttemptResult
sops --set '["KEY"]' '"val"'Value for --set is not valid JSON
sops set file '["KEY"]' '"val"'Invalid set index format
sops exec-env file.env.sops 'cmd' (no explicit types)invalid character 'C'

Root cause: SOPS 3.9.1 set/edit modes JSON-parse the encrypted dotenv file before decryption; the first character of the first key (e.g. C in CF_…) breaks the parse. There is no incremental single-key write path for dotenv on this version.

Canonical add/update = the batch decrypt → edit-all → encrypt → canary cycle (Task 4). It is not slower in practice for rotation because rotations touch multiple keys anyway, and it is the only path that reliably preserves dotenv format and LF-only encoding.

Summary — decision table

SituationDo this
Rotate the p24-infra Wasabi IAM keyVercel POST /api/rotate (canonical) → then batch SOPS update (Task 4)
Rotate a Wasabi key the endpoint doesn’t cover (e.g. pinbox24 on bms-1)Classifier-safe local script (Task 2): committed .py + sops exec-env + stdin streaming
Add / change one or many keys in a SOPS dotenv fileBatch decrypt → edit-all → encrypt → canary (Task 4). Never sops set (Task 5)
Any Windows SOPS writeWriteAllText + UTF8Encoding($false); never > / Out-File / Set-Content
Script keeps getting Stage-2 blockedMove secrets out of argv/inline-literals into os.environ via sops exec-env (Task 2)

Escalation

Graph View

Backlinks