p24-infra Docs

sops-windows-crlf

8 min read

Playbook: SOPS File Corruption on Windows (CRLF / BOM)

Systematic fix — already in place

.gitattributes enforces text eol=lf on secrets/*.env.sops. Git normalizes any CRLF to LF on every git add, so even if a Windows write produces CRLF, it is fixed before the commit lands. This is the primary protection — it works regardless of write method or OS.

secrets/*.env.sops text eol=lf

The previous entry was -text (binary) which disabled normalization — that was wrong. SOPS files are YAML-format text that requires LF. The fix was applied 2026-06-24.

The remaining defense-in-depth rules (use WriteAllText($false), run canary decrypt) protect against catching corruption before git add rather than relying solely on git.

.gitattributes eol=lf does NOT catch every CRLF case — see Failure mode 2 below. It normalizes the line endings of the .sops file structure. It cannot touch a \r that was inside the plaintext at encryption time — that byte is encrypted into the value’s ciphertext, so the .sops file has zero CR bytes yet every decrypted value ends in \r. The canary decrypt also passes (no parse error). This is silent.

Failure mode 2: CRLF baked into encrypted VALUES (silent — no parse error)

Discovered on #2238 (WEBHOOK_SECRET 401). Distinct from Failure mode 1 above:

Failure mode 1 (CR in file)Failure mode 2 (CR in values)
Where the \r lives.sops file line structureinside each encrypted value
CR bytes in .sops file> 00
sops --decrypterrors: parsing time "...Z\x0d"succeeds — no error
.gitattributes eol=lf catches ityesno
Symptomdecrypt/deploy fails loudlysecret silently mismatches downstream (e.g. 401)

Root cause: the plaintext fed to sops --encrypt had CRLF line endings (Windows temp edit file, or a CRLF source .env). SOPS encrypts each value verbatim, so a trailing \r is baked into every value. When the value is later piped to a consumer (printf '%s' "$val" | wrangler secret put, Docker env, etc.) the \r rides along and the remote value no longer byte-matches what the other side sends → silent auth failure.

Detect (Linux — the only reliable check)

# Count VALUES (not file bytes) ending in CR. ANY > 0 = corruption, even if the file is clean.
sops -d --input-type dotenv --output-type dotenv secrets/whatsup.env.sops | grep -acP '\r$'
# List the affected key NAMES (no values printed):
sops -d --input-type dotenv --output-type dotenv secrets/whatsup.env.sops \
  | grep -aP '\r$' | sed -E 's/=.*/  <-- trailing CR/'

Fix (Linux — strip CR from plaintext, re-encrypt with the file’s recipients)

sops 3.9.x loads .sops.yaml by CWD and refuses --age/SOPS_AGE_RECIPIENTS when a config is present but the input path doesn’t match a creation rule. Two ways around it:

export SOPS_AGE_KEY_FILE=/home/claude-runner/.age/p24-infra-keys.txt   # worker path
umask 077
TB=$(mktemp); TA=$(mktemp); ENC=$(mktemp)
sops -d --input-type dotenv --output-type dotenv secrets/whatsup.env.sops > "$TB"
tr -d '\r' < "$TB" > "$TA"                       # strip ONLY \r — never touch \n or content
# Recipients = the 4 age keys from .sops.yaml (developer + CI + vps-i1 + bms-4 runner).
RECIPS="age1...dev,age1...ci,age1...i1,age1...bms4"
( cd /tmp && SOPS_AGE_RECIPIENTS="$RECIPS" sops --encrypt \
    --input-type dotenv --output-type dotenv "$TA" > "$ENC" )   # run from /tmp: no .sops.yaml
# Verify BEFORE moving into place: 0 CR values, key count unchanged, all 4 recipients present.
sops -d --input-type dotenv --output-type dotenv "$ENC" | grep -acP '\r$'   # expect 0
mv "$ENC" secrets/whatsup.env.sops
shred -u "$TB" "$TA"                              # destroy plaintext temps

Never sops --encrypt ... > secrets/whatsup.env.sops directly — the shell truncates the target file before sops runs, so a sops failure leaves the file empty (recover with git checkout -- secrets/whatsup.env.sops). Always encrypt to a temp, verify, then mv.

After fixing SOPS, re-push every consumer that was deployed from the corrupted value (e.g. wrangler secret put for the CF Worker) — fixing SOPS alone does not update an already-deployed secret.

What triggers this problem (historical / if gitattributes is ever absent)

Any time a SOPS-encrypted .env.sops file is written on Windows using PowerShell’s native file-writing cmdlets (> redirect, Out-File, Set-Content, Add-Content) AND the .gitattributes text eol=lf rule is not in place.

  • > and Out-File write CRLF line endings. SOPS stores a timestamp in the encrypted header; when it re-reads that timestamp the \r causes: Error: parsing time "2026-06-24T17:49:44Z\x0d": extra text: "\x0d"
  • Out-File -Encoding utf8 (PowerShell 5.1 default) also writes a BOM (\xef\xbb\xbf). When the BOM lands at the start of a plaintext env file that is then encrypted, the first key name gets a hidden BOM prefix and is never matched by grep or shell sourcing.

Both bugs are invisible in text editors. The file looks correct but SOPS (or Docker env sourcing) silently fails.

How to confirm it

# CRLF symptom -- parse error
sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops 2>&1
# Output contains: parsing time "...Z\x0d": extra text: "\x0d"
 
# MAC failure symptom -- file was modified after encryption (different corruption)
# Output contains: cipher: message authentication failed
 
# Check for CRLF in a SOPS file
$bytes = [System.IO.File]::ReadAllBytes("C:\code_2026\p24-infra\secrets\monitoring.env.sops")
$crCount = ($bytes | Where-Object { $_ -eq 13 }).Count
Write-Host "CR bytes: $crCount"   # anything > 0 = CRLF corrupted

Step-by-step fix

1. Fetch the corrupted file from the branch

git fetch origin <branch-name>
git show origin/<branch-name>:secrets/monitoring.env.sops | Set-Content $env:TEMP\monitoring-test.env.sops -Encoding UTF8

2. Strip CR characters and test decryption

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$testPath = "$env:TEMP\monitoring-test.env.sops"
 
# Get file from branch and strip CR
$rawBytes = git show origin/<branch-name>:secrets/monitoring.env.sops
$content = ($rawBytes -join "`n").Replace("`r", "")
[System.IO.File]::WriteAllText($testPath, $content, [System.Text.UTF8Encoding]::new($false))
 
# Verify decryption works
$result = sops --decrypt --input-type dotenv --output-type dotenv $testPath 2>&1
if ($LASTEXITCODE -eq 0) { Write-Host "Decrypt OK after CR strip" }
else { Write-Host "Still failing: $result" }

3. If decrypt succeeds — fix the file in a worktree and push

# Create worktree for the branch
git worktree add C:\tmp\fix-sops-crlf <branch-name>
 
# Fix all SOPS files that have CRLF
foreach ($file in Get-ChildItem "C:\tmp\fix-sops-crlf\secrets\*.env.sops") {
  $raw = git -C C:\code_2026\p24-infra show "origin/<branch-name>:secrets/$($file.Name)"
  $fixed = ($raw -join "`n").Replace("`r", "")
  [System.IO.File]::WriteAllText($file.FullName, $fixed, [System.Text.UTF8Encoding]::new($false))
  Write-Host "Fixed $($file.Name)"
}
 
# Verify all decrypt cleanly
foreach ($file in Get-ChildItem "C:\tmp\fix-sops-crlf\secrets\*.env.sops") {
  sops --decrypt --input-type dotenv --output-type dotenv $file.FullName | Out-Null
  if ($LASTEXITCODE -eq 0) { Write-Host "OK: $($file.Name)" }
  else { Write-Host "STILL BROKEN: $($file.Name)" }
}
 
# Commit and push
Set-Location C:\tmp\fix-sops-crlf
git add secrets\*.env.sops
git commit -m "fix: strip CRLF corruption from SOPS files (re-encode LF-only)"
git push origin <branch-name>

4. If MAC failure (not just CRLF)

MAC failure means the file was modified after encryption — the encrypted content no longer matches the MAC. CRLF-stripping will not fix this; the file must be reconstructed.

Recovery source — in priority order:

  1. Live server .env file (already deployed, same as what was encrypted):
    • vps-i1: ssh root@217.154.82.162 'cat /opt/p24-infra/monitoring/.env'
    • bms-4: ssh root@54.36.123.110 'cat /opt/p24-infra/bms-4/.env'
  2. .env.local on the developer workstation (may be stale)
  3. On-server .env.bak files

Reconstruct from live server:

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
# Capture live env (pipe through a filter -- never display in chat)
$liveEnv = ssh root@217.154.82.162 'cat /opt/p24-infra/monitoring/.env'
$plainText = ($liveEnv -join "`n") + "`n"
[System.IO.File]::WriteAllText("$env:TEMP\monitoring-reconstruct.env", $plainText, [System.Text.UTF8Encoding]::new($false))
# Copy to secrets/ (must match path_regex for SOPS to pick up the right age keys)
Copy-Item "$env:TEMP\monitoring-reconstruct.env" "C:\code_2026\p24-infra\secrets\monitoring-reconstruct.env.sops"
$enc = sops --encrypt --input-type dotenv --output-type dotenv "C:\code_2026\p24-infra\secrets\monitoring-reconstruct.env.sops"
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\monitoring.env.sops",
  ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
# Canary
sops --decrypt --input-type dotenv --output-type dotenv "C:\code_2026\p24-infra\secrets\monitoring.env.sops" | Out-Null
if ($LASTEXITCODE -eq 0) { Write-Host "Reconstructed and verified OK" }
[System.IO.File]::Delete("C:\code_2026\p24-infra\secrets\monitoring-reconstruct.env.sops")
[System.IO.File]::Delete("$env:TEMP\monitoring-reconstruct.env")

Prevention — ALWAYS use this write pattern on Windows

Preferred: sops --output bypasses PowerShell entirely (no string capture)

# BEST: SOPS writes the encrypted file itself in binary mode — no PowerShell string pipeline
$TEMP_PLAIN = "secrets\monitoring-edit.env.sops"   # plaintext temp (matches path_regex)
$TEMP_ENC   = "secrets\monitoring-enc-tmp.env.sops" # encrypted temp before moving into place
 
# Write plaintext with WriteAllText, then encrypt via --output (not stdout capture)
[System.IO.File]::WriteAllText($TEMP_PLAIN, $content, [System.Text.UTF8Encoding]::new($false))
sops --encrypt --input-type dotenv --output-type dotenv --output $TEMP_ENC $TEMP_PLAIN
 
# Canary on the temp, then atomic move — production file is never touched if encrypt fails
sops --decrypt --input-type dotenv --output-type dotenv $TEMP_ENC | Out-Null
if ($LASTEXITCODE -ne 0) { Remove-Item $TEMP_PLAIN, $TEMP_ENC -Force; throw "SOPS corrupt" }
Move-Item $TEMP_ENC "secrets\monitoring.env.sops" -Force
Remove-Item $TEMP_PLAIN -Force

Fallback: if stdout capture is unavoidable, use WriteAllText on the captured output

# CORRECT: LF-only, no BOM
[System.IO.File]::WriteAllText($path, ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
 
# WRONG: adds CRLF
$enc > secrets\monitoring.env.sops
Out-File -FilePath secrets\monitoring.env.sops
 
# WRONG: adds BOM (corrupts first key name in plaintext files)
Out-File -FilePath secrets\monitoring.env.sops -Encoding utf8
Set-Content secrets\monitoring.env.sops
 
# ALSO WRONG for plaintext temp files:
sops --decrypt ... | Out-File secrets\monitoring-edit.env.sops -Encoding utf8
Add-Content secrets\monitoring-edit.env.sops "NEW_KEY=value"   # CRLF
Set-Content secrets\monitoring-edit.env.sops $lines             # BOM

Mandatory canary after every SOPS write:

sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS corrupt -- do NOT git add" }
Write-Host "Canary OK"

Escalation path

If the fix doesn’t work after two attempts:

  1. Discord alert via P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL
  2. GitHub issue in radieu/p24-infra with label human-action
  3. Fall back to on-server .env.bak as the authoritative source for reconstruction

Graph View

Backlinks