Secrets Management — p24-infra Credential Playbook

PRIMARY REFERENCE: docs/playbooks/secret-manager.md That document is the single source of truth for all secret operations. Use /role-secret-manager in any p24-infra session to load the full role.

STOP — read this first: All p24-infra secrets are in SOPS only (secrets/*.env.sops in git, encrypted with age). Infisical CE has been fully decommissioned (2026-06-21). Do NOT use Infisical for any secrets. Full SOPS guide: secrets-sops-age.md

Overview — Single-Tier SOPS System

p24-infra uses a single credential store for all secrets:

Tier	Tool	What it covers	Source of truth
All secrets	SOPS + age	All infra and app credentials	`secrets/*.env.sops` in git (encrypted)

Infisical CE was decommissioned 2026-06-21. All projects (monitoring, n8n-bms4, vps-h1, bms-servers, art-agency, radekkonarski-brand, brandpilot) are now in SOPS. GitHub Actions CI secrets remain in GH Secrets directly.

Where Each Secret Lives

Secret category	SOPS file	On-server path
Monitoring stack (Prometheus, Grafana, Alertmanager, exporters)	`secrets/monitoring.env.sops`	`/opt/p24-infra/monitoring/.env` on vps-i1
n8n + bms-4 services	`secrets/n8n-bms4.env.sops`	`/opt/p24-infra/bms-4/.env` on bms-4
vps-h1 services (Traefik, general AI worker — WAHA decommissioned 2026-07-02)	`secrets/vps-h1.env.sops`	`/root/.env` on vps-h1
BMS-1/2/3 root + MongoDB passwords	`secrets/bms-servers.env.sops`	Not deployed to disk — operator decrypt on demand
GitHub Actions CI/CD	GH Secrets (`radieu/p24-infra`)	`${{ secrets.KEY_NAME }}` in workflows
Art Agency (Supabase, PayPal, KDP, Drive)	`secrets/art-agency.env.sops`	`Art-Agency/.env.local` (decrypt from SOPS)
Personal brand (LinkedIn, HeyGen, n8n)	`secrets/radekkonarski-brand.env.sops` (in radekkonarski-personal-brand repo)	`radekkonarski-personal-brand/.env.local` fallback
BrandPilot	`secrets/brandpilot.env.sops` (in p24-infra repo)	`brandpilot/.env.local` fallback
Developer local fallback	`d:\code_2026\p24-infra\.env.local`	Not committed — may be stale

Key	Fingerprint
Developer (local)	`age1utmdm6cj2ray70g5nnupza8uvyuren26xk6a6nd4a58arax5xa7qaax7al`
CI (`AGE_KEY_GHA` GH Secret)	`age1quhdtumn09l4muh7vznh8sfxnlkm4h8p33md3s9yrsse0sggefcq5ac3ek`

dev-laptop (lap1) — local age key, not a CI push target (issue #2945)

dev-laptop / lap1 (the Acer Aspire F5-573G queue worker) is intentionally not a secrets-sync.yml push target for role credentials or any other secret. It holds its own age key (generated 2026-06-30 at /home/radieu/.age/p24-infra-keys.txt) and decrypts SOPS files it needs directly, locally — the same way the developer workstation does. This was the accepted resolution of GAP-7 (2026-07-06 secrets-sync audit): dev-laptop is a workstation-class host that already needs an age key to do any SOPS work, so pushing a second plaintext copy of every credential to it via CI would add credential material at rest on the least-controlled host to save a decrypt step it can already perform. The push model exists so that servers need not hold an age key; that rationale does not apply to a workstation.

Do not re-file this as an audit “gap.” It is a deliberate design decision, recorded here so a future secrets-sync audit does not re-recommend adding a sync-lap1 push job.

One exception — the five role-*.env.sops files. dev-laptop’s key is a recipient of the general 6-recipient SOPS files but not of secrets/role-{sys-admin,secret-manager,dev-coder,dev-reviewer,dev-tester}.env.sops. Plan #4556 Phase 3 (merged 2026-07-29) narrowed those five files to developer + CI (AGE_KEY_GHA) only, so a local sops -d of a role-*.env.sops file on lap1 will correctly fail with “no matching recipient.” A lap1 worker that genuinely needs role credentials is therefore the one case still open — its only architecturally-sound path is a future CI-mediated sync-lap1 push job (deferred; it depends on lap1 reachability + at least one live lap1 worker). Full detail and the deferred-work criteria live in docs/playbooks/lap1-worker-credentials.md.

How to Decrypt Locally

Prerequisites: sops and age installed; developer private key at ~/.age/p24-infra-keys.txt.

export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"
 
# Print all secrets for a stack:
sops --decrypt --input-type dotenv --output-type dotenv secrets/monitoring.env.sops
 
# Read a single key:
sops --decrypt --input-type dotenv --output-type dotenv secrets/monitoring.env.sops | grep SMTP_PASSWORD
 
# Open in editor for inspection (read-only intent — use sops edit to change):
sops --decrypt secrets/monitoring.env.sops

Never paste decrypted output into chat. Reference variable names only.

How to Add a New Secret

Open the relevant SOPS file in your editor via sops (requires the developer age key):

export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"
sops secrets/monitoring.env.sops
# Add: MY_NEW_VAR=<value>
# Save and exit — sops re-encrypts automatically

Reference the new variable in the relevant docker-compose.yml service block:
```
environment:
  - MY_NEW_VAR=${MY_NEW_VAR}
```

Commit and push to main (or merge via PR):

git add secrets/monitoring.env.sops
git commit -m "chore: add MY_NEW_VAR to monitoring secrets"
git push origin main

secrets-sync.yml auto-triggers and SCPs the decrypted .env to the server. Watch it:
```
gh run watch --repo radieu/p24-infra
```

Restart only the affected container(s):

ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose up -d --no-deps <service>"

Update .env.local on the developer workstation if the variable is also needed locally.

How to Rotate an Existing Secret

Edit the SOPS file (same as adding a new secret — step 1 above). Change the value, save, exit.

Commit and push. CI auto-deploys the new value to the server.

git add secrets/monitoring.env.sops
git commit -m "chore: rotate SMTP_PASSWORD"
git push origin main
gh run watch --repo radieu/p24-infra

Verify the rotated credential works on the target service.
Revoke the old value at the source (provider dashboard, DB command, etc.) only after the new value is confirmed live.

Append a row to docs/secrets-rotation-log.md:

| 2026-XX-XX | SMTP_PASSWORD | monitoring | Scheduled rotation | radieu |

Key Backup and age Keypair Management

Developer private key

The developer private key (~/.age/p24-infra-keys.txt) is the local master decryption key.

Back it up encrypted in KeePass or an offline password manager immediately after generation.
Without it, SOPS files can still be decrypted by CI (AGE_KEY_GHA), but you lose local sops edit access.
Never commit it to git. Never paste it into chat.

Rotating the age keypairs

Do this only if a key is suspected compromised, or on the annual rotation schedule.

Generate a new developer keypair:

age-keygen -o ~/.age/p24-infra-keys-new.txt
# Note the public key printed to stdout

Generate a new CI keypair:

age-keygen -o ~/.age/p24-infra-ci-keys-new.txt

Update the GH Secret with the new CI private key:

gh secret set AGE_KEY_GHA --body "$(cat ~/.age/p24-infra-ci-keys-new.txt)" --repo radieu/p24-infra

Update .sops.yaml with both new public keys (replace age1... values for developer and ci recipients).
Re-encrypt all SOPS files with the new keys (old key still works until re-keyed).

Not sops updatekeys — it is broken on dotenv *.env.sops files in SOPS 3.9.1 and no flag fixes it (issue #4601). Use the no-op rewrite, which re-encrypts under whatever recipients the matching .sops.yaml creation_rule now lists and verifies the resulting recipient count:
```
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
Get-ChildItem secrets\*.env.sops | ForEach-Object {
  .\scripts\sops-set.ps1 -SopsFile $_.FullName -RekeyOnly
}
```
See playbooks/sops-edit-operations.md §“sops updatekeys is unusable on dotenv files”.

Verify decrypt works with the new key:

SOPS_AGE_KEY_FILE=~/.age/p24-infra-keys-new.txt \
  sops --decrypt secrets/monitoring.env.sops | wc -l

Shred the old keys:

shred -u ~/.age/p24-infra-keys.txt ~/.age/p24-infra-ci-keys.txt
mv ~/.age/p24-infra-keys-new.txt ~/.age/p24-infra-keys.txt

Commit .sops.yaml and all re-keyed SOPS files, then push.

Emergency Recovery

CI is down but keys are intact

SSH to the target server and manually SCP the decrypted file:

export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"
 
# Decrypt locally to a temp file, SCP, then shred
sops --decrypt --input-type dotenv --output-type dotenv secrets/monitoring.env.sops > /tmp/monitoring.env
scp /tmp/monitoring.env root@217.154.82.162:/opt/p24-infra/monitoring/.env
shred -u /tmp/monitoring.env
 
# Restart affected services on the server
ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose up -d"

Repeat for bms-4 (secrets/n8n-bms4.env.sops → /opt/p24-infra/bms-4/.env) and vps-h1 (secrets/vps-h1.env.sops → /root/.env).

Both age keys are lost (worst case)

If both the developer key AND AGE_KEY_GHA are simultaneously lost:

SSH to each server and read the live plaintext .env (this is the recovery source):

ssh root@217.154.82.162 "cat /opt/p24-infra/monitoring/.env"
ssh root@54.36.123.110 "cat /opt/p24-infra/bms-4/.env"
ssh root@72.60.32.61 "cat /root/.env"

Do NOT paste these values into chat — save them to local files only.

Generate new age keypairs (developer + CI) following the key rotation steps above.

Re-create the SOPS files from the recovered .env files:

export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"
sops --encrypt --input-type dotenv --output-type dotenv \
  --age age1<developer-pubkey>,age1<ci-pubkey> \
  /tmp/monitoring.env > secrets/monitoring.env.sops
# Repeat for n8n-bms4 and vps-h1

Commit, push, verify CI can decrypt.
Shred all local temp files.

What NOT to Do

Never commit a plaintext .env file — always use sops --encrypt first.
Never hardcode credential values in docker-compose.yml, scripts, or n8n node parameters — use env var references.
Never paste decrypted secret values into chat or GitHub issues/PRs.
Never store any secrets in Infisical CE — it was decommissioned 2026-06-21; use SOPS for everything.
Never generate a secret and store it only in .env.local — it will be lost on rotation.
Never skip the docs/secrets-rotation-log.md entry after a rotation.

p24-infra Docs

Explorer

secrets-management

Secrets Management — p24-infra Credential Playbook

Overview — Single-Tier SOPS System

Where Each Secret Lives

dev-laptop (lap1) — local age key, not a CI push target (issue #2945)

How to Decrypt Locally

How to Add a New Secret

How to Rotate an Existing Secret

Key Backup and age Keypair Management

Developer private key

Rotating the age keypairs

Emergency Recovery

CI is down but keys are intact

Both age keys are lost (worst case)

What NOT to Do

Graph View

Table of Contents

Backlinks

p24-infra Docs

Explorer

secrets-management

Secrets Management — p24-infra Credential Playbook

Overview — Single-Tier SOPS System

Where Each Secret Lives

age public keys (safe to share)

dev-laptop (lap1) — local age key, not a CI push target (issue #2945)

How to Decrypt Locally

How to Add a New Secret

How to Rotate an Existing Secret

Key Backup and age Keypair Management

Developer private key

Rotating the age keypairs

Emergency Recovery

CI is down but keys are intact

Both age keys are lost (worst case)

What NOT to Do

Graph View

Table of Contents

Backlinks