Plan: monitoring.env.sops — Role-Based Secret Reorganisation

Context

monitoring.env.sops currently holds 89 keys serving two unrelated consumers:

The monitoring Docker Compose stack on vps-i1 (/opt/p24-infra/monitoring/.env)
The claude-runner worker process on all three servers (/opt/p24-infra/{server}/.env)

This creates an “admin keys in monitoring” anti-pattern: Grafana password breach → attacker also gets SSH private key, Supabase admin, CF DNS edit, Vercel deployment token, GitHub write access.

The approved policy is:

Monitoring = read-only observability connectors only (Grafana auth, Alertmanager, exporters)
Secrets Manager role = master credentials (admin APIs for rotation — Supabase admin, Vercel admin, Mailgun admin, Sentry)
Sys Admin role = worker operational credentials (SSH for server ops, GH token, queue Supabase, Discord error reporting)
Services = own credentials only (W3 uses W3 DB user, W4 uses W4 DB user, et-op has dedicated Pinbox account)

Root cause: one SOPS file serves two deployment targets that have completely different trust requirements. Fix: split by deployment target, assign each key to its correct layer.

Tracking issue: #3178 Related: #3171 (W3+W4 CI/CD — shares secrets-sync.yml, combined PR-D)

Layered Architecture

Layer 0: Administration (developer-only, never deployed by CI)
  administration.env.sops  →  server root passwords, MongoDB admin, OVH write-scope keys

Layer 1: Roles (worker agent credentials, deployed to /opt/p24-infra/{server}/role-{role}.env)
  role-secret-manager.env.sops  →  management API tokens (admin rotation capability)
  role-sys-admin.env.sops       →  SSH key, operational GitHub/Supabase tokens
  role-dev-coder.env.sops       →  GitHub PAT for code push only

Layer 2: Worker queue (deployed to /opt/p24-infra/{server}/.env — the main worker env)
  worker-queue.env.sops  [NEW]  →  Supabase queue connection, GH token, Discord errors

Layer 3: Services (deployed to each service's runtime location, read-scope)
  monitoring.env.sops           →  observability stack ONLY (shrunk from 89 → ~50 keys)
  cloudflare.env.sops  [NEW]    →  all CF API tokens (DNS + Workers deploy)
  traccar.env.sops     [NEW]    →  Traccar + MySQL credentials
  whatsup.env.sops              →  p24-wa-a credentials (add sync job — currently orphan)
  pinbox24-w3/w4.env.sops       →  no change, already per-service ✓
  et-operational-platform.env.sops  →  no change (PINBOX24_MONGODB_URI already dedicated) ✓

SOPS File Changes

1. `secrets/worker-queue.env.sops` — CREATE NEW

Keys moved FROM monitoring.env.sops:

VPS_SSH_PRIVATE_KEY             ← claude-admin SSH key (workers SSH to other servers)
SUPABASE_URL
SUPABASE_SERVICE_KEY            ← queue read/write; intentionally separate from SERVICE_ROLE_KEY
                                   even if current values are identical — scope will diverge in Phase 2
GH_TOKEN                        ← worker PR creation
QUEUE_API_KEY
QUEUE_API_URL
P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL
P24_DISCORD_INFRA_WEEKLY_WEBHOOK_URL
P24_INFRA_MEZMO_SERVICE_KEY     ← workers query Mezmo logs

Destination per server (CRITICAL — each server has different worker env path):

Server	Worker env path	Source SOPS	Notes
vps-h1	`/opt/p24-infra/vps-h1/.env`	worker-queue.env.sops	Currently populated by sync-vps-h1 “Extract worker keys” step
bms-4	`/home/claude-runner/.env`	worker-queue.env.sops	NOT `/opt/p24-infra/bms-4/.env` (that’s n8n stack)
vps-i1	`/opt/p24-infra/vps-i1/.env`	worker-queue.env.sops	vps-i1 disabled as worker currently — deploy for future use
dev-laptop	`/opt/p24-infra/dev-laptop/.env`	worker-queue.env.sops	Variable uptime — SSH from CI may fail (use `

sync-worker-queue job deploys to all four paths. New runner servers added here on provisioning.

2. `secrets/role-sys-admin.env.sops` — EXPAND

Add from worker-queue.env.sops (sys-admin jobs have access to worker-queue keys via role sourcing):

VPS_SSH_PRIVATE_KEY             ← ops SSH (user confirmed: sys-admin needs this)

Note: other worker-queue keys (Supabase, GH, Discord) are sourced from .env before role selection in spawn-worker.sh — they stay in worker-queue.env.sops, not role files.

Stays as-is: ROLE_SYS_ADMIN_GITHUB_CLASSIC_PAT, ROLE_GIT_SIGNING_KEY

3. `secrets/role-secret-manager.env.sops` — EXPAND

Move from monitoring.env.sops:

SUPABASE_SERVICE_ROLE_KEY       ← admin (bypasses RLS) — rotation use only
VERCEL_TOKEN
VERCEL_ADMIN_TOKEN

Remove duplicates (already in role-secret-manager, wrongly also in monitoring):

MAILGUN_ADMIN_API_KEY           ← already in role-secret-manager ✓ remove from monitoring
OPENAI_ADMIN_KEY                ← already in role-secret-manager ✓ remove from monitoring
SENTRY_AUTH_TOKEN               ← already in role-secret-manager ✓ remove from monitoring

4. `secrets/cloudflare.env.sops` — CREATE NEW

Move from monitoring.env.sops:

CF_API_TOKEN
CF_EDIT_ALL_ZONES_API_TOKEN
CF_WORKERS_API_TOKEN
CF_ZONE_ID
CF_ZONE_ID_RADEKKONARSKI
CF_RADEKKONARSKI_DNS_TOKEN
CLOUDFLARE_TOKEN_ZINTEGROWANA

Used by: developer locally (DNS ops, wrangler deploy), CI/CD wrangler steps. NOT deployed to any server at runtime. Auto-triggered on push.

5. `secrets/traccar.env.sops` — CREATE NEW

Move from monitoring.env.sops:

MYSQL_ROOT_PASSWORD             ← Traccar DB root (vps-i1 Traccar MySQL)
TRACCAR_ADMIN_KEY
TRACCAR_FORWARD_TOKEN
TRACCAR_GW_ADMIN_KEY
TRACCAR_PASSWORD

Deployed to: /opt/p24-infra/vps-i1/traccar/.env

Phase 2: add TRACCAR_MONITORING_TOKEN (new read-only token) → monitoring.env.sops exporter

6. `secrets/whatsup.env.sops` — ADD SYNC JOB

Already has correct keys. Remove from monitoring.env.sops (duplicates):

SUPABASE_WAP_ANON_KEY           ← already in whatsup.env.sops
WAP_OPENAI_KEY_MINI             ← already in whatsup.env.sops

Add sync-whatsup job to secrets-sync.yml: deploy to vps-h1 as /opt/p24-infra/vps-h1/whatsup.env

7. `secrets/administration.env.sops` — EXPAND

Move from monitoring.env.sops (developer-only, never deployed by CI):

DISCORD_BOT_TOKEN               ← bot OAuth token (not Alertmanager webhook)
DISCORD_SERVER_ID
discord_radieu_password
RADEK_BRAND_RESEND_API_KEY      ← belongs in radekkonarski-brand secrets
EXTERNAL_DB_SYM_KEY             ← no active code consumer found; TTL 2026-09-07 then remove

8. `secrets/n8n-bms4.env.sops` — ADD missing ATRAX keys

ATRAX_* are used exclusively by n8n workflows on bms-4 (fleet updates, ecodriving, reports). They are already PARTIALLY in n8n-bms4: ATRAX_CLIENT_ID/SECRET/USERNAME/PASSWORD, N8N_ATRAX_REPORT_SECRET.

Add the 3 missing keys to n8n-bms4.env.sops (copy value from monitoring, then remove from monitoring):

ATRAX_AUTH_STRING               ← currently only in monitoring — move here
ATRAX_SCOPE                     ← currently only in monitoring — move here
ATRAX_WEBAPP_INSTANCE_ID        ← currently only in monitoring — move here

Remove from monitoring: all 9 ATRAX_* keys + N8N_ATRAX_REPORT_WEBHOOK_URL (there is no ATRAX billing/cost Prometheus exporter — confirmed by codebase search)

9. `secrets/monitoring.env.sops` — SHRINK (~89 → ~42 keys)

KEEP (observability stack only):

# Grafana/Prometheus/Loki internal
GRAFANA_ADMIN_USER, GRAFANA_ADMIN_PASSWORD, GF_RENDERING_RENDERER_TOKEN
PROMETHEUS_BASIC_AUTH_*, LOKI_BASIC_AUTH_PASSWORD, STATUS_BASIC_AUTH_*

# Alertmanager notifications (write = their job)
SMTP_HOST/USER/PASSWORD, MAILGUN_API_KEY, MAILGUN_EU_DOMAIN, MAILGUN_REPORT_FROM
EMAIL_SENDER_API_KEY, EMAIL_SENDER_URL
DISCORD_P24_ISSUES_WEBHOOK_URL, p24-infra-1h-check-claude-discord

# Wasabi — read-only cost-exporter key (ListBucket + GetBucketLocation ONLY)
WASABI_ACCESS_KEY, WASABI_SECRET_KEY
# IAM policy must restrict to: s3:ListBucket + s3:GetBucketLocation on [p24-infra, thanos-*] buckets
# Verify in Phase 2 — if key has broader scope, rotate to limited IAM user

# Wasabi — Thanos Sidecar write key (ONE bucket only: thanos-metrics)
P24_INFRA_WASABI_ACCESS_KEY, P24_INFRA_WASABI_SECRET_KEY
# IAM policy must restrict to: s3:PutObject + s3:GetObject + s3:ListBucket on thanos-metrics bucket ONLY
# NOT production files management — if current key has broader scope, create dedicated key in Phase 2

# Supabase DB (pg-stats-exporter) — Phase 2: replace with monitoring_reader account
SUPABASE_URL, SUPABASE_DB_HOST/PORT/USER/PASSWORD/SSLMODE, SUPABASE_GRAFANA_PASSWORD

# Mezmo ingestion
MEZMO_INGESTION_KEY

# Exporter read-only API keys (billing/infrastructure metrics)
OVH_APP_KEY/SECRET/CONSUMER     ← read-only billing scope (cost-exporter); different from ovh-api.env.sops
SYS_APP_KEY/SECRET/CONSUMER     ← read-only billing (SoYouStart cost-exporter)
OVH_ENDPOINT, IONOS_HOSTING_TOKEN_*
N8N_BMS4_API_KEY, N8N_BMS4_BASE_URL, N8N_CLOUD_API_KEY
NEXCON_API_KEY
QUEUE_API_KEY, QUEUE_API_URL    ← monitoring-watchdog reads queue depth (read)
CRON_SECRET

# REMOVE in Phase 2 when monitoring_reader Supabase account is created:
# SUPABASE_SERVICE_KEY — currently needed by pg-stats-exporter fallback path

# KEYS REMOVED from monitoring (vs current state):
# ATRAX_* (9 keys) + N8N_ATRAX_REPORT_* (2) → n8n-bms4.env.sops
# VPS_SSH_PRIVATE_KEY, SUPABASE_SERVICE_KEY, GH_TOKEN, QUEUE_API_*, P24_DISCORD_*, P24_INFRA_MEZMO → worker-queue.env.sops
# CF_* (7) → cloudflare.env.sops
# MYSQL_ROOT_PASSWORD, TRACCAR_* (5) → traccar.env.sops
# SUPABASE_WAP_ANON_KEY, WAP_OPENAI_KEY_MINI → remove (duplicates in whatsup.env.sops)
# MAILGUN_ADMIN_API_KEY, OPENAI_ADMIN_KEY, SENTRY_AUTH_TOKEN → remove (duplicates in role-secret-manager)
# SUPABASE_SERVICE_ROLE_KEY, VERCEL_TOKEN, VERCEL_ADMIN_TOKEN → role-secret-manager.env.sops
# DISCORD_BOT_TOKEN, DISCORD_SERVER_ID, discord_radieu_password, RADEK_BRAND_RESEND_API_KEY, EXTERNAL_DB_SYM_KEY → administration.env.sops

secrets-sync.yml Changes

New jobs to add

sync-worker-queue — deploys worker-queue.env.sops to .env on vps-i1, vps-h1, bms-4

Runs on: self-hosted bms4
Trigger: push to secrets/worker-queue.env.sops or target=worker-queue
Destination: /opt/p24-infra/{server}/.env (restart claude-runner or signal workers)
SSH via: VPS_ROOT_SSH_KEY (GH Secret, same pattern as other jobs)

sync-cloudflare — deploys cloudflare.env.sops to developer tooling / CF Worker env

Trigger: push to secrets/cloudflare.env.sops or target=cloudflare
Does NOT need to deploy to a server — developer uses locally via SOPS decrypt
BUT: wrangler CI jobs in meta-dispatcher/p24-auth-worker need CF tokens → inject as GH Secrets via sync-gh-secrets job

sync-traccar — deploys traccar.env.sops to vps-i1

Destination: /opt/p24-infra/vps-i1/traccar/.env

sync-whatsup — deploys whatsup.env.sops to vps-h1

Destination: /opt/p24-infra/vps-h1/whatsup.env
⚠️ CORRECTION (2026-07-08, #2943): this vps-h1 destination is stale. p24-wa-a does NOT run on vps-h1 (WAHA was decommissioned there 2026-07-02) — the live consumer is the Cloudflare Worker p24-wa-app-api (bindings via wrangler secret put) plus the p24-wa-a Vercel web app. An scp of whatsup.env to vps-h1 would be consumed by nothing. The correct automation is a wrangler/Vercel sync job (needs a CF token with Workers-Scripts-Edit scope); until it exists the file is synced manually and is intentionally excluded from secrets-sync.yml. Do NOT build the vps-h1 variant.

Modify existing jobs

sync-vps-h1 — REMOVE the “Extract worker keys” step (lines 239-257 in secrets-sync.yml):

BEFORE: step greps monitoring.env.sops for SUPABASE_*, P24_INFRA_MEZMO_SERVICE_KEY, VPS_SSH_PRIVATE_KEY → /tmp/worker-keys.env → deployed to /opt/p24-infra/vps-h1/.env
AFTER: step removed entirely; sync-worker-queue deploys worker-queue.env directly to /opt/p24-infra/vps-h1/.env
Verify: no SCP of worker-keys.env in sync-vps-h1 after change

sync-bms-4 — TWO changes:

REMOVE the copy step (lines 636-637): install … /opt/p24-infra/bms-4/.env /home/claude-runner/.env — n8n env no longer propagates to worker
REMOVE extraction of P4_INFRA_MEZMO_SERVICE_KEY from monitoring.env.sops (line 517-525) — move to worker-queue.env.sops

The n8n stack .env at /opt/p24-infra/bms-4/.env is UNCHANGED in content and path — only the copy-to-claude-runner step is removed
AFTER: sync-worker-queue deploys worker-queue.env to /home/claude-runner/.env on bms-4

sync-vps-i1 — NO CHANGE to existing job needed:

sync-vps-i1 deploys monitoring.env to /opt/p24-infra/monitoring/.env only (no worker .env today)
VPS_SSH_PRIVATE_KEY is used inline (line 130-131) for the SCP SSH key — this stays as-is
sync-worker-queue adds NEW deployment to /opt/p24-infra/vps-i1/.env on vps-i1

auto-trigger paths + target dispatch enum (lines ~15-27 in secrets-sync.yml):

Add to target enum: worker-queue, traccar, whatsup (cloudflare = manual-only, no auto-trigger)
REMOVE from exclusion list: !secrets/whatsup.env.sops (line 26) — currently excluded; must be removed when sync-whatsup job is added so changes auto-deploy
New SOPS files (worker-queue.env.sops, traccar.env.sops) are NOT in exclusion list → they auto-trigger when created. Ensure sync jobs exist in secrets-sync.yml BEFORE merging the new SOPS files to main (separate PRs).
Add !secrets/cloudflare.env.sops to exclusion list (CF tokens only needed by developer locally + wrangler CI, not by server sync)

Phase 2 (service accounts — requires human action in target systems)

System	Action	New key in SOPS
Supabase	Create `monitoring_reader` PostgreSQL user with `pg_monitor` role	`SUPABASE_MONITORING_DB_USER/PASSWORD` in monitoring.env.sops → remove `SUPABASE_SERVICE_KEY` from monitoring
Traccar	Create read-only API token in Traccar admin panel	`TRACCAR_MONITORING_TOKEN` in monitoring.env.sops
OVH/SoYouStart	Verify billing-only scope on current OVH_APP_KEY (already read-only per code comment)	rename to `OVH_BILLING_APP_KEY` to make scope explicit
Wasabi cost-exporter	Verify `WASABI_ACCESS_KEY` IAM = `s3:ListBucket + GetBucketLocation` only — background, non-blocking	rotate if broader; document in credential-design
Wasabi Thanos	Verify `P4_INFRA_WASABI_ACCESS_KEY` IAM = write to thanos-metrics bucket ONLY — background, non-blocking	rotate to limited-scope key if admin-scope; rename `WASABI_THANOS_ACCESS_KEY` in SOPS

MongoDB Account Distribution (design + Phase 2 implementation)

Required MongoDB accounts with their SOPS location and server targets:

Account	Purpose	SOPS file	Deployed to
`p24-admin`	Cluster admin, rs0 management	`administration.env.sops`	Developer only — never CI-deployed
`p24-monitoring`	Read-only Prometheus scrape (future metrics)	`monitoring.env.sops`	vps-i1 monitoring stack
`w3.pinbox24.com`	W3 app read/write to W3 collections	`pinbox24-w3.env.sops` → `V32_NEW_MONGODB_URI`	bms-1 (v32-prod, s3-v32-prod)
`w4.pinbox24.com`	W4 app read/write to W4 collections	`pinbox24-w4.env.sops` → `V42_NEW_MONGODB_URI`	bms-1 (v42-prod, s3-v42-prod)
`et-operational-platform`	et-op read access to W4 collections	`et-operational-platform.env.sops` → `PINBOX24_MONGODB_URI`	Vercel (et-op)

Rotation protocol (secrets-manager):

W4 app password changed → update V42_NEW_MONGODB_URI in pinbox24-w4.env.sops → secrets-sync sync-bms-1 → docker-compose up -d
et-op access changed → update PINBOX24_MONGODB_URI in et-operational-platform.env.sops → secrets-sync sync-et-operational-platform → Vercel redeploy
p24-admin → update administration.env.sops + manual mongosh on rs0 primary

Phase 2 verification: confirm PINBOX24_MONGODB_URI in et-op uses dedicated et-operational-platform account (not w4.pinbox24.com). If shared → create separate MongoDB user before proceeding.

Supabase Key Clarification

SUPABASE_SERVICE_KEY → worker-queue.env.sops — queue worker read/write to p24_queue table. Same sb_secret value as SERVICE_ROLE_KEY today; will diverge when restricted Supabase user created for queue.
SUPABASE_SERVICE_ROLE_KEY → role-secret-manager.env.sops — admin bypass RLS, rotation operations only.
SUPABASE_SERVICE_ROLE_KEY → et-operational-platform.env.sops — et-op server-side operations (existing, no change).
Vercel et-op: 2 projects (production = master, staging) — confirmed same Supabase project + same SERVICE_ROLE_KEY.

Merged Sequencing with issue #3171 (W3/W4 CI/CD)

Both plans share secrets-sync.yml. Audit result: only one real conflict — line 15 target enum (both add new values). Resolution: combine ALL secrets-sync.yml changes into one PR (PR-D).

PR sequence (both tracks, no conflicts)

PR-A  Merge PR #3163 → main (nginx fix + PM2 fork mode — prerequisite for everything)

PR-B  SOPS-only, no secrets-sync changes (parallel):
      - Track A Phase 0: add keys to pinbox24-backends.env.sops + pinbox24-w4.env.sops
      - Track B Step 1: add ATRAX_AUTH_STRING/SCOPE/WEBAPP_INSTANCE_ID to n8n-bms4.env.sops

PR-C  Track A Phase 1: GitLab CI changes (mailgun + s3-v2 — no secrets-sync.yml changes)

PR-D  COMBINED secrets-sync.yml PR (ALL changes from both tracks at once):
      Track A:
        - sync-bms-1 extension (bms1-all target, full W3/W4 env delivery)
      Track B:
        - Add sync-worker-queue, sync-cloudflare, sync-traccar, sync-whatsup jobs
        - Modify sync-vps-h1: remove "Extract worker keys" step (lines 239-257)
        - Modify sync-bms-4: remove copy to /home/claude-runner/.env (lines 635-637)
        - Remove P4_INFRA_MEZMO_SERVICE_KEY extraction from monitoring in sync-bms-4
      Shared:
        - target enum line 15: add bms1-all + worker-queue + traccar + whatsup
        - Remove !secrets/whatsup.env.sops from exclusion list (line 26)
        - Add !secrets/cloudflare.env.sops to exclusion list

PR-E  Track B: Create new SOPS files (worker-queue.env.sops, cloudflare.env.sops, traccar.env.sops)
      SOPS files trigger sync jobs from PR-D automatically → verify workers still work

PR-F  Track A Phase 2: verify bms1-all sync works → all W3+W4 env files on bms-1

PR-G  Track B: Shrink monitoring.env.sops (remove ~47 keys progressively, canary before each commit)
      Sub-steps (each as canary + commit):
      5a. Remove ATRAX_* + N8N_ATRAX_REPORT_* (→ already in n8n-bms4)
      5b. Remove worker keys (VPS_SSH_PRIVATE_KEY, SUPABASE_*, GH_TOKEN, QUEUE_*, P24_DISCORD_*, MEZMO)
      5c. Remove CF_*, Traccar keys
      5d. Remove WAP duplicates (SUPABASE_WAP_ANON_KEY, WAP_OPENAI_KEY_MINI)
      5e. Remove admin dupes (MAILGUN_ADMIN, OPENAI_ADMIN, SENTRY)
      5f. Move DISCORD_BOT_TOKEN, DISCORD_SERVER_ID, etc. → administration.env.sops
      5g. Expand role-secret-manager with SUPABASE_SERVICE_ROLE_KEY, VERCEL tokens

PR-H  Track A Phases 3A+3B: W4+W3 deploy scripts cleanup (GitLab)

PR-I  Track A Phases 3C+3D: Image CI/CD (GitLab triggers)

PR-J  Track A Phase 4: Credential rotations (MAILGUN, JWT, Redis, Wasabi)

PR-K  Track A Phase 5: PM2 monitoring integration

PR-L  Track B Phase 2: Service accounts (human action: Supabase monitoring_reader, MongoDB p24-monitoring)
      Background task: Wasabi IAM scope verification

Key rule: PR-D combines ALL secrets-sync.yml changes → no merge conflict possible. Key rule: PR-E SOPS files only after PR-D merged → sync jobs already exist.

Migration Order (Track B standalone steps within PR-E to PR-G above)

n8n-bms4.env.sops: add 3 ATRAX keys → merge → verify sync-bms-4 picks them up
Create worker-queue.env.sops, cloudflare.env.sops, traccar.env.sops (in PR-E)
secrets-sync.yml combined PR-D → verify workers on bms-4 and vps-h1 still work
Progressively remove keys from monitoring.env.sops (PR-G sub-steps) — canary before each commit
Verify: sops --decrypt monitoring.env.sops | wc -l → ~42 keys

Do NOT remove key from monitoring.env.sops until the new destination is deployed and verified (PR-E merged and secrets-sync triggered).

Key Files to Modify

.github/workflows/secrets-sync.yml — add 4 jobs, modify 3 jobs, update auto-trigger paths
secrets/monitoring.env.sops — remove ~47 keys (SOPS in-place edit, Windows WriteAllText)
secrets/n8n-bms4.env.sops — add 3 ATRAX keys (ATRAX_AUTH_STRING, ATRAX_SCOPE, ATRAX_WEBAPP_INSTANCE_ID)
secrets/role-sys-admin.env.sops — add VPS_SSH_PRIVATE_KEY
secrets/role-secret-manager.env.sops — add SUPABASE_SERVICE_ROLE_KEY, VERCEL tokens
secrets/worker-queue.env.sops — create new
secrets/cloudflare.env.sops — create new
secrets/traccar.env.sops — create new
secrets/administration.env.sops — add DISCORD_BOT_TOKEN, DISCORD_SERVER_ID, discord_radieu_password, RADEK_BRAND_RESEND_API_KEY, EXTERNAL_DB_SYM_KEY (TTL 2026-09-07)
secrets/bms-servers.env.sops — remove bare mongodb_w3_app_password + mongodb_w4_app_password (reduce duplication; URI keys in per-app SOPS are source of truth)
docs/playbooks/secret-manager.md — update SOPS file map
CLAUDE.md (p24-infra) — update SOPS file map table

Verification

After each phase step:

# 1. SOPS canary — all new files decrypt cleanly
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
foreach ($f in "worker-queue","cloudflare","traccar") {
  sops --decrypt --input-type dotenv --output-type dotenv "secrets\$f.env.sops" | Out-Null
  Write-Host "$f canary: $LASTEXITCODE"
}
 
# 2. Run secrets-sync jobs
gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=worker-queue
gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=vps-i1
 
# 3. Verify workers still function on bms-4
# Check spawn-worker.sh picks up Supabase from new worker-queue.env location
 
# 4. Verify monitoring stack still runs
# ssh vps-i1: docker compose ps (all services Up)
# https://grafana.vps-i1.infra.zintegrowana.online → login
 
# 5. Verify api.w4.pinbox24.com still responds (W4 CI/CD unaffected)
1..3 | % { (Invoke-WebRequest "https://api.w4.pinbox24.com/api/i18n/langs" -UseBasicParsing).StatusCode }

Resolved Decisions

EXTERNAL_DB_SYM_KEY → archive in administration.env.sops (add 2026-07-07, delete 2026-09-07 if no consumer found)
bms-servers.env.sops mongodb_w3/w4_app_password → reduce duplication: remove bare password keys; source of truth = URI keys in per-app SOPS files (pinbox24-w3/w4.env.sops)
Wasabi IAM scope verification → background task (spawn sys-security agent to verify policies); does NOT block Phase 1 migration steps
Vercel et-op staging → confirmed SAME Supabase credentials as production (same SERVICE_ROLE_KEY)

Execution log — PR-E (2026-08-02)

State re-verified against live monitoring.env.sops before touching anything (secret-manager session, #3178). Findings and corrections applied, key NAMES only, no values shown:

monitoring.env.sops is now 99 keys, not 89. The 89 figure in this plan’s original Context section is stale (plan written 2026-07-07). ATRAX_* / N8N_ATRAX_REPORT_* keys (PR-B target, 3237, merged) are still present in monitoring.env.sops — per this plan’s own rule they were correctly left in place (only added to n8n-bms4.env.sops, not yet removed from monitoring; removal is a PR-G step, not done here).
Traccar family grew by 2 keys since this plan was written. 5148 (all merged 2026-08-02, same day as this session) added TRACCAR_ADMIN_EMAIL and split MYSQL_PASSWORD out of TRACCAR_PASSWORD. None of that work touched the SOPS file layout — it was rotation/tier/dev_r_services registration work on keys that still live in monitoring.env.sops. §5’s original 5-key list (MYSQL_ROOT_PASSWORD + 4 TRACCAR_*) is corrected to 7 keys: add TRACCAR_ADMIN_EMAIL and MYSQL_PASSWORD. traccar.env.sops (created in PR-E) uses the corrected 7-key list.
Parallel initiative 4556 already built “Layer 1: Roles” (role-sys-admin.env.sops, role-secret-manager.env.sops, etc.) independently of this plan. Its own plan doc (docs/plans/plan-4729-sops-role-project-reorg.md §0) explicitly names this plan as “Track M” and says to execute it as-is — confirmed no boundary conflict, the family split (roles vs. services vs. admin vs. products) is consistent between the two plans.
New finding — do NOT execute §3’s dedup-removal (MAILGUN_ADMIN_API_KEY, OPENAI_ADMIN_KEY, SENTRY_AUTH_TOKEN) without re-checking role-*.env.sops recipient scope first. These three keys are indeed already duplicated in role-secret-manager.env.sops (confirmed), but role-*.env.sops files had their .sops.yaml recipients narrowed to 2 (dev + CI only) by the #4556 Phase 3 work (2026-07-29) — the four host runner keys (vps-i1/bms-4/dev-laptop/vps-h1) can no longer sops -d them directly. monitoring.env.sops still uses the full 6-recipient rule. docs/playbooks/secret-manager.md §“Provider master-key coverage” documents relying on monitoring.env.sops specifically as the worker-reachable copy of MAILGUN_ADMIN_API_KEY for a standard secret-manager worker session. Removing it from monitoring.env.sops (PR-G step 5e) would silently degrade that rotation path to human-action unless/until the worker gets a plaintext-deployed role-secret-manager.env copy it can read instead (it already does, via secrets-sync.yml, at /opt/p24-infra/{host}/role-secret-manager.env — but confirm the rotation scripts actually read from that path before removing the monitoring copy). Flagged on issue #3178, not resolved here — PR-G must address this explicitly, not assume the plan’s original “safe to remove, already duplicated” reasoning still holds.
New finding — MYSQL_ROOT_PASSWORD’s value in monitoring.env.sops is currently empty (confirmed via length check, not content). Pre-existing gap, not introduced by this session. traccar.env.sops (PR-E) faithfully mirrors the empty value rather than fabricating one. Flagged for secret-manager follow-up.
VPS_SSH_PRIVATE_KEY deliberately NOT moved to worker-queue.env.sops in PR-E. It is a multi-line PEM value (root SSH key for every VPS/BMS host) and this session’s rule of never displaying secret values means the only verifiable copy method is a byte-for-byte hash comparison of an untouched string — safe for the other 8 worker-queue.env.sops keys (single-line tokens/URLs) but judged too risky to script blind for a multi-line PEM without a way to sanity check line-ending/quoting behavior went through the dotenv codec correctly. It is also already duplicated in role-secret-manager.env.sops (added by the unrelated 4733 initiative) — open question for the follow-up PR: does worker-queue.env.sops need its own third copy at all, or should the worker bootstrap source the existing role-secret-manager.env copy instead?
PR-E delivered: secrets/cloudflare.env.sops (7 keys), secrets/traccar.env.sops (7 keys, corrected list), secrets/worker-queue.env.sops (8 of 9 planned keys, VPS_SSH_PRIVATE_KEY deferred). All three canary-decrypt clean and every copied key’s SHA-256 hash was verified to match its monitoring.env.sops source line exactly (faithful copy, no corruption, no 3223-style empty-value regression). monitoring.env.sops itself is untouched — still 99 keys, canary-clean. The three new files are excluded from secrets-sync.yml’s push trigger until PR-D adds their sync jobs (see that workflow’s paths: list).
Deferred to follow-up (PR-D, PR-G, and a small VPS_SSH_PRIVATE_KEY-specific PR): creating the sync-cloudflare / sync-traccar / sync-worker-queue jobs, repointing sync-vps-h1 / sync-bms-4, verifying every consumer host still authenticates, then progressively shrinking monitoring.env.sops. None of that is done yet — do not assume PR-E implies distribution.

p24-infra Docs

Explorer

monitoring-env-sops-reorganisation

Plan: monitoring.env.sops — Role-Based Secret Reorganisation

Context

Layered Architecture

SOPS File Changes

1. `secrets/worker-queue.env.sops` — CREATE NEW

2. `secrets/role-sys-admin.env.sops` — EXPAND

3. `secrets/role-secret-manager.env.sops` — EXPAND

4. `secrets/cloudflare.env.sops` — CREATE NEW

5. `secrets/traccar.env.sops` — CREATE NEW

6. `secrets/whatsup.env.sops` — ADD SYNC JOB

7. `secrets/administration.env.sops` — EXPAND

8. `secrets/n8n-bms4.env.sops` — ADD missing ATRAX keys

9. `secrets/monitoring.env.sops` — SHRINK (~89 → ~42 keys)

secrets-sync.yml Changes

New jobs to add

Modify existing jobs

Phase 2 (service accounts — requires human action in target systems)

MongoDB Account Distribution (design + Phase 2 implementation)

Supabase Key Clarification

Merged Sequencing with issue #3171 (W3/W4 CI/CD)

PR sequence (both tracks, no conflicts)

Migration Order (Track B standalone steps within PR-E to PR-G above)

Key Files to Modify

Verification

Resolved Decisions

Execution log — PR-E (2026-08-02)

Graph View

Table of Contents

p24-infra Docs

Explorer

monitoring-env-sops-reorganisation

Plan: monitoring.env.sops — Role-Based Secret Reorganisation

Context

Layered Architecture

SOPS File Changes

1. secrets/worker-queue.env.sops — CREATE NEW

2. secrets/role-sys-admin.env.sops — EXPAND

3. secrets/role-secret-manager.env.sops — EXPAND

4. secrets/cloudflare.env.sops — CREATE NEW

5. secrets/traccar.env.sops — CREATE NEW

6. secrets/whatsup.env.sops — ADD SYNC JOB

7. secrets/administration.env.sops — EXPAND

8. secrets/n8n-bms4.env.sops — ADD missing ATRAX keys

9. secrets/monitoring.env.sops — SHRINK (~89 → ~42 keys)

secrets-sync.yml Changes

New jobs to add

Modify existing jobs

Phase 2 (service accounts — requires human action in target systems)

MongoDB Account Distribution (design + Phase 2 implementation)

Supabase Key Clarification

Merged Sequencing with issue #3171 (W3/W4 CI/CD)

PR sequence (both tracks, no conflicts)

Migration Order (Track B standalone steps within PR-E to PR-G above)

Key Files to Modify

Verification

Resolved Decisions

Execution log — PR-E (2026-08-02)

Graph View

Table of Contents

1. `secrets/worker-queue.env.sops` — CREATE NEW

2. `secrets/role-sys-admin.env.sops` — EXPAND

3. `secrets/role-secret-manager.env.sops` — EXPAND

4. `secrets/cloudflare.env.sops` — CREATE NEW

5. `secrets/traccar.env.sops` — CREATE NEW

6. `secrets/whatsup.env.sops` — ADD SYNC JOB

7. `secrets/administration.env.sops` — EXPAND

8. `secrets/n8n-bms4.env.sops` — ADD missing ATRAX keys

9. `secrets/monitoring.env.sops` — SHRINK (~89 → ~42 keys)