Infisical CE — Operations Workbook

DECOMMISSIONED 2026-06-21 — Infisical CE containers have been stopped and removed from vps-i1 as part of the SOPS+age migration (PR #806). This document is retained as a historical reference only. All secrets are now managed via SOPS+age — see docs/secrets-management.md and docs/secrets-sops-age.md.

Service: infisical-ce
Host: vps-i1 (217.154.82.162) — Docker Compose stack monitoring/
Version: infisical/infisical:v0.161.0
UI: https://infisical.vps-i1.infra.zintegrowana.online
Compliance: dev_r_services.service_name = 'infisical-ce'


0. Admin credentials

Where they live

LocationKey
.env.local on developer workstationINFISICAL_ADMIN_EMAIL, INFISICAL_ADMIN_PASSWORD
Infisical p24-monitoring projectsecrets named INFISICAL_ADMIN_EMAIL, INFISICAL_ADMIN_PASSWORD

The admin account is the first account created during initial setup. It has full org-owner access.

Reset procedure (when password is forgotten)

The “Forgot password” email flow requires SMTP to be configured in the Infisical container. If SMTP is not wired up, reset via Postgres directly:

# 1. Get a bcrypt hash for your new password (run locally)
python3 -c "import bcrypt; print(bcrypt.hashpw(b'YOUR_NEW_PASSWORD', bcrypt.gensalt(12)).decode())"
 
# 2. Update the hash in Infisical's Postgres
ssh root@217.154.82.162
docker exec -it monitoring-infisical-postgres-1 psql -U infisical -d infisical \
  -c "UPDATE users SET password_hash = 'HASH_FROM_STEP_1' WHERE email = 'radieu@gmail.com';"
 
# 3. Verify login at https://infisical.vps-i1.infra.zintegrowana.online

Note: bcrypt is available via pip install bcrypt if not already installed.


1. Overview

Infisical Community Edition is the secrets vault for the entire p24-infra ecosystem. All other services (monitoring stack, n8n, audit-engine, GitHub Actions workflows) draw credentials from it via Machine Identity Universal Auth.

Three containers form the stack:

ContainerImageRole
monitoring-infisical-1infisical/infisical:v0.161.0API + web UI
monitoring-infisical-postgres-1postgres:16.9-alpineSecret storage (encrypted)
monitoring-infisical-redis-1redis:7.4.2-alpineSession cache

Projects (Infisical slugs):

Project slugUsed byMI env var prefix
p24-monitoring-f5devps-i1 monitoring stack + audit-engineINFISICAL_MI_P24_MONITORING_*
n8n-bms4-ywf-z (latest)bms-4 n8n stackINFISICAL_MI_N8N_BMS4_*
vps-h1-ntjx (latest)vps-h1 WAHA stackINFISICAL_MI_VPS_H1_*
bms-servers-mq-he (latest)BMS server agentsINFISICAL_MI_BMS_SERVERS_*
github-actions-s-m15All GitHub Actions workflowsINFISICAL_MI_GITHUB_ACTIONS_*
art-agencyArt Agency portal (Supabase, PayPal, KDP, Google Drive)INFISICAL_MI_ART_AGENCY_*
radekkonarski-brandRadek personal brand (LinkedIn, HeyGen, Wasabi, n8n, Anthropic)INFISICAL_MI_RADEK_BRAND_*
brandpilot-ovb-0BrandPilot AI brand automation (GitHub: p24-infra/brandpilot)INFISICAL_MI_BRANDPILOT_*

2. Backup

What is backed up

The nightly ionos backup (scripts/backup-ionos.sh, cron 30 2 * * *) includes:

ArtifactPath in archivePurpose
infisical_postgres.dump$WORK/infisical_postgres.dumpFull database dump (pg_dump -Fc custom format)
infisical-keys/infisical-env.txt$WORK/infisical-keys/infisical-env.txtENCRYPTION_KEY + JWT secrets extracted from container
monitoring.env$WORK/monitoring.envFull monitoring stack env (all credentials)

Destination: s3://ecotrans-backups/vps-i1/<YYYY-MM-DD>/backup.tar.zst.age
Encryption: age (recipient = AGE_PUBKEY from /root/.backup-env)
Retention: 30 days daily (managed by backup-common.sh parent job)

Critical: ENCRYPTION_KEY

The INFISICAL_ENCRYPTION_KEY env var is the AES master key that encrypts every secret value in the database. Without it, the database dump cannot be decrypted. It is captured two ways:

  1. infisical-keys/infisical-env.txt — extracted from the running container
  2. monitoring.env — full env file snapshot

Both are inside the age-encrypted archive, so the age private key (stored externally) is the single recovery prerequisite.

Age private key storage

The age private key is not stored on any VPS. It must be kept in an offline/out-of-band secret store (e.g. Bitwarden vault, paper backup, GPG-encrypted USB). Without it the backup archive cannot be decrypted.

To check which public key is used:

grep AGE_PUBKEY /root/.backup-env

3. Restore procedure

Prerequisites

  • Age private key available locally
  • Wasabi access credentials (WASABI_BACKUP_ACCESS_KEY / WASABI_BACKUP_SECRET_KEY)
  • A running Docker host with monitoring/docker-compose.yml deployed (vps-i1 or replacement)

Step 1 — Download and decrypt archive

# Install age and aws CLI if not present
apt-get install -y age awscli zstd
 
# Download the backup
aws --endpoint-url https://s3.eu-central-1.wasabisys.com \
    s3 cp "s3://ecotrans-backups/vps-i1/<DATE>/backup.tar.zst.age" /tmp/backup.tar.zst.age
 
# Decrypt and extract
age -d -i /path/to/age-private-key /tmp/backup.tar.zst.age \
  | zstd -d \
  | tar -xf - -C /tmp/restore/

Step 2 — Recover INFISICAL_ENCRYPTION_KEY

# The key is in either of:
cat /tmp/restore/infisical-keys/infisical-env.txt   # extracted vars
grep INFISICAL_ENCRYPTION_KEY /tmp/restore/monitoring.env
grep INFISICAL_JWT /tmp/restore/monitoring.env

Note these values — you will need them in Step 4.

Step 3 — Start a fresh Infisical stack with the original keys

Ensure monitoring/.env on the recovery host contains the original values from /tmp/restore/monitoring.env. The critical vars are:

INFISICAL_ENCRYPTION_KEY=<original value>
INFISICAL_AUTH_SECRET=<original value>
INFISICAL_JWT_AUTH_SECRET=<original value>
INFISICAL_JWT_SIGNUP_SECRET=<original value>
INFISICAL_JWT_REFRESH_SECRET=<original value>
INFISICAL_JWT_MFA_OTP_SECRET=<original value>
INFISICAL_JWT_PROVIDER_AUTH_SECRET=<original value>
INFISICAL_DB_PASSWORD=<original value>

Start containers (postgres only first):

cd /opt/p24-infra/monitoring
docker compose up -d infisical-postgres infisical-redis
# Wait for postgres healthy
docker compose ps

Step 4 — Restore the database dump

# Copy dump into a temp location accessible by the container
docker cp /tmp/restore/infisical_postgres.dump monitoring-infisical-postgres-1:/tmp/restore.dump
 
# Drop and recreate the database
docker exec -it monitoring-infisical-postgres-1 psql -U infisical -c "DROP DATABASE IF EXISTS infisical;"
docker exec -it monitoring-infisical-postgres-1 psql -U infisical -c "CREATE DATABASE infisical;"
 
# Restore
docker exec -it monitoring-infisical-postgres-1 \
  pg_restore -U infisical -d infisical --no-owner --no-acl /tmp/restore.dump
 
# Clean up
docker exec monitoring-infisical-postgres-1 rm /tmp/restore.dump

Step 5 — Start Infisical and verify

docker compose up -d infisical
sleep 30
docker compose ps   # infisical should be healthy
 
# Smoke test: health endpoint
curl -k https://infisical.vps-i1.infra.zintegrowana.online/api/status
# Expected: {"date":"...","message":"OK","..."}

Step 6 — Test Machine Identity auth

# From vps-i1 — test p24-monitoring MI
source /opt/infisical/monitoring.env
curl -sk -X POST "$INFISICAL_URL/api/v1/auth/universal-auth/login" \
  -H 'Content-Type: application/json' \
  -d "{\"clientId\":\"$INFISICAL_CLIENT_ID\",\"clientSecret\":\"$INFISICAL_CLIENT_SECRET\"}" \
  | python3 -c "import sys,json; t=json.load(sys.stdin); print('OK' if t.get('accessToken') else 'FAIL')"

Step 7 — Restart all dependent services

# Restart monitoring stack to pick up Infisical secrets
cd /opt/p24-infra/monitoring
./start.sh   # re-runs infisical run -- docker compose up -d
 
# On bms-4: restart n8n stack
ssh root@54.36.123.110 "cd /root && bash /root/start.sh"

4. Encryption key rotation

Machine Identity client secrets do not expire. Rotate manually if a MI is compromised:

  1. Log in to Infisical UI → Settings → Machine Identities → select identity → Regenerate secret
  2. Update the credential in the affected service’s env:
    • vps-i1 monitoring: update INFISICAL_CLIENT_SECRET in /opt/infisical/monitoring.env and restart
    • bms-4 n8n: update INFISICAL_CLIENT_SECRET in /opt/infisical/n8n.env (canonical MI bootstrap file) and restart
  3. Update .env.local on the developer workstation

INFISICAL_ENCRYPTION_KEY rotation requires decrypting and re-encrypting all secrets — do not rotate unless the key is confirmed compromised. Contact the Infisical CE docs for the rotation procedure.


4b. App-secrets backup policy (ENCRYPTION_KEY + JWT secrets)

The following vars are Infisical CE app secrets — they are not managed inside Infisical itself (chicken-and-egg). They must be kept in sync across three locations whenever rotated.

SecretRoleRecoverable without it?
INFISICAL_ENCRYPTION_KEYAES master key — encrypts every secret value in the DBNo — DB dump is unreadable
INFISICAL_AUTH_SECRETGeneral auth signingYes — regenerate, all sessions invalidated
INFISICAL_JWT_AUTH_SECRETJWT auth tokensYes — regenerate, all sessions invalidated
INFISICAL_JWT_SIGNUP_SECRETSignup flow JWTsYes
INFISICAL_JWT_REFRESH_SECRETRefresh token signingYes
INFISICAL_JWT_MFA_OTP_SECRETMFA OTP signingYes
INFISICAL_JWT_PROVIDER_AUTH_SECRETOAuth provider tokensYes
INFISICAL_DB_PASSWORDPostgres passwordYes — can reset via ALTER ROLE

Three mandatory backup locations

Every one of the above must exist in all three of these locations at all times:

LocationHow to update
d:\code_2026\p24-infra\.env.local (developer workstation)Edit directly after rotation
GitHub Secrets — radieu/p24-infraWrite-Output $newVal | gh secret set KEY --repo radieu/p24-infra
monitoring/.env on vps-i1 (runtime source)Already updated by the rotation; verify with grep KEY /opt/p24-infra/monitoring/.env

INFISICAL_ENCRYPTION_KEY must also be recorded in the nightly backup. It is captured automatically from the running container by scripts/backup-ionos.sh — confirm backup ran successfully after rotation.

Rotation checklist

When rotating any of these secrets:

[ ] 1. Update monitoring/.env on vps-i1
[ ] 2. Restart Infisical container: docker compose -f /opt/p24-infra/monitoring/docker-compose.yml up -d --no-deps infisical
[ ] 3. Verify Infisical healthy: curl -sk https://infisical.vps-i1.infra.zintegrowana.online/api/status
[ ] 4. Update .env.local on developer workstation
[ ] 5. Sync to GH Secrets: Write-Output $newVal | gh secret set INFISICAL_XXX --repo radieu/p24-infra
[ ] 6. Append rotation entry to docs/secrets-rotation-log.md
[ ] 7. ENCRYPTION_KEY only: wait for next nightly backup and confirm BackupStale alert is green

secrets-rotation-log.md entry format (for these keys)

| YYYY-MM-DD | INFISICAL_ENCRYPTION_KEY | <reason> | <rotator> | yes — .env.local + GH Secret + vps-i1 monitoring/.env + nightly backup confirmed |
| YYYY-MM-DD | INFISICAL_JWT_*           | <reason> | <rotator> | yes — .env.local + GH Secret + vps-i1 monitoring/.env |

5. Day-to-day: adding or rotating a secret

Step-by-step

  1. Log in to the Infisical UI → select the correct project → production environment.
  2. Add or update the secret.
  3. Regenerate .env.bak on vps-i1 — this is mandatory after every secret change:
    ssh root@217.154.82.162 "bash /opt/infisical/regen-monitoring-envbak.sh"
    Expected output: 119 secrets written to /opt/p24-infra/monitoring/.env.bak
  4. Restart the affected container(s) with the new .env.bak:
    ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && \
      docker compose --env-file .env.bak up -d --no-deps <service>"
    Replace <service> with the container name, e.g. alertmanager, cost-exporter, grafana. Omit --no-deps and the service name to restart the entire stack.
  5. If the secret also belongs to another project (n8n-bms4, vps-h1, github-actions) — update it there too and trigger that stack’s regen (see project table below).

Per-project regen commands

Infisical projectHostRegen command
p24-monitoringvps-i1ssh root@217.154.82.162 "bash /opt/infisical/regen-monitoring-envbak.sh"
n8n-bms4bms-4ssh root@54.36.123.110 "bash /opt/infisical/regen-n8n-envbak.sh"
vps-h1vps-h1ssh root@72.60.32.61 "bash /opt/infisical/regen-vps-h1-envbak.sh"
github-actionsN/ARun scripts/sync-github-secrets.ps1 or push to main

All three regen scripts run automatically via daily cron (06:00/06:05/06:10 UTC). Logs: /var/log/infisical-regen.log on each host.


6. Vault coverage and exceptions

Infisical CE is the single source of truth for all p24-infra credentials. Every secret that flows into a running container or script must live in Infisical first.

What IS managed in Infisical

ProjectCovers
p24-monitoringAll monitoring stack containers (Prometheus, Grafana, Alertmanager, exporters, Caddy, pdf-service, audit-engine)
n8n-bms4n8n stack on bms-4 (n8n encryption key, Supabase, Gmail, webhook secrets)
vps-h1WAHA stack on vps-h1 (WAHA API key, HMAC secret, Traefik config)
bms-serversBMS bare-metal credentials (MongoDB admin, keyFile, prometheus user, root passwords, bms-1 Pinbox24 env)
github-actionsSecrets synced into GitHub Actions (CI/CD credentials, Vercel token, deploy keys)
art-agencyArt Agency portal (Supabase, PayPal, KDP, Google Drive — used by Art-Agency repo)
radekkonarski-brandRadek personal brand automation (LinkedIn, HeyGen, Wasabi, n8n, Anthropic — used by radekkonarski-personal-brand repo)
brandpilot-ovb-0BrandPilot AI brand automation (GitHub: p24-infra/brandpilot org repo) — MI env var prefix: INFISICAL_MI_BRANDPILOT_*

Exceptions — what is NOT in Infisical

Secret / locationWhy it’s outside InfisicalWhere it lives instead
/opt/infisical/monitoring.envBootstrap credentials for Infisical itself (chicken-and-egg: can’t fetch from Infisical to auth to Infisical)File on vps-i1, backed up in nightly Wasabi archive
/opt/infisical/n8n.envBootstrap for bms-4 n8n MI (chicken-and-egg)File on bms-4 — sourced by start.sh and regen script
/opt/infisical/vps-h1.envBootstrap for vps-h1 WAHA MIFile on vps-h1 — sourced by regen script
GH Secrets (radieu/p24-infra)GitHub Actions reads secrets directly from GH Secrets, not from Infisical at runtimeGitHub Secrets — synced from github-actions project via scripts/sync-github-secrets.ps1
BMS-1/2/3 root + MongoDB passwordsbms-1/2/3 have no Infisical agent (servers predate the rollout)Primary: bms-servers Infisical project (readable by vps-i1 MI). Backup: GH Secrets BMS_ROOT_PASSWORD, MONGODB_RS0_KEYFILE, BMS1_PINBOX24_ENV
d:\code_2026\p24-infra\.env.localDeveloper workstation local override and rotation scratch-padLocal file — not committed. Synced manually from Infisical or updated during rotation scripts
secrets/*.sops.yamlLegacy SOPS-encrypted files from before Infisical rollout (2026-06). May contain stale values.Git repo (encrypted). No longer the source of truth — migrate any remaining active secrets to Infisical and archive these files
Age private keyRequired to decrypt SOPS backup archives — cannot be stored in InfisicalBitwarden vault / offline backup (never on any VPS)
monitoring/.env.bakDerived cache generated from Infisical — not a primary storevps-i1 /opt/p24-infra/monitoring/.env.bak — regenerate after every Infisical change (see §5)

SOPS migration status

The secrets/*.sops.yaml files are legacy. After the Infisical rollout (2026-06-16), all active secrets should be in Infisical. The SOPS files are kept as a historical reference and for the age-encrypted Wasabi backup restore path. Do not add new secrets to SOPS files — add them to Infisical instead.


7. Health monitoring

Infisical is monitored via its Docker health check (wget -qO- http://localhost:8080/api/status). The Grafana monitoring stack scrapes container states via cAdvisor.

Current gaps:

  • No dedicated Prometheus exporter for Infisical secret-access metrics
  • Backup success is tracked via backup_last_success_timestamp metric on vps-i1

To check health manually:

docker exec monitoring-infisical-1 wget -qO- http://localhost:8080/api/status

8. Upgrading Infisical CE

  1. Check release notes at https://github.com/Infisical/infisical/releases
  2. Update the image tag in monitoring/docker-compose.yml
  3. Take a manual backup before upgrading:
    /opt/p24-infra/scripts/backup-ionos.sh 2>&1 | tail -5
  4. Run docker compose pull infisical && docker compose up -d infisical
  5. Verify health and smoke-test MI auth (see §7 step 6)

9. Disaster recovery checklist

#StepDone?
1Download + decrypt Wasabi backup
2Recover INFISICAL_ENCRYPTION_KEY from archive
3Restore monitoring/.env with original Infisical vars
4Start fresh postgres + redis containers
5Restore pg_dump
6Start Infisical, verify /api/status
7Test Machine Identity auth for each project
8Restart all dependent services (monitoring, n8n, audit-engine)
9Run nightly backup immediately to confirm backup pipeline works
10Update DNS if host IP changed