n8n Credential Portability — Playbook & Standards
Covers: bms-4 n8n (queue mode, PostgreSQL, 2.26.3) Secrets management: SOPS + age — see
docs/secrets-management.mdCompanion to: n8n-operations.md
1. Honest Weak Point Analysis
Weak Point 1 (CRITICAL): N8N_ENCRYPTION_KEY is NOT passed to containers
What’s happening: N8N_ENCRYPTION_KEY is absent from every environment block in bms-4/docker-compose.yml. n8n auto-generates a random key on first start and stores it in /home/node/.n8n/config inside the n8n_data Docker volume.
Consequences:
- If
n8n_datavolume is lost → encryption key is gone → all n8n credentials become unrecoverable permanently - The restore doc already acknowledges this: “Credentials must be re-entered manually”
Fix (Phase 1): Extract the current key, add it to secrets/n8n-bms4.env.sops, and add it explicitly to all container environment blocks. Key then travels with every secrets-sync.yml deploy — volume loss no longer kills the credential store.
Weak Point 2 (MEDIUM): $env exposes infrastructure secrets to every workflow
N8N_EXPOSE_ENVIRONMENT_VARIABLES=true + N8N_BLOCK_ENV_ACCESS_IN_NODE=false is set on all containers. This means any workflow can read $env.DB_POSTGRESDB_PASSWORD, $env.REDIS_PASSWORD, $env.GITHUB_PAT_ALL_WRITES, etc.
n8n Community Edition has no selective env var filtering — it’s all-or-nothing.
Accepted risk + mitigations:
- Only n8n admin can create/import workflows (guard UI + API key)
- Never import untrusted workflow JSON
- DB + Redis are internal Docker network only — exploitation requires n8n access first
Weak Point 3 (MEDIUM): Secret values appear in execution logs for 7 days
When $env.ATRAX_PASSWORD flows through an HTTP Request node, n8n stores the evaluated value in execution history. Unlike n8n credential store (which masks values), $env values are not masked.
Mitigation already in place: EXECUTIONS_DATA_PRUNE=true + MAX_AGE=168h (7 days) — auto-purged.
Accepted trade-off: Zero credential re-entry on migration > 7-day log exposure window for this infra use case.
Weak Point 4 (MEDIUM): No Docker volume backup
Neither n8n_data (encryption key) nor n8n_postgres_data (execution history + credential vault) is backed up. Nightly GH Action backs up workflow JSON + credential metadata only.
Fix (Phase 3): Weekly pg_dump n8n to Wasabi. Once N8N_ENCRYPTION_KEY is in SOPS, n8n_data becomes non-critical.
Weak Point 5 (LOW): OAuth credentials don’t survive domain change
If n8n moves to a new domain, OAuth callback URLs change and token refresh fails at the provider. Currently no OAuth credentials in use (LinkedIn uses a static bearer token). Document as a future concern.
Weak Point 6 (LOW): Secret rotation requires container restart (~30s downtime)
LINKEDIN_ACCESS_TOKEN expires every 60 days. Rotation: edit SOPS → commit → push → secrets-sync.yml → start.sh restarts containers. Watchdogs and retry logic cover the window.
2. Architecture Decision: What Goes Where
| Credential type | Mechanism | Portable? | Masked in logs? |
|---|---|---|---|
| API keys, tokens, passwords | n8n credential vault via sync-n8n-credentials.py | Yes — if N8N_ENCRYPTION_KEY in SOPS | Yes — masked in logs |
| OAuth (future) | n8n credential store + shared N8N_ENCRYPTION_KEY | Yes — if key in SOPS | Yes |
| Non-secret config (URLs, IDs) | n8n Variables ($vars.VAR) | Partial — values must exist on target | N/A |
| Hardcoded in Set node values | Never | No | No |
Rule (updated #1231): API keys and tokens in workflow nodes must use the n8n credential vault (httpHeaderAuth) populated by sync-n8n-credentials.py --cred KEY. Using $env.SOMETHING directly in node parameters is permitted only for non-secret config (IDs, URLs) and for secrets not yet migrated. If a node has a literal API key typed in, that is a bug.
3. Implementation Plan
Phase 1 — Fix the encryption key (1h, one-time) ← URGENT
Goal: N8N_ENCRYPTION_KEY lives in secrets/n8n-bms4.env.sops and is explicitly passed to all containers. Volume loss no longer destroys the credential store.
Step 1.1 — Extract current key from running container
ssh root@54.36.123.110
docker exec bms-4-n8n-1 cat /home/node/.n8n/config | python3 -m json.tool | grep encryptionKeyDo NOT paste the key value in chat. Copy it directly into the SOPS editor in Step 1.2.
Step 1.2 — Add to SOPS
# Local dev machine (requires age key at ~/.age/p24-infra-keys.txt)
export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"
sops secrets/n8n-bms4.env.sops
# Add line: N8N_ENCRYPTION_KEY=<extracted value>
# Save and exit — SOPS re-encrypts automaticallyStep 1.3 — Add to docker-compose.yml
In bms-4/docker-compose.yml, add to the environment: block of n8n, n8n-worker-1, n8n-worker-2, n8n-worker-3 (after EXECUTIONS_MODE=queue):
- N8N_ENCRYPTION_KEY=${N8N_ENCRYPTION_KEY}Step 1.4 — Commit and push to main
git add secrets/n8n-bms4.env.sops bms-4/docker-compose.yml
git commit -m "fix(#789): pin N8N_ENCRYPTION_KEY from SOPS + add to all n8n containers"
git push origin main
# secrets-sync.yml auto-triggers → deploys .env to bms-4 → start.sh restarts containersStep 1.5 — Verify
ssh root@54.36.123.110
docker exec bms-4-n8n-1 printenv N8N_ENCRYPTION_KEY
# Must print the key (non-empty)
curl -s https://n8n.bms-4.infra.zintegrowana.online/healthz
# Expected: {"status":"ok"}Phase 2 — Add n8n Variables for non-secret config
BLOCKED — Community Edition limitation.
feat:variablesrequires a paid n8n license. Verified 2026-06-21:GET /api/v1/variablesreturns HTTP 403"Your license does not allow for feat:variables".Workaround: Use n8n “Set” nodes with hardcoded non-secret values (URLs, bucket names), or keep them in
$envvia SOPS and docker-compose. The$envpattern already covers secrets; for public non-secret config (Supabase URL, Wasabi endpoint), hardcoded values in Set nodes are acceptable — they are not secrets and don’t need rotation.
Skip this phase until an Enterprise license is obtained.
Phase 3 — Add PostgreSQL nightly backup (2h)
Goal: Nightly pg_dump to Wasabi. Combined with N8N_ENCRYPTION_KEY from SOPS, enables full restore with zero manual credential re-entry.
In .github/workflows/n8n-backup.yml, add after existing workflow export step:
- name: Dump n8n PostgreSQL to Wasabi
env:
AWS_ACCESS_KEY_ID: ${{ secrets.P24_INFRA_WASABI_ACCESS_KEY }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.P24_INFRA_WASABI_SECRET_KEY }}
SSH_KEY: ${{ secrets.VPS_ROOT_SSH_KEY }}
run: |
mkdir -p ~/.ssh
echo "$SSH_KEY" | base64 -d > ~/.ssh/id_bms4 && chmod 600 ~/.ssh/id_bms4
ssh-keyscan -H 54.36.123.110 >> ~/.ssh/known_hosts
DATE=$(date +%Y-%m-%d)
ssh -i ~/.ssh/id_bms4 root@54.36.123.110 \
"docker exec bms-4-n8n-postgres-1 pg_dump -U n8n n8n | gzip > /tmp/n8n-postgres-${DATE}.sql.gz"
scp -i ~/.ssh/id_bms4 root@54.36.123.110:/tmp/n8n-postgres-${DATE}.sql.gz /tmp/
aws s3 cp /tmp/n8n-postgres-${DATE}.sql.gz \
s3://p24-infra/n8n/postgres-${DATE}.sql.gz \
--endpoint-url https://s3.eu-central-2.wasabisys.com
ssh -i ~/.ssh/id_bms4 root@54.36.123.110 "rm /tmp/n8n-postgres-${DATE}.sql.gz"
rm -f /tmp/n8n-postgres-${DATE}.sql.gz ~/.ssh/id_bms4Phase 4 — Audit credential store
Audited 2026-06-21. 45 credentials found. Categories:
| Category | Count | Action |
|---|---|---|
| OAuth 2.0 (Gmail, Google Drive, YouTube, Calendar, WhatsApp) | ~10 | Keep in store — OAuth tokens cannot meaningfully be stored as env vars |
| Telegram bots (service, prod, dev, manager, et-furpac, WAHA) | 6 | Keep in store |
| Supabase / PostgreSQL | 6 | Acceptable in store; also covered by $env for workflow expressions |
| AWS / Wasabi | 2 | Covered by $env.WASABI_ACCESS_KEY; credential entries are redundant but harmless |
| API keys (Anthropic, Groq, OpenAI, Gemini, Cohere, OpenRouter, xAi) | ~8 | Keep in store (masked in logs unlike $env) |
| Webhooks / HMAC / bearer tokens | ~7 | Keep in store |
| SMTP / IMAP | 4 | Keep in store (OAuth or password-based) |
| Other (GitLab, p24ic, basic auth) | ~2 | Review when encountered |
Finding: The credential store is used correctly. OAuth credentials cannot be replaced with $env.
For API key credentials, the store is actually preferable to $env because n8n masks stored credential values in execution logs — $env values are visible in logs for 7 days (see Weak Point 3).
Risk mitigation complete: N8N_ENCRYPTION_KEY is in SOPS + pg_dump runs nightly to Wasabi.
Loss of the n8n container volume no longer destroys credentials.
4. Migration Playbook — Moving n8n to a New Server
Pre-flight checklist
-
N8N_ENCRYPTION_KEYis insecrets/n8n-bms4.env.sops(Phase 1 complete) -
secrets-sync.ymlhas run successfully on the source server —.envis current - Confirm key matches:
docker exec bms-4-n8n-1 printenv N8N_ENCRYPTION_KEY - Note any OAuth credentials needing re-auth if domain changes
Step 1 — Export workflows
curl -s -H "X-N8N-API-KEY: ${BMS4_N8N_API_KEY}" \
https://n8n.bms-4.infra.zintegrowana.online/api/v1/workflows \
> /tmp/n8n-workflows-$(date +%Y-%m-%d).jsonStep 2 — Dump PostgreSQL
# On source server
docker exec bms-4-n8n-postgres-1 pg_dump -U n8n n8n \
| gzip > /tmp/n8n-postgres-export.sql.gz
scp root@54.36.123.110:/tmp/n8n-postgres-export.sql.gz /tmp/Step 3 — Provision new server
- Clone repo:
git clone git@github.com:radieu/p24-infra.git /opt/p24-infra - Run
secrets-sync.ymltargeting new server (or manually SCP the.envfrom SOPS) bash /opt/p24-infra/bms-4/start.sh— sources.env(containsN8N_ENCRYPTION_KEYfrom SOPS) → starts containers- Wait:
curl -s http://localhost:5678/healthz
Step 4a — Restore via PostgreSQL dump (preferred)
# On new server
docker exec -i bms-4-n8n-postgres-1 psql -U n8n n8n \
< <(zcat /tmp/n8n-postgres-export.sql.gz)
docker compose restart n8n n8n-worker-1 n8n-worker-2 n8n-worker-3Same N8N_ENCRYPTION_KEY from SOPS → all credential values decrypt. Zero manual re-entry.
Step 4b — Restore from Wasabi backup (fallback — no source server)
DATE="YYYY-MM-DD" # latest available backup
aws s3 cp s3://p24-infra/n8n/postgres-${DATE}.sql.gz /tmp/ \
--endpoint-url https://s3.eu-central-2.wasabisys.com
docker exec -i bms-4-n8n-postgres-1 psql -U n8n n8n \
< <(zcat /tmp/n8n-postgres-${DATE}.sql.gz)
docker compose restart n8n n8n-worker-1 n8n-worker-2 n8n-worker-3Step 5 — Activate workflows and verify
# Activate all workflows
for ID in $(curl -s -H "X-N8N-API-KEY: ${BMS4_N8N_API_KEY}" \
http://localhost:5678/api/v1/workflows | python3 -c "import sys,json; [print(w['id']) for w in json.load(sys.stdin)['data']]"); do
curl -s -X PATCH -H "X-N8N-API-KEY: ${BMS4_N8N_API_KEY}" \
-H "Content-Type: application/json" \
-d '{"active":true}' "http://localhost:5678/api/v1/workflows/$ID"
done
# Check ATRAX GPS sync
curl -s -H "X-N8N-API-KEY: ${BMS4_N8N_API_KEY}" \
"http://localhost:5678/api/v1/executions?workflowId=CCx9UMdphmGficDX&limit=3" \
| python3 -m json.tool | grep '"status"'Step 6 — OAuth re-auth (if domain changed)
If new server runs under a different domain, for each OAuth credential:
- Update redirect URI in provider’s OAuth app console to new
https://<domain>/rest/oauth2-credential/callback - In n8n UI: Credentials → open → Reconnect → complete OAuth flow
5. New Workflow Authoring Standards
Correct — vault credential (preferred, log-safe)
API keys and tokens must use the n8n credential vault so values are masked in execution logs.
{
"name": "Call Supabase",
"type": "n8n-nodes-base.httpRequest",
"typeVersion": 4.2,
"parameters": {
"authentication": "genericCredentialType",
"genericAuthType": "httpHeaderAuth",
"url": "https://mwkqmgadqnkkihjdeqsi.supabase.co/rest/v1/my_table",
"sendHeaders": true,
"headerParameters": {
"parameters": [
{ "name": "Authorization", "value": "=\{{ \"Bearer \" + $credentials.httpHeaderAuth.value }}" },
{ "name": "Content-Type", "value": "application/json" }
]
}
},
"credentials": {
"httpHeaderAuth": { "id": "<cred-id>", "name": "supabase-service-role-key" }
}
}Notes:
- The
apikeyheader is auto-injected by thehttpHeaderAuthcredential (configured withname: "apikey"). Authorization: Bearerreferences the vault value via$credentials.httpHeaderAuth.value.- To push/update the vault value:
python3 bms-4/sync-n8n-credentials.py --cred SUPABASE_SERVICE_ROLE_KEY
Acceptable — $env for non-secret config or unmigrated credentials
{
"name": "Call Supabase (legacy pattern)",
"type": "n8n-nodes-base.httpRequest",
"parameters": {
"headerParameters": {
"parameters": [
{ "name": "apikey", "value": "={{ $env.SUPABASE_SERVICE_KEY }}" },
{ "name": "Authorization", "value": "=Bearer {{ $env.SUPABASE_SERVICE_KEY }}" }
]
}
}
}Wrong — hardcoded secret
{ "name": "apiKey", "value": "sk-real-key-here", "type": "string" }Adding a new secret to n8n (vault path)
- Add the key to
secrets/n8n-bms4.env.sopsand register it inCREDENTIALSlist inbms-4/sync-n8n-credentials.py secrets-sync.ymldeploys SOPS → bms-4.env, then runpython3 bms-4/sync-n8n-credentials.py --cred MY_KEYon bms-4- In workflow node: set
authentication: genericCredentialType,genericAuthType: httpHeaderAuth, addcredentials.httpHeaderAuthreference
6. Secret Rotation Schedule
| Secret | Expiry | Rotation path | Downtime |
|---|---|---|---|
LINKEDIN_ACCESS_TOKEN | 60 days | sops edit secrets/n8n-bms4.env.sops → commit → push | ~30s |
ATRAX_PASSWORD | On demand | Same | ~30s |
WASABI_ACCESS_KEY/SECRET | Quarterly | Wasabi IAM rotation → sops edit → commit → push | ~30s |
N8N_ENCRYPTION_KEY | Never (unless compromised) | Export credentials first → sops edit → push → verify | ~5 min |
SUPABASE_SERVICE_ROLE_KEY | On demand | SOPS edit → commit → push → python3 bms-4/sync-n8n-credentials.py --cred SUPABASE_SERVICE_ROLE_KEY on bms-4 | Zero — vault PATCH is live |
7. What’s Already Good (Don’t Change)
- n8n credential vault (
httpHeaderAuth) for API keys — log-safe, synced viasync-n8n-credentials.py(#1231) EXECUTIONS_DATA_PRUNE=true+ 7-day TTL — mitigates log exposure- SOPS + age as single source of truth for all n8n secrets (
secrets/n8n-bms4.env.sops) - Nightly workflow JSON backup to Wasabi
- All workers share
n8n_datavolume (encryption key access for workers) N8N_RUNNERS_ENABLED=false(hotfix stable) — don’t touch