p24-infra Docs

secrets-sync-shared-home-ssh-key-race

4 min read

secrets-sync — shared-$HOME SSH key race between concurrent deploy jobs

Applies to: .github/workflows/secrets-sync.yml, any workflow with multiple runs-on: [self-hosted, secrets-deploy] jobs that SSH to the same host. First diagnosed: 2026-07-19 (#4332). Related: #3625 (same root class, sops binary).

Symptom

A deploy step fails mid-job with an SSH key that was working seconds earlier in the same step:

Warning: Identity file /home/claude-runner/.ssh/id_bms1 not accessible: No such file or directory.
root@94.23.26.113: Permission denied (publickey).
lost connection
##[error]Process completed with exit code 1.

Tell-tale signature: an earlier ssh/scp in the same step succeeded, and the failure lands on a later command. The key is not wrong or expired — it disappeared between the two commands.

The failure is intermittent and looks unrelated to whatever change triggered the run. Jobs that start last fail most often, because they are the ones still running when siblings finish and clean up.

Root cause

The secrets-deploy runners are multiple runner instances that share $HOME=/home/claude-runner. Several jobs in secrets-sync.yml target bms-1 and run in parallel:

  • sync-bms-1-mezmo
  • sync-pinbox24-w3
  • sync-pinbox24-w4
  • sync-pinbox24-backends
  • sync-bms1-all

Historically each of them wrote the key to the same absolute path:

printf '%s' "$SSH_KEY" | base64 -d > ~/.ssh/id_bms1      # write  (per step)
...
rm -f /tmp/<x>.env ~/.ssh/id_bms1                        # scrub  (end of job)

Because $HOME is shared, ~/.ssh/id_bms1 is one file for all of them. When any one job reaches its “Scrub plaintext secrets” step, its rm -f deletes the key out from under every sibling job still mid-transfer. Those siblings then fail with Permission denied (publickey).

Observed instance (run 29691164407):

JobResultWindow
sync-pinbox24-w3success14:36:25 → 14:37:23 ← scrub fires ~14:37:21
sync-pinbox24-w4failure14:37:13 → 14:37:25 ← scp starts 14:37:21.11

This is the same shared-$HOME hazard already worked around in the Install sops step (#3625, where a concurrent curl writing ~/.local/bin/sops produced Text file busy).

Fix — per-job key path

Use $RUNNER_TEMP instead of $HOME/.ssh. RUNNER_TEMP is unique per runner instance and cleaned between jobs, so concurrent jobs cannot see, overwrite, or delete each other’s key file.

# BEFORE — shared, racy
touch ~/.ssh/id_bms1 && chmod 600 ~/.ssh/id_bms1
printf '%s' "$SSH_KEY" | base64 -d > ~/.ssh/id_bms1
scp -i ~/.ssh/id_bms1 ...
rm -f /tmp/x.env ~/.ssh/id_bms1
 
# AFTER — per-job, collision-free
touch "$RUNNER_TEMP/id_bms1" && chmod 600 "$RUNNER_TEMP/id_bms1"
printf '%s' "$SSH_KEY" | base64 -d > "$RUNNER_TEMP/id_bms1"
scp -i "$RUNNER_TEMP/id_bms1" ...
rm -f /tmp/x.env "$RUNNER_TEMP/id_bms1"

Keep the explicit rm -f (repointed): the runner cleans RUNNER_TEMP on its own, but scrubbing promptly keeps key material on disk for the shortest possible window.

mkdir -p ~/.ssh stays — it is still needed for ssh-keyscan >> ~/.ssh/known_hosts. Appending to known_hosts is race-tolerant; only the key file needed isolating.

Verification

# 1. No shared-path key writes remain
grep -c '~/\.ssh/id_bms1' .github/workflows/secrets-sync.yml    # must be 0
 
# 2. Workflow still parses and the job graph is unchanged
python3 -c "import yaml; yaml.safe_load(open('.github/workflows/secrets-sync.yml'))"
docker run --rm -v "$PWD:/repo" -w /repo rhysd/actionlint:latest .github/workflows/secrets-sync.yml
# (the 'label "secrets-deploy" is unknown' warnings are pre-existing and expected)
 
# 3. End-to-end
gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=pinbox24-all

Which key names are affected

Only key names written by more than one job can race. As of 2026-07-19:

Key nameJobs writing itRacy?
id_bms15 (mezmo, w3, w4, backends, bms1-all)yes — fixed here
id_bms41 (sync-bms-4)no
id_ed255191 (sync-vps-i1)no
id_vps_h1, id_jump1 (sync-vps-h1)no

If you add a new job that SSHes to an already-targeted host, write its key to $RUNNER_TEMP — never to a shared $HOME path — or you reintroduce this bug for every existing job on that host.

Error notification

A failure of this class surfaces via the standard secrets-sync error path (Discord P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL + GH issue with label bug). If you see Identity file ... not accessible in a secrets-deploy job, check sibling job timings in the same run before suspecting the SSH key itself or the deploy target.

Graph View