p24-infra Docs

ssh-pub-sidecar-ci-runner

3 min read

Playbook: SSH auth fails due to stale .pub sidecar on CI runner

Trigger

secrets-sync.yml sync-vps-i1 (or any other SSH step) fails with:

Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

…even though the private key is correct and authorized_keys on the target server has been updated.

What causes this

Self-hosted GitHub Actions runners reuse the same home directory between jobs. When a previous run wrote ~/.ssh/id_ed25519 (private key) and a paired ~/.ssh/id_ed25519.pub (public key), the cleanup step typically removes only the private key but leaves the .pub sidecar.

On the next run, ssh-keygen -l -f ~/.ssh/id_ed25519 reads the sidecar (not the private key) and reports the old fingerprint in logs — masking the actual key in use. More critically, OpenSSH presents the public key from the sidecar to the server during the authentication handshake. The server checks authorized_keys for that (old) public key, doesn’t find it, and rejects the connection.

This happens even when the private key written this run is perfectly correct and its corresponding public key IS in authorized_keys.

How to confirm

  1. Add ssh-keygen -y -f ~/.ssh/id_ed25519 to the step to log the true fingerprint derived from the private key.
  2. Check whether ~/.ssh/id_ed25519.pub exists on the runner before the key write step.
  3. If ssh-keygen -l and ssh-keygen -y show different fingerprints → stale sidecar confirmed.

Fix

Add rm -f ~/.ssh/id_ed25519.pub immediately before writing the new private key:

- name: Ship to vps-i1
  run: |
    set -e
    mkdir -p ~/.ssh
    rm -f ~/.ssh/id_ed25519.pub  # stale sidecar causes SSH to present wrong pubkey
    # ... write new key, chmod, ssh-keyscan, scp/ssh

Also add the following diagnostic to help future debugging:

echo "::notice::key fp: $(ssh-keygen -l -f ~/.ssh/id_ed25519 2>&1 | awk '{print $2}')"
echo "::notice::key pub: $(ssh-keygen -y -f ~/.ssh/id_ed25519 2>&1 | awk '{print $2}')"

If these two notices show different fingerprints, the sidecar cleanup is missing.

Permanent prevention

Ensure the Scrub plaintext secrets step removes both:

rm -f ~/.ssh/id_ed25519 ~/.ssh/id_ed25519.pub

Escalation

If the fix above still fails, check:

  1. ssh -v claude-admin@<host> for “No more authentication methods” — may indicate authorized_keys issue
  2. ssh-keygen -y -f ~/.ssh/id_ed25519 | ssh-keygen -l -f - — confirm private key fingerprint
  3. ssh-keygen -l -f /home/claude-admin/.ssh/authorized_keys on the target server — confirm key is listed
  4. ls -la ~/.ssh/ on the runner before the SSH step — look for unexpected files

History

  • Issue #2965: sync-vps-i1 SSH auth failed for sessions 45–52 (2026-07-05 to 2026-07-06). Multiple key rotations and SOPS migrations attempted before root cause identified.
  • Fix merged: PR #3071 (2026-07-06)
  • Verified: run 28826826253 — step 11 first success after 7 sessions

Graph View