Worker /tmp secret hygiene

Class: security guardrail · Origin: incident #7724 · Owner: sys-admin / dev-coder

The incident (#7724)

Worker sessions on the shared bms-4 host improvised plaintext “worker env” files in /tmp (/tmp/wenv.sh, /tmp/w<issue>_env.sh, /tmp/sops_plain_tmp.env, …) that exported Supabase / Discord-webhook / GitHub secrets and then sourced them. Two compounding faults:

1. Secrets written to a file at all. The mandated primitive is scripts/lib/extract_env_key.sh — it writes the value to stdout only, never a file, and the anchored ^KEY= match is collision-proof (#3494).
2. World-readable. The worker command shell runs with the ambient umask 0002/0022, so any file it creates is mode 0664/0644 — readable by every local account (claude-runner, claude-runner-2, ubuntu, …). The #7724 sweep found ~960 of ~985 accumulated worker scratch files on bms-4 world-readable, including a decrypted SOPS env sitting root:root mode 644 for 3+ months.

sourceing a file out of world-writable /tmp is also the #5078 execute-the-env exposure: a single stray char turns a KEY=value assignment into an executed command that echoes the value.

The guardrails (1–3 shipped under #7724; 4 under #7813)

1. Runtime: umask 077 at every worker launch — the effective fix

The worker’s command shell is non-login + non-interactive, so neither /etc/profile.d nor ~/.bashrc is sourced by it — a umask set there is theatre. The only reliable lever is the umask inherited across exec from the launcher process. Each spawn script therefore sets umask 077 immediately before invoking claude:

• scripts/spawn-worker.sh — both the systemd/bare-host path (bms-4, vps-i1) and the Docker/windows-dev path.
• scripts/spawn-infra-task-worker.sh.

With umask 077, every file the worker (and every Bash-tool shell it spawns) creates is 0600 owner-only — so even a worker that ignores rule 1 above cannot leave a world-readable secret behind. The shared worker $LOG is pre-created in an explicit ( umask 000; … ) subshell before the launch, so its append-mode is unaffected; umask 077 only tightens new files.

Nightly agent launchers (run-nightly-triage.sh, run-nightly-design-issues.sh, claude-nightly-et-op.sh, p24-infra-nightly.sh) are the same class; harden them the same way if they ever start handling secret scratch.

2. Committed code: the lint-worker-tmp-secrets CI check

scripts/lint/check-no-tmp-secret-writes.sh, wired into .github/workflows/lint-worker-tmp-secrets.yml, blocks a PR that commits the anti-pattern into the worker-behaviour surface (scripts/, infra/agent-prompts/, .claude/, any *.sh). It scans only changed files and flags:

Rule	Pattern
source-from-tmp	source / . of a /tmp/… path (always unsafe — #5078)
wenv-scratch-file	the /tmp/wenv*.sh · /tmp/w<n>_env.sh · /tmp/*env.sh scratch family
secret-redirect-to-tmp	a sops -d stream or an exported secret var redirected (>/tee) into /tmp

It prints only FILE:LINENO:<rule> — never the matched line content. It deliberately does not scan .github/workflows/** (an ephemeral single-tenant CI runner legitimately uses /tmp for SOPS scratch) or .json config (cannot carry the inline opt-out).

Legitimate exception (a doc example, a secret-scanner test fixture, a self-cleaning helper): append # lint-allow-tmp-secret to that line.

3. Runtime detection: the TmpSecretScratchExposed alert

umask 077 (guardrail 1) stops new world-readable scratch and the CI lint (guardrail 2) stops the anti-pattern reaching the repo — but neither detects a relapse: a launcher that predates the umask fix on a fresh host, a hand-chmod, or a source /tmp/…env improvised at the Bash-tool level (never committed, so the lint never sees it). The #7724 incident sat undetected for three months precisely because nothing watched the scratch dirs. scripts/check-tmp-secret-scratch.sh is that watch:

• Runs hourly as root on every common-role host (same cron + textfile-collector mechanism as the #3808 check-env-file-perms.sh sibling — see ansible/roles/common/tasks/main.yml).
• Scans worker scratch roots (/tmp, /var/tmp, /dev/shm, maxdepth 1) for files that are both world-accessible and carry a token-value pattern (sb_secret_, ghp_, github_pat_, AKIA…, AGE-SECRET-KEY-, a Discord webhook URL). Gating on world-accessibility first means the 0600 files guardrail 1 now produces — and the broker’s 0600 mktemp files — are ignored; maxdepth 1 means /tmp/worker-*/ repo checkouts (which carry this repo’s own scanner fixtures) are not descended.
• Metadata + filenames only — matched with grep -l, never grep -n/cat; emits only gauges (counts, ages, host/class labels) and never a secret value.
• Emits two textfile metric files each run (both rewritten in full — stable, bounded series set):
  • tmp_secret_scratch_count.prom → the aggregate gauge p24_tmp_secret_scratch_count{host}.
  • tmp_secret_files.prom → the per-class gauges (#7813 item 4) p24_tmp_secret_files{host,class,world_readable} (count per credential family) and p24_tmp_secret_file_oldest_age_seconds{host,class,world_readable} (age of the oldest file in that class). class is one of supabase_service_key, github_token, github_pat, aws_access_key, age_secret_key, discord_webhook; world_readable is always "true" (the detector scans world-accessible files only — the exploitable class).
• Feeds two alerts in monitoring/prometheus/rules/security.yml:
  • TmpSecretScratchExposed (p24_tmp_secret_scratch_count > 0) — fires the same hour any world-readable token-class file lands.
  • TmpSecretScratchStale (p24_tmp_secret_file_oldest_age_seconds{world_readable="true"} > 86400, #7813) — escalates when such a file has persisted past 24 h, i.e. the #7724 chronic-accumulation failure mode (files sat unnoticed for three months). {{ $labels.class }} names the leaked credential family.
  • On fire (either): treat as a live exposure — check /var/log/p24-tmp-secret-scratch.log on that host (paths only), delete the offending file, and have a secret-manager verify whether a still-live credential leaked and rotate per static-api-key-incident-rotation.md.

4. Runtime: a per-worker private TMPDIR (#7813)

umask 077 makes new files owner-only (0600); a per-worker TMPDIR goes one step further and hides the filenames too. scripts/spawn-worker.sh creates an owner-only 0700 dir once per launch and exports it:

WORKER_TMPDIR="$(mktemp -d "${TMPDIR:-/tmp}/p24-worker-${ISSUE_NUM}-XXXXXX")"
chmod 700 "$WORKER_TMPDIR"
export TMPDIR="$WORKER_TMPDIR"

Every tool that honours TMPDIR (Python tempfile, library scratch, the worker’s own mktemp defaults, the heartbeat-key file) now writes into that un-listable subtree instead of loose into world-traversable shared /tmp — so a relapse is confined to one directory other accounts cannot even enumerate, and cleanup is a single rm -rf. The name is p24-worker-* (distinct from the worker-* repo checkout $WORKDIR) and 0700, so the guardrail-3 sweep — maxdepth 1, world-access gated — never descends it.

Cleanup is colocated with the $WORKDIR teardown in each launch path, not a parent trap … EXIT. The default path launches the worker inside a detached systemd-run --scope … bash -c that outlives spawn-worker.sh; a parent EXIT trap would delete the live worker’s TMPDIR the instant the launcher returns. So the dir is reaped inside the scope’s own teardown (next to the $WORKDIR rm -rf), inside the foreground Docker path’s teardown, and in the spawn-fail / subscription-limit / OAuth-revoked kill paths (where the worker was killed before it could reap its own). The success path deliberately does not reap — the detached worker owns it.

Not yet extended to scripts/spawn-infra-task-worker.sh: that launcher is nohup-detached with no worker-exit hook to hang cleanup on. umask 077 (guardrail 1) already covers the leak class there; a per-worker TMPDIR is a future enhancement.

Do this instead

# ✅ value to stdout, never a file:
source /opt/p24-infra/scripts/lib/extract_env_key.sh
TOKEN=$(extract_env_key CF_API_TOKEN "$ENV_FILE"); some-cmd --token "$TOKEN"; unset TOKEN
 
# ✅ if you genuinely need a temp file: random name, owner-only, self-cleaning:
TMP=$(mktemp); chmod 600 "$TMP"; trap 'rm -f "$TMP"' EXIT

Deleting a secret scratch file — the sanctioned wipe (shred is blocked) (#7826)

When a secret-bearing temp file must be removed, do not reach for shred. shred, like dd of=/dev/…, mkfs, wipefs, fdisk, parted, gdisk/sgdisk, is blocked by the DISK DESTRUCTION guard in .claude/hooks/pre-bash-safety.sh (and the -windows.ps1 sibling). The guard refuses the whole Bash-tool call before it runs — so a shred -u "$F" || rm -f "$F" one-liner never reaches the || rm -f fallback; the worker just gets BLOCKED and the file is left on disk. That is exactly the #7812 fallback that forced plain rm -f on the bms-4 sweep (52 files), leaving content unlinked but not overwritten.

We deliberately do NOT relax the guard for convenience (option (b), #7826). The sanctioned best-effort wipe for a regular secret scratch file is:

# ✅ sanctioned best-effort wipe — works inside the guarded worker Bash shell:
F=/tmp/some-secret-scratch        # a regular file under /tmp, /var/tmp or /home/claude-runner*/
truncate -s 0 -- "$F" && rm -f -- "$F"

This is NOT forensic-grade. On ext4 / mdraid / SSD (copy-on-write, wear-levelling, journaled metadata) neither truncate nor shred guarantees the old blocks are unrecoverable. The truncate -s 0 drops the file’s data extents and then rm unlinks the inode — better than a bare rm (the content is no longer reachable through the path and the extents are freed for reuse) but it is a hygiene step, not a guarantee the bytes are gone. The only real mitigation for a credential that has already sat in a world-readable file is rotation (see the rule below), not any flavour of local wipe.

The best fix is to never write the secret to a file at all (the extract_env_key.sh / mktemp+trap patterns above). The truncate+rm wipe is for the case where a temp file was genuinely unavoidable, or when sweeping a pre-existing backlog.

Exposed-credential rotation rule — world-readable > 24 h ⇒ treat as exposed (#7826)

A best-effort wipe closes the file; it does not un-expose a value that other local accounts could already have read. Therefore:

Any credential that lived in a world-readable file (mode g+r/o+r) for more than 24 h MUST be treated as exposed and queued for rotation via a secret-manager request (docs/playbooks/secret-manager-request.md / docs/playbooks/static-api-key-incident-rotation.md). The 24 h threshold is the same window the TmpSecretScratchStale alert (guardrail 3) escalates on.

A sweep/deletion task must output the list of affected credential NAMES (key names only — never values, per the ABSOLUTE no-secret-in-output rule) so a secret-manager can rotate each one. The deletion itself (plain rm/truncate+rm) does not discharge the exposure — rotation does.

Worked example — the #7724 → #7812 lineage:

• #7724 — the original incident: ~960 of ~985 worker scratch files on bms-4 world-readable, including a decrypted SOPS env (/tmp/sops_plain_tmp.env) root:root mode 644 for 3+ months.
• #7793 — the Discord-webhook rotation opened off the #7724 sweep (a credential that had sat exposed, so it was queued for rotation rather than just deleted).
• #7812 — the owner-approved bulk deletion (52 files) that had to fall back to plain rm -f because shred was blocked — the trigger for this playbook section and the sanctioned wipe above.

# ❌ never — named secret scratch in /tmp, world-readable, then sourced:
sops -d secrets/monitoring.env.sops > /tmp/wenv.sh
source /tmp/wenv.sh

Remediation of the live backlog (separate, not shipped here)

The one-time bulk cleanup of the ~985 pre-existing /tmp scratch files on bms-4 (oldest-first, starting with /tmp/sops_plain_tmp.env) is a sys-admin / infra-task action requiring a root shell and secret-manager verification of any still-live credential — see the #7724 sweep comment. It is intentionally out of scope for this dev-coder guardrail PR.

Related

• docs/playbooks/env-key-extraction-safety.md — the extract_env_key.sh primitive (#3494)
• docs/playbooks/static-api-key-incident-rotation.md — rotation path once a value has leaked
• Global CLAUDE.md §Secrets — the ABSOLUTE no-secret-in-output rule
• #5078 — the source-the-env exposure class · #7791 — the Discord-webhook rotation still open on #7724