`ANTHROPIC_ADMIN_API_KEY` — provisioning playbook

Key name: ANTHROPIC_ADMIN_API_KEY SOPS file: secrets/anthropic-admin.env.sops — narrow, 2 recipients only (developer + CI AGE_KEY_GHA). Isolated out of secrets/monitoring.env.sops (#5697/#5685) because that file’s env_file: .env pattern fans this org-admin-tier credential out to every container that reads monitoring/.env — including alertmanager, mezmo-exporter, and caddy (the internet-facing reverse proxy on ports 80/443). Do not add this key back to monitoring.env.sops. Provider: Anthropic (Claude Console — organization admin) Tier: 3 — manual only (see secret-rotation-access-matrix.md §Tier 3) Consumer: cost-exporter on vps-i1 → collect_anthropic() → anthropic_token_spend_usd Origin: #4970 (ADD request, filed from #4964 · collector added in #4528)

Why this is Tier 3 and not Tier 1/2

There is no automation path to mint this key — verified against Anthropic’s own Admin API documentation on 2026-08-01 (https://platform.claude.com/docs/en/api/administration-api):

Can I create new API keys through the Admin API? No, new API keys can only be created through the Claude Console for security reasons. The Admin API can only manage existing API keys.

Three consequences, each independently sufficient to make this human-only:

No create endpoint. /v1/organizations/api_keys supports list / get / update only. There is no POST that returns a key value — for ordinary keys or for Admin keys.
No bootstrap. Even if an endpoint existed, calling it would require an Admin API key or an org:admin OAuth token. No Anthropic credential of any kind exists in any secrets/*.env.sops file (verified by key-NAME existence scan, values never read) — every ANTHROPIC_* key was revoked without replacement on 2026-07-06 under incident #2970, see anthropic-api-key-rotation.md. There is nothing to bootstrap from.
Admin role + Console UI. “Only organization members with the admin role can provision one,” and provisioning happens in the Console. Console login for this org is Google SSO (radieu@gmail.com) — see anthropic-api-key-rotation.md.

Why the Tier 2 Playwright path does not apply

ANTHROPIC_API_KEY sits in Tier 2 with a Playwright route (console.anthropic.com/settings/keys, prerequisite: an authenticated Google session in the Playwright profile). That row does not transfer here:

Different Console surface and a higher privilege level (org-admin key, billing scope).
The Tier 2 prerequisite is provisioned on bms-4 (playwright_enabled: true). This worker runs on vps-i1, where no Playwright/Chromium cache exists at all — the Tier 2 prerequisite check fails closed.
Automating an org-admin credential mint behind Google SSO is out of scope for the autonomous worker regardless of host (worker-secret-manager.md §Never self-heal).

The `org:admin` OAuth alternative — also human, and unsuitable here

GET /v1/organizations/cost_report accepts either an Admin API key (x-api-key) or an org:admin OAuth bearer token. The OAuth path (ant auth login --profile admin --scope "org:admin") still requires an interactive browser login by an admin/owner, and mints a short-lived token — unusable as a long-lived exporter secret. Workload Identity Federation could issue org:admin tokens non-interactively, but the service-account, federation-issuer, and federation-rule endpoints that configure WIF reject Admin API keys and require an org:admin OAuth token themselves. The bootstrap is human either way. The Admin API key is the correct credential for this consumer.

Step 1 — HUMAN: mint the key (Claude Console)

Performed by an organization admin / owner of the Anthropic org:

Sign in at https://platform.claude.com/ (Google SSO, radieu@gmail.com).
Go to Settings → Admin keys (see https://platform.claude.com/docs/en/manage-claude/admin-api-keys).
Create Admin API key. Name it p24-infra-cost-exporter.
Copy the value immediately — it is shown once. Value starts with sk-ant-admin.
Hand it to the secret-manager over a secure channel. Never paste it into a GitHub issue, PR, chat message, or commit.

Scope note: an Admin API key is organization-wide. The cost-exporter only performs GET /v1/organizations/cost_report, but the key itself is not scope-restrictable at creation time. Treat it as a privileged credential; it is a 180-day rotation candidate.

Step 2 — secret-manager: add to SOPS

secrets/anthropic-admin.env.sops is a narrow, exact-filename-anchored file in .sops.yaml (2 recipients: developer + CI AGE_KEY_GHA) — the same pattern as supabase-ci.env.sops / pinbox24-gitlab.env.sops. On Windows, use scripts/sops-set.ps1 (adds the file fresh if it doesn’t exist yet, or updates the key if it does):

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:NEW_VALUE = "paste-here"   # human sets this, never in chat
.\scripts\sops-set.ps1 -SopsFile secrets\anthropic-admin.env.sops -Key ANTHROPIC_ADMIN_API_KEY
$env:NEW_VALUE = ""

Linux worker pattern (worker-secret-manager.md §General pattern) — an add, so append rather than sed-replace:

export SOPS_AGE_KEY_FILE="${HOME}/.age/p24-infra-keys.txt"
bash /opt/p24-infra/scripts/lib/ensure-fresh-checkout.sh /opt/p24-infra main || exit 1
 
SOPS_FILE="/opt/p24-infra/secrets/anthropic-admin.env.sops"
SOPS_TMP="${SOPS_FILE%.env.sops}-tmp.env.sops"
 
# NEW_VAL is read from a mode-600 file the human dropped, never from a shell literal
NEW_VAL=$(cat /path/to/handoff-file)
 
{ [ -f "$SOPS_FILE" ] && sops --decrypt --input-type dotenv --output-type dotenv "$SOPS_FILE"
  printf 'ANTHROPIC_ADMIN_API_KEY=%s\n' "$NEW_VAL"; } > "$SOPS_TMP"
unset NEW_VAL
shred -u /path/to/handoff-file
 
sops --encrypt --input-type dotenv --output-type dotenv --in-place "$SOPS_TMP" \
  || { rm -f "$SOPS_TMP"; echo "[ERROR] encrypt failed"; exit 1; }
 
# Canary on the TEMP file before the production file is touched
sops --decrypt --input-type dotenv --output-type dotenv "$SOPS_TMP" > /dev/null \
  || { rm -f "$SOPS_TMP"; echo "[ERROR] SOPS corrupt — production file untouched"; exit 1; }
mv "$SOPS_TMP" "$SOPS_FILE"

Temp file must be named *-tmp.env.sops inside secrets/ — but because anthropic-admin.env.sops is matched by an exact-filename path_regex (not the general secrets/*.env.sops catch-all), a *-tmp.env.sops staging name falls through to the 6-recipient catch-all rule instead of the intended 2-recipient rule (this is exactly the bug that broke PR #5698 — 6 recipients + an empty value, fixed in PR #5737). Verify recipient count and a real ENC[...] payload before committing:

grep -c 'map_recipient=' secrets/anthropic-admin.env.sops   # must be exactly 2
grep '^ANTHROPIC_ADMIN_API_KEY=ENC\[' secrets/anthropic-admin.env.sops   # must match (non-empty)

Step 3 — distribution

Target	Needed?	Why
`secrets/anthropic-admin.env.sops`	✅	Source of truth
vps-i1 `/opt/p24-infra/monitoring/.env.anthropic-admin`	✅ automatic	`secrets-sync.yml`’s dedicated `sync-anthropic-admin` job (#5697) decrypts this file with `AGE_KEY_GHA` and installs it 0600 root:root on merge to `main`
`cost-exporter` container restart	✅ automatic, scoped	Same job runs `docker compose up -d --no-deps --force-recreate cost-exporter` — only that container, not the full monitoring stack
GH Secret	❌	No GH Actions workflow consumes this key
Vercel	❌	No Vercel project consumes it
n8n credential	❌	Not an n8n credential

Commit + PR to main (never push to main directly). Merging does not auto-trigger sync-anthropic-admin unless the commit touches secrets/anthropic-admin.env.sops itself (the workflow’s push-path filter is secrets/*.env.sops) — a docker-compose/workflow-only change needs a manual dispatch: gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=vps-i1 (this also re-runs the unrelated sync-vps-i1 full-stack job in the same invocation — that job’s own force-recreate-everything behaviour is pre-existing and out of scope here, see #5697 Task 3).

Step 4 — verify (never print the value)

Provider-side, status code only:

KEY=$(sops --decrypt --input-type dotenv --output-type dotenv \
  /opt/p24-infra/secrets/anthropic-admin.env.sops | grep '^ANTHROPIC_ADMIN_API_KEY=' | cut -d= -f2-)
curl -s -o /dev/null -w '%{http_code}\n' \
  -H "x-api-key: $KEY" -H "anthropic-version: 2023-06-01" \
  "https://api.anthropic.com/v1/organizations/cost_report?starting_at=$(date -u +%Y-%m-01T00:00:00Z)&bucket_width=1d&limit=1"
unset KEY
# 200 = key valid with billing read access · 401 = bad key · 403 = insufficient role

Consumer-side, on vps-i1:

cd /opt/p24-infra/monitoring
ls -la .env.anthropic-admin   # confirm delivered, 0600 root:root
# char count only — never print the value
docker compose exec -T cost-exporter printenv ANTHROPIC_ADMIN_API_KEY | wc -c
curl -s http://127.0.0.1:9210/metrics | grep -c '^anthropic_token_spend_usd'   # > 0
curl -s http://127.0.0.1:9210/metrics | grep 'cost_collector_errors_total{collector="anthropic"}'
docker compose logs --tail=50 cost-exporter | grep -i anthropic | grep -vi 'key\|token\|secret'

The exporter has no container_name:, so the container resolves as monitoring-cost-exporter-1; /metrics is bound to 127.0.0.1:9210 only. collect_anthropic() runs on COST_REFRESH_INTERVAL_S (default 24 h) — force an early cycle with a container restart rather than waiting a day.

Success criteria: anthropic_token_spend_usd series present and cost_collector_errors_total{collector="anthropic"} no longer incrementing each cycle.

Step 5 — log it

Append a row to docs/secrets-rotation-log.md (newest first; columns are Date | Secret | Reason | Rotator | Confirmed in sync). Open the entry as pending before touching the key and close it only once distribution is verified:

| YYYY-MM-DD | **ANTHROPIC_ADMIN_API_KEY** — `secrets/anthropic-admin.env.sops` | `new credential` (#4970) — cost-exporter `collect_anthropic()` (#4528) had no key provisioned; Tier 3, minted by an org admin in the Claude Console (no create API — see `anthropic-admin-api-key.md`) | <human who minted> + claude (secret-manager) | pending — HH:MM UTC |

Rotation

Same procedure — there is no rotate-in-place API. Mint a new Admin key in the Console first, distribute it, verify 200, and only then deactivate the old key. The old key can be deactivated programmatically (this is the one Admin-API-automatable half):

# status=inactive, by api_key_id from GET /v1/organizations/api_keys
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
  -H "x-api-key: $NEW_KEY" -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" --data '{"status":"inactive"}' \
  "https://api.anthropic.com/v1/organizations/api_keys/<api_key_id>"

Verify-first is mandatory: never deactivate the old key before the new one returns 200.

secret-rotation-access-matrix.md — Tier 3 row for this key
anthropic-api-key-rotation.md — the 2026-07-06 revoke-only operation (#2970) that left the org with no Anthropic credential
secret-manager.md — SOPS operations + distribution chain
../monitoring-exporters-operations.md — cost-exporter operations

p24-infra Docs

Explorer

anthropic-admin-api-key

`ANTHROPIC_ADMIN_API_KEY` — provisioning playbook

Why this is Tier 3 and not Tier 1/2

Why the Tier 2 Playwright path does not apply

The `org:admin` OAuth alternative — also human, and unsuitable here

Step 1 — HUMAN: mint the key (Claude Console)

Step 2 — secret-manager: add to SOPS

Step 3 — distribution

Step 4 — verify (never print the value)

Step 5 — log it

Rotation

Graph View

Table of Contents

p24-infra Docs

Explorer

anthropic-admin-api-key

ANTHROPIC_ADMIN_API_KEY — provisioning playbook

Why this is Tier 3 and not Tier 1/2

Why the Tier 2 Playwright path does not apply

The org:admin OAuth alternative — also human, and unsuitable here

Step 1 — HUMAN: mint the key (Claude Console)

Step 2 — secret-manager: add to SOPS

Step 3 — distribution

Step 4 — verify (never print the value)

Step 5 — log it

Rotation

Related

Graph View

Table of Contents

`ANTHROPIC_ADMIN_API_KEY` — provisioning playbook

The `org:admin` OAuth alternative — also human, and unsuitable here