p24-infra Docs

ovh-api-operations

32 min read

OVH API Operations Playbook

Scope: Autonomous OVH API operations available to the p24-infra Claude role. All operations use the OVH Python SDK (ovh package) with credentials stored in secrets/monitoring.env.sops. No secret values appear in this document — reference key names only.

Last updated: 2026-08-05

1. Prerequisites

OVH Python SDK

pip install ovh

Credentials

All OVH credentials live in secrets/monitoring.env.sops. Decrypt safely before use:

# Extract a single key without printing it
$env:OVH_APP_KEY = (sops --decrypt --input-type dotenv --output-type dotenv `
    "C:\code_2026\p24-infra\secrets\monitoring.env.sops" |
    Select-String "^OVH_APPLICATION_KEY=").ToString().Split("=", 2)[1].Trim()
# Clear after use
$env:OVH_APP_KEY = ""

Required keys:

Key nameDescription
OVH_APPLICATION_KEYApplication key (AK) — identifies the app
OVH_APPLICATION_SECRETApplication secret (AS) — signs all requests
OVH_APPLICATION_CONSUMER_KEYConsumer key (CK) — user authorization token
OVH_ENDPOINTAlways ovh-eu for p24-infra

Note: Some older scripts use OVH_CONSUMER_KEY as the key name — both map to the same CK value.

Consumer key route grants

A consumer key (CK) carries a baked-in ACL of permitted routes. Before using any section below, verify the CK includes the required routes (see each section). If routes are missing, generate a new CK — see §5.

age key

SOPS decryption requires the p24-infra age key:

C:\Users\konar\.age\p24-infra-keys.txt

If this file is absent the machine cannot decrypt SOPS files — stop and escalate.

2. Rotate OVH Valkey (Redis) Password — kr40258-001

Required CK route

POST /hosting/privateDatabase/kr40258-001/user/default/changePassword

OVH constraint

OVH enforces a 30-character maximum on Valkey/Redis passwords. Always generate passwords with secrets.token_hex(14) which produces exactly 28 hex characters. Do NOT use token_urlsafe or longer hex values — the API silently truncates or rejects them.

Rotation script

The full rotation script lives in the session scratchpad as valkey_rotate_v2.py. The core API call (credentials injected from environment, no secrets printed):

import ovh
import os
import secrets
import time
 
client = ovh.Client(
    endpoint=os.environ["OVH_ENDPOINT"],
    application_key=os.environ["OVH_APPLICATION_KEY"],
    application_secret=os.environ["OVH_APPLICATION_SECRET"],
    consumer_key=os.environ["OVH_APPLICATION_CONSUMER_KEY"],
)
 
# 28 chars — stays under the 30-char OVH limit
new_password = secrets.token_hex(14)
 
result = client.post(
    "/hosting/privateDatabase/kr40258-001/user/default/changePassword",
    password=new_password,
)
print(f"Task created: {result}")  # prints task ID only, not the password
 
# Wait for OVH propagation before verifying
time.sleep(16)
# Verify: redis-cli -h <host> -a "$NEW_PASSWORD" PING

Post-rotation steps

After the API call succeeds and redis-cli PING returns PONG:

  1. Update SOPS secrets — edit secrets/pinbox24-w3.env.sops for V32_REDIS_PASSWORD and secrets/pinbox24-w4.env.sops for V42_REDIS_PASSWORD:

    Use the Windows-safe SOPS write pattern (see §5 for the pattern reference).

  2. Update backend environment files on bms-1 (root@94.23.26.113, SSH key C:\Users\konar\.ssh\id_ed25519):

    /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env
/root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/backend-environment.env

  3. Recreate containers on bms-1 to pick up the new password:

    docker compose up -d --force-recreate v32-prod
docker compose up -d --force-recreate v42-prod

  4. Verify each container is healthy after recreation.

  5. Commit the updated SOPS file to git (canary decrypt first — see §5).

3. Reboot a Dedicated Server

Required CK route

POST /dedicated/server/*/reboot

Server name map

Server labelIPOVH serviceNameAccount
bms-2145.239.133.104ns3087638OVH main
bms-351.68.155.224ns3129867OVH main
bms-454.36.123.110ns3101999OVH main
bms-194.23.26.113—Kimsufi account — NOT available via this CK

bms-1 is on the Kimsufi account which uses a separate application key / consumer key. Reboot bms-1 via the Kimsufi control panel or by contacting OVH support.

Python snippet

import ovh
import os
import time
 
client = ovh.Client(
    endpoint=os.environ["OVH_ENDPOINT"],
    application_key=os.environ["OVH_APPLICATION_KEY"],
    application_secret=os.environ["OVH_APPLICATION_SECRET"],
    consumer_key=os.environ["OVH_APPLICATION_CONSUMER_KEY"],
)
 
SERVICE_NAME = "ns3101999"  # bms-4 — change as needed
 
task = client.post(f"/dedicated/server/{SERVICE_NAME}/reboot")
task_id = task["id"]
print(f"Reboot task created: id={task_id}, function={task['function']}, status={task['status']}")
 
# Poll until done (typically 2-5 minutes for a hard reboot)
for _ in range(30):
    time.sleep(15)
    status = client.get(f"/dedicated/server/{SERVICE_NAME}/task/{task_id}")
    print(f"  Task {task_id}: {status['status']}")
    if status["status"] in ("done", "error"):
        break

Task response fields

FieldMeaning
idTask ID — use for polling
statusinit → doing → done (or error)
functionhardReboot for a standard reboot
commentHuman-readable progress note

Poll GET /dedicated/server/{serviceName}/task/{taskId} every 15 s until status == "done".

4. Rescue Mode for Dedicated Servers

Use rescue mode when a server is unresponsive at the OS level but still reachable at the hardware level (IPMI/IDRAC). Rescue boots a minimal Linux environment from the network.

Required CK routes

GET  /dedicated/server/*/boot
PUT  /dedicated/server/*
POST /dedicated/server/*/reboot
GET  /dedicated/server/*/task/*

Step-by-step

Step 1 — Discover the rescue boot ID

boots = client.get(
    f"/dedicated/server/{SERVICE_NAME}/boot",
    bootType="rescue",
)
# boots is a list of boot IDs — usually one rescue option per server
print(f"Rescue boot IDs: {boots}")
rescue_boot_id = boots[0]

Step 2 — Set next boot to rescue

client.put(
    f"/dedicated/server/{SERVICE_NAME}",
    bootId=rescue_boot_id,
)
print(f"Next boot set to rescue (bootId={rescue_boot_id})")

Step 3 — Trigger reboot

task = client.post(f"/dedicated/server/{SERVICE_NAME}/reboot")
print(f"Reboot task: {task['id']}")
# Poll as shown in §3

Step 4 — Return to normal boot after repair

# Get the harddisk boot ID
normal_boots = client.get(
    f"/dedicated/server/{SERVICE_NAME}/boot",
    bootType="harddisk",
)
normal_boot_id = normal_boots[0]
 
# Set normal boot
client.put(f"/dedicated/server/{SERVICE_NAME}", bootId=normal_boot_id)
 
# Reboot again
task = client.post(f"/dedicated/server/{SERVICE_NAME}/reboot")
print(f"Returning to normal boot, task: {task['id']}")

Warning

After Step 2 sets the rescue bootId, any subsequent reboot (whether OVH API, IPMI, or shutdown -r) will boot into rescue mode. Reset the bootId to harddisk (Step 4) immediately after completing repairs — do not leave the rescue boot set overnight.

5. Expanding Consumer Key Permissions

No-browser alternative: on the main OVH realm you can skip the browser-authorization step below entirely by using OAuth2 IAM service accounts — see §5a (#5410). §5 remains the only path for the SoYouStart/Kimsufi realm, which has no OAuth2 endpoint.

Why you cannot amend an existing CK

Consumer key permissions are baked at creation time. The OVH API provides no endpoint to add routes to an existing CK. If a new operation is needed that the current CK does not permit, generate a new CK that includes all required routes.

Generating a new CK

import ovh
import os
 
client = ovh.Client(
    endpoint=os.environ["OVH_ENDPOINT"],
    application_key=os.environ["OVH_APPLICATION_KEY"],
    application_secret=os.environ["OVH_APPLICATION_SECRET"],
    # No consumer_key — we are creating one
)
 
# Full access rules for all p24-infra operations
access_rules = [
    # Valkey / private databases
    {"method": "GET",  "path": "/hosting/privateDatabase"},
    {"method": "GET",  "path": "/hosting/privateDatabase/*"},
    {"method": "POST", "path": "/hosting/privateDatabase/*/user/*/changePassword"},
 
    # Dedicated servers — reboot + rescue + task polling
    {"method": "GET",  "path": "/dedicated/server"},
    {"method": "GET",  "path": "/dedicated/server/*"},
    {"method": "PUT",  "path": "/dedicated/server/*"},
    {"method": "POST", "path": "/dedicated/server/*/reboot"},
    {"method": "GET",  "path": "/dedicated/server/*/boot"},
    {"method": "GET",  "path": "/dedicated/server/*/task"},
    {"method": "GET",  "path": "/dedicated/server/*/task/*"},
]
 
validation = client.request_consumerkey(access_rules)
print(f"Authorize URL: {validation['validationUrl']}")
print(f"New CK (pending authorization): {validation['consumerKey']}")
# Do NOT print the CK value here in a shared log — store it immediately
  1. Copy the validationUrl and open it in a browser.
  2. Log in with the OVH account that owns the servers.
  3. Click Authorize — the CK becomes active.
  4. Update OVH_APPLICATION_CONSUMER_KEY in secrets/monitoring.env.sops.

Windows-safe SOPS write pattern

# 1. Decrypt to plaintext in memory — never print
$plain = (sops --decrypt --input-type dotenv --output-type dotenv `
    "C:\code_2026\p24-infra\secrets\monitoring.env.sops") -join "`n"
 
# 2. Replace the old CK line with the new value
$newCK = "..." # store in variable, never echo
$plain = $plain -replace "(?m)^OVH_APPLICATION_CONSUMER_KEY=.*$",
                          "OVH_APPLICATION_CONSUMER_KEY=$newCK"
 
# 3. Write plaintext with LF only (no BOM, no CRLF)
$tmpPath = "C:\code_2026\p24-infra\secrets\monitoring_new.env.sops"
[System.IO.File]::WriteAllText(
    $tmpPath,
    $plain,
    [System.Text.UTF8Encoding]::new($false)
)
 
# 4. Encrypt in-place
sops --encrypt --input-type dotenv --output-type dotenv `
    --age (Get-Content "C:\Users\konar\.age\p24-infra-keys.txt" |
           Select-String "^# public key:" | ForEach-Object { $_ -replace "# public key: ", "" }) `
    $tmpPath | Set-Content "C:\code_2026\p24-infra\secrets\monitoring_new.env.sops"
# Then rename to monitoring.env.sops — or use --in-place flag
 
# 5. Canary decrypt — MUST pass before git add
sops --decrypt --input-type dotenv --output-type dotenv `
    "C:\code_2026\p24-infra\secrets\monitoring.env.sops" | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS corrupt — do NOT commit" }
 
# 6. Stage and commit
git -C "C:\code_2026\p24-infra" add secrets/monitoring.env.sops

Full SOPS write procedure: docs/playbooks/sops-windows-crlf.md Secret manager playbook: docs/playbooks/secret-manager.md

5a. OAuth2 IAM Service Accounts — no-browser alternative to §5 (#5410)

The browser-authorization step in §5 is only required for the classic AK/AS/CK model. OVHcloud’s newer OAuth2 IAM “service account” mechanism has no browser step at all — POST /me/api/oauth2/client (flow CLIENT_CREDENTIALS) returns a non-expiring clientId/clientSecret pair, and permissions are granted afterwards with POST /v2/iam/policy. Because minting the client is itself just an authenticated API call, it can be bootstrapped with an already-valid classic credential — so the one-time migration needs zero browser interaction, and all future rotation is fully scriptable (Tier 1).

Realm support — live-probed 2026-08-04 (#5410)

RealmEndpointGET /me/api/oauth2/clientOAuth2 service accounts
OVH maineu.api.ovh.com (ovh-eu)200 OK (via OVH_APPLICATION_*; 403 “not granted” for CKs lacking the route)✅ Supported
SoYouStart / Kimsufiapi.soyoustart.com (soyoustart-eu)404 “invalid/empty URL” even with a valid, authenticated CK❌ Not supported

→ Only the main realm can be migrated. SYS_* / SoYouStart credentials stay on the classic browser-CK model (§5) until OVH ships OAuth2 on that realm — see the Tier-3 row in secret-rotation-access-matrix.md.

Bootstrap credential (important)

The mint call requires a CK that carries the /me/api/oauth2/client route. Only OVH_APPLICATION_* (app p24-infra-claude, IAM policy ovh-role-admin, in secrets/monitoring.env.sops) has it — live-confirmed 200. The OVH_INFRA_* / OVH_APP_* consumer keys return 403 “not granted” on that route and therefore cannot mint. Classic X-Ovh-Signature auth also works on the /v2/iam base (confirmed: GET /v2/iam/policy → 200 under OVH_APPLICATION_*), so the IAM policy can be attached in the same first-run pass.

Migration / rotation script

scripts/rotate/ovh-api-credentials.sh (target OVH_TARGET=ovh only) implements the full flow — first-run classic→OAuth2 migration (bootstraps with OVH_APPLICATION_*) and, once OVH_INFRA_CLIENT_ID/OVH_INFRA_CLIENT_SECRET exist in secrets/ovh-api.env.sops, true Bearer-based rotation. It refuses OVH_TARGET=sys. IAM scope is set via IAM_ACTIONS / IAM_RESOURCE_URN env vars (default dedicatedServer:apiovh:* / urn:v1:eu:resource:dedicatedServer:* — verify against the live account before the first run).

  • First-run migration is a secret-manager operation (it mints + stores a live secret) — do not run it from a dev/sys-admin session. No OAuth2 client exists yet (GET /me/api/oauth2/client → []).
  • Smoke test before revoking the legacy AK/AS/CK: §3 (dedicated-server reboot / boot-mode) only. OVH_INFRA_CLIENT_ID’s IAM policy is scoped to dedicatedServer:apiovh:* on urn:v1:eu:resource:dedicatedServer:* — it was never granted hosting/privateDatabase (Valkey) permissions, which live on the completely unrelated OVH_APPLICATION_* credential family (§2 uses OVH_APPLICATION_* explicitly, not OVH_INFRA_CLIENT_ID). §2 Valkey password rotation is a smoke test for OVH_APPLICATION_*’s own health, unrelated to this migration — do not run it as a verification step for OVH_INFRA_CLIENT_ID (a review pass on #5410 found this exact mix-up propagated through several earlier writeups on this page — “Bug 2”, “Bug 3”, and “Round 5” below originally each listed “§2/§3 smoke tests” as a pair; corrected in place to §3-only alongside this note, since those sections are otherwise an accurate chronological incident record). Leave the legacy keys in place until the §3 reboot smoke test passes.
  • Cleanup after verification: remove the stale OVH_INFRA_APP_KEY/APP_SECRET/CONSUMER_KEY names from secrets/ovh-api.env.sops and revoke the old Application (DELETE /1.0/me/api/application/{id}). As of the fix below, the script’s own best-effort revoke (its Step 8) does not run by default — it is a separate, deliberate opt-in (SKIP_CLEANUP=0), run only after the smoke tests above pass. Once a migration has completed and OVH_INFRA_CLIENT_ID/CLIENT_SECRET are live in SOPS, ordinary invocations always pick oauth2 AUTH_MODE and can no longer reach Step 8’s legacy-revoke logic — use CLEANUP_TARGET=legacy SKIP_CLEANUP=0 bash rotate/ovh-api-credentials.sh (a dedicated cleanup-only mode, redesigned in #5410 review iteration 2 — see that section below for the before/after behavior change before relying on it).

Bootstrap CK upgraded — write routes added (#5410, 2026-08-04)

The first live mint attempt (below) discovered the bootstrap OVH_APPLICATION_* CK described above did not actually carry POST /me/api/oauth2/client or POST /v2/iam/policy//iam/policy — the earlier GET /me/api/oauth2/client → 200 probe only proved the GET route was granted; OVH’s classic CK ACL grants methods independently per path, and the write verb was never exercised until the first real mint attempt. A new CK was requested via client.request_consumerkey() with the full legacy route set plus these two write routes, authorized by a human via the returned validationUrl (one-time browser step), and deployed to OVH_APPLICATION_CONSUMER_KEY in secrets/monitoring.env.sops. Live-confirmed via GET /auth/currentCredential: 18 total rules, both write routes present. This CK no longer needs a browser step for the OAuth2-mint action itself.

First live mint attempt — HALTED on a verify-endpoint bug, not a credential bug (#5410, 2026-08-04)

With the upgraded CK live on main, scripts/rotate/ovh-api-credentials.sh SKIP_CLEANUP=1 was run on bms-4. Steps 1-3 succeeded for real: the OAuth2 client minted (clientId=EU.0ff9b7d0a01b84cf) and its IAM policy attached (p24-infra-rotate-2026-08-04-policy, dedicatedServer:apiovh:* on urn:v1:eu:resource:dedicatedServer:* — exactly the documented §5a default scope). Step 4’s own verify call, Bearer GET /me, returned 403 — /me is an account-level route that a dedicatedServer-scoped IAM policy never grants (confirmed by reading the policy back: no account:* permission was attached, by design — least privilege). This is a verify-endpoint design bug in the script, not evidence the mint or the policy attach failed. Because Step 4 failed, Steps 5-7 (payload encrypt + dispatch to sops-sync-receiver.yml) never ran, so no OVH_INFRA_CLIENT_ID/CLIENT_SECRET were written to secrets/ovh-api.env.sops — the run is a no-op from SOPS’s point of view. The orphaned, now-secret-less client was deleted (DELETE /me/api/oauth2/client/EU.0ff9b7d0a01b84cf, confirmed via a follow-up empty GET /me/api/oauth2/client list) using the bootstrap CK’s DELETE /me/api/oauth2/client/* route. Its IAM policy row could not be deleted the same way — DELETE /v2/iam/policy/{id} is not in the bootstrap CK’s granted routes — but is inert: it references only the now-deleted client’s identity URN and grants nothing to anything else.

Fix needed before the next attempt (design decision, not re-attempted here — see #5410): VERIFY_URL (${API_BASE}/me) must be changed to a route the granted IAM scope actually covers — e.g. GET /dedicated/server — or the IAM policy must additionally grant an account-identity action that covers /me. Whichever is chosen, re-verify against the actual target route (§3’s dedicated-server reboot smoke test — not §2, which authenticates with the unrelated OVH_APPLICATION_* credential family, see the corrected note above), not just against /me, before treating a future mint as complete.

Bug found + fixed before first live use (#5410)

A secret-manager review of the script ahead of the first-ever live mint (halted before making any live OVH API call or SOPS write — see the halt comment on #5410) found that its Step 8 cleanup logic would have deleted the wrong Application on a real (non---dry-run) run:

  • The classic first-run path reads the legacy OVH_INFRA_APP_KEY/APP_SECRET/CONSUMER_KEY into CUR_APP_KEY/CUR_APP_SECRET/CUR_CONSUMER_KEY, then overwrites those same variables in place with the admin-scoped OVH_APPLICATION_* bootstrap credential needed to sign the mint call. The original legacy value was never preserved elsewhere.
  • Step 8 (cleanup) still matched MATCH_KEY == CUR_APP_KEY to find which Application to DELETE — but by that point CUR_APP_KEY held the admin bootstrap credential’s own key, so the match would resolve to (and delete) the admin OVH_APPLICATION_* Application itself — the credential secrets/monitoring.env.sops and cost-exporter depend on — instead of the intended legacy OVH_INFRA_* Application.
  • Step 8 also ran unconditionally right after mint + a trivial Bearer GET /me smoke check, with no opt-out gate, contradicting the “mint → verify live → then separately decide to clean up” ordering described above.

Fix (this PR): the original OVH_INFRA_APP_KEY/APP_SECRET/CONSUMER_KEY values are now captured under distinct LEGACY_APP_KEY/LEGACY_APP_SECRET/LEGACY_CONSUMER_KEY variables before the bootstrap overwrite, and Step 8 matches against LEGACY_APP_KEY (never CUR_APP_KEY), plus a defense-in-depth check refuses to delete if the two ever resolve equal. Step 8 is now also gated behind SKIP_CLEANUP (default 1 = skipped), mirroring SKIP_IAM_POLICY — a mint-only run never auto-deletes anything; set SKIP_CLEANUP=0 explicitly to run the revoke after the smoke tests pass. No live mint was performed to find or fix this — it was caught by code review of the script alone.

Bug 2 found during the first live mint attempt (#5410, 2026-08-04)

With the bootstrap CK upgraded (see secrets-rotation-log.md), a secret-manager session ran the script for real on bms-4. Steps 1–3 (mint + IAM policy attach) genuinely succeeded — the OAuth2 client was created (clientId=EU.0ff9b7d0a01b84cf) and the IAM policy attached and was confirmed via GET /v2/iam/policy to grant exactly dedicatedServer:apiovh:* on urn:v1:eu:resource:dedicatedServer:*, per the documented least-privilege scope above. But Step 4’s verify call — Bearer GET /me — returned 403, so the script treated the whole run as failed and (safely, but incorrectly) deleted the freshly minted, working client.

Root cause: /me is an account-identity route. The IAM policy this script attaches is deliberately scoped tightly to dedicated-server management only and never grants /me — a correctly-scoped credential will always 403 there. The verify step was probing a route the credential was never meant to have, not testing whether the mint actually worked.

Fix: VERIFY_URL now targets GET /dedicated/server instead of GET /me — a route the attached dedicatedServer:apiovh:* policy genuinely covers (see §6 “List dedicated servers” below; this route is live and returns the account’s service names, e.g. ['ns3087638', 'ns3129867', 'ns3101999']). This is also a more meaningful smoke test than a generic /me liveness check: it proves the credential can do the job it was minted for, not just that it can authenticate. Regression guard: scripts/tests/test_ovh_rotate_cleanup_invariant.py::test_verify_url_targets_a_route_the_attached_iam_policy_covers.

No SOPS write resulted from the halted run — OVH_INFRA_CLIENT_ID/OVH_INFRA_CLIENT_SECRET were never written, the orphaned client was deleted, and the legacy OVH_INFRA_APP_KEY/APP_SECRET/ CONSUMER_KEY triple remains the live credential. A follow-up secret-manager mint retry — no browser step needed, the bootstrap CK is already live — is ready to go once this fix merges.

Bug 3 — mint retry succeeded and stored, but IAM policy silently failed to attach (#5410, 2026-08-04)

With the VERIFY_URL fix (#5463) merged, a secret-manager session re-ran SKIP_CLEANUP=1 bash scripts/rotate/ovh-api-credentials.sh on bms-4. This time the run completed end-to-end and exited 0: a new client (clientId=EU.4c99e13de90e293b) was minted, Bearer GET /dedicated/server verify passed 200, and the payload was dispatched to sops-sync-receiver.yml (workflow run succeeded) — OVH_INFRA_CLIENT_ID/OVH_INFRA_CLIENT_SECRET are now live in secrets/ovh-api.env.sops on main (commit 74a2050f).

However, the iam-policy step’s own log line was a WARN, not an ok:

WARN: iam-policy classic v2 attach returned 409000 -- new client EU.4c99e13de90e293b has NO
PERMISSIONS yet; attach manually via OVH Manager ...

Root cause: the script names the IAM policy ${APP_NAME}-policy, and APP_NAME is p24-infra-rotate-$(date -u +%Y-%m-%d) — date-based, not run-unique. The previous halted attempt (Bug 2, same day) had already created a policy named p24-infra-rotate-2026-08-04-policy (still correctly scoped, but bound to the identity of the client that attempt later deleted). OVH policy names must be unique per account, so this run’s POST /v2/iam/policy for the same name returned 409 Conflict instead of creating a new policy for the new client’s identity. The script logs a WARN on any non-2xx here (deliberately best-effort, see the IAM_POLICY_CAVEAT header comment) and continues rather than failing — which is reasonable in isolation, but combined with Bug 2’s fix it created a second, more subtle failure mode: the run now looks fully successful (exit 0, verify 200) while the new credential has no custom write policy.

Confirmed via a read-only GET /v2/iam/policy (safe, no secret values) — only 3 policies exist on the account: ovh-default (read-only, account-level urn:v1:eu:identity:account:kr40258-ovh), ovh-role-admin (read-only, admin group), and the orphaned p24-infra-rotate-2026-08-04-policy (id 29d24d12-3d64-4ce0-a13a-24605b88d469, still pointing at the deleted client oauth2-EU.0ff9b7d0a01b84cf). No policy grants dedicatedServer:apiovh:* to the new client’s identity (urn:v1:eu:identity:credential:kr40258-ovh/oauth2-EU.4c99e13de90e293b). Step 4’s verify passed only because GET /dedicated/server happens to be a read covered by the broad, pre-existing ovh-default account-level policy — it does not exercise the write scope the credential actually needs (§3 dedicated-server reboot; §2 Valkey password rotation is a separate credential family, OVH_APPLICATION_*, and is not part of this credential’s scope at all — see the corrected note above this section).

Net effect: OVH_INFRA_CLIENT_ID/CLIENT_SECRET are stored and can authenticate, but are very likely not yet usable for their intended purpose. Do not run the §3 smoke test or set SKIP_CLEANUP=0 against this credential until the policy gap is closed.

Fix needed before the next attempt (dev-coder scope, not done here — same “stop and comment, don’t improvise a further live fix” rule as Bug 2):

  1. Make the IAM policy name run-unique (e.g. suffix with ${NEW_CLIENT_ID} or a timestamp instead of just the date), so a same-day retry never collides with a prior attempt’s leftover policy.
  2. Treat a non-2xx iam-policy response as a hard failure (or at minimum a distinct, loudly logged exit code) rather than a WARN-and-continue, since Step 4’s verify route is too broad to reliably catch a missing write policy — the current design let a real permissions gap silently pass as a “successful” run.
  3. As a one-time manual cleanup (secret-manager or sys-admin, read/write IAM only, not a script change): either delete the orphaned p24-infra-rotate-2026-08-04-policy and create a fresh correctly-named one for EU.4c99e13de90e293b, or PUT/reuse it with the new client’s identity — then re-verify with a route the policy actually covers requiring a write permission (e.g. a harmless PUT/no-op against a real dedicated-server route, not just a GET) before smoke-testing.

Bug 3/4 fix — run-unique policy name + hard-fail on attach failure (#5410, dev-coder)

Items 1 and 2 from Bug 3’s recommendation above are now implemented:

  1. Run-unique policy name. The IAM policy attached in Step 3 is now named "${APP_NAME}-${NEW_CLIENT_ID}-policy" instead of "${APP_NAME}-policy". NEW_CLIENT_ID is the freshly minted OAuth2 client’s own clientId, which OVH guarantees is globally unique per mint — so no two runs, same day or otherwise, can ever collide on this name again. (APP_NAME alone stays date-based and unchanged for the OAuth2 client’s own display name/description, which does not need to be unique — only the policy name does.)
  2. Hard failure on attach error. The classic-mode IAM policy-attach branch (the one that hit the 409 above) now calls fail 2 "POST v2/iam/policy (classic v2 signing) returned ${IAM_RESP} ..." on any non-2xx response, exactly mirroring the oauth2-mode branch immediately below it, which already hard-failed the same way. A mint whose policy can’t be attached now stops the run and reports failure — it can never again exit 0 with a credential minted, stored, but permission-incomplete.

Both changes are covered by new regression tests in scripts/tests/test_ovh_rotate_cleanup_invariant.py: test_iam_policy_name_includes_new_client_id_for_run_uniqueness and test_classic_iam_policy_attach_hard_fails_on_non_2xx (plus “guard the guard” scanner-sanity tests for both, following the same pattern as the Bug 1/2 tests already in that file).

Item 3 (orphaned policy cleanup) — deliberately not done here. The orphaned p24-infra-rotate-2026-08-04-policy (id 29d24d12-3d64-4ce0-a13a-24605b88d469) identified in Bug 3 above is a read-only-role change (secret-manager/sys-admin IAM read/write), out of dev-coder scope, and — per the credential used for the mint not carrying a confirmed DELETE /v2/iam/policy/{id} grant — not safe to attempt blind from this session. It remains harmless in place: it references only the identity URN of the already-deleted client EU.0ff9b7d0a01b84cf (GET /me/api/oauth2/client confirmed empty after that client’s deletion, per the secret-manager comment on #5410), so it grants nothing to anything live, and the new run-unique naming scheme (fix 1 above) guarantees no future run will ever collide with it again. No action required unless an operator wants tidier GET /v2/iam/policy output.

Once this fix merges, a secret-manager mint retry against the now-permission-complete client (or a fresh mint if the current EU.4c99e13de90e293b needs to be superseded rather than repaired in place) can proceed straight to the §3 smoke test before any SKIP_CLEANUP=0 legacy-key revoke.

Round 5 (2026-08-05) — a new, distinct blocker at each stage; still not migrated

Picked up after PR #5476 (Bug 3/4 fix) merged. Two separate findings, both live-confirmed on bms-4:

Finding A — AUTH_MODE selection trusts a stored credential’s presence, not its health. Because secrets/ovh-api.env.sops already held OVH_INFRA_CLIENT_ID/CLIENT_SECRET from the round-4 partial success (the inert EU.4c99e13de90e293b client — mint succeeded, IAM policy attach did not, per Bug 3 above), Step 1’s if [[ -n "$CUR_CLIENT_ID" ... ]] picked AUTH_MODE=oauth2 on a plain re-run — --dry-run confirmed this before any live call was made. In oauth2 mode the script bootstraps the mint call using a Bearer token from the existing (inert) credential, never touching the working classic OVH_APPLICATION_* path at all. A live run then failed exactly as predicted: POST /me/api/oauth2/client -> 403 {"unauthorizedActionsByIAM":"account:apiovh:me/api/oauth2/client/create"} — the inert client’s only permissions come from the account-wide ovh-default read-only policy, which does not cover self-minting. Fix applied this round (secret-manager, SOPS-only, PR #5481, merged): removed the two dead OVH_INFRA_CLIENT_ID/CLIENT_SECRET keys from secrets/ovh-api.env.sops, restoring the classic OVH_INFRA_APP_KEY/APP_SECRET/CONSUMER_KEY triple (untouched throughout) as the only recognised shape, so the next run correctly picks AUTH_MODE=classic. No OVH-side state changed by this fix — SOPS-only. This is a data-hygiene workaround, not a code fix — the underlying AUTH_MODE selection logic in the script still cannot distinguish “existing oauth2 credential, working” from “existing oauth2 credential, permission-less” and will make the same wrong choice again if a future partial-success run leaves another inert credential in SOPS. Recommended follow-up (dev-coder, not done here): before trusting AUTH_MODE=oauth2, have the script do a cheap self-check (e.g. try GET /v2/iam/policy scoped to the stored identity, or just attempt the mint and fall back to classic bootstrap on a 403 specifically) rather than an unconditional presence check.

Finding B (Bug 6) — the run-unique policy name from the Bug 3/4 fix breaks on OVH’s own client-ID format. With AUTH_MODE=classic correctly selected, the mint itself succeeded (clientId=EU.53581cd6e0d47ad2) — but the IAM policy attach then failed with a new error, HTTP 400, not the 409 collision Bug 3/4 fixed:

{"class":"Client::BadRequest::INVALID_FORMAT","message":"Policy name is not formatted properly,
expected only alphanumeric characters and '-', '/', '_', '+'"}

Root cause: POLICY_NAME="${APP_NAME}-${NEW_CLIENT_ID}-policy" (the Bug 3/4 fix) embeds NEW_CLIENT_ID verbatim, but every OVH OAuth2 clientId OVH has issued so far in this migration has the form EU.<hex> — a literal dot — and OVH’s IAM policy-name validator does not allow dots (only [A-Za-z0-9\-/_+]). The Bug 3/4 fix (uniqueness) and this bug (format) are in tension: the whole point of suffixing with NEW_CLIENT_ID was to use an OVH-guaranteed-unique token, but the only such token readily available (the clientId) is not policy-name-safe as-is. The hard-fail from the same PR #5476 worked exactly as intended here — the run stopped cleanly with exit 2 instead of silently reporting success with a permission-incomplete credential, unlike Bug 3.

Confirmed via manual reproduction (same classic signing, same body) that a sanitized name — same string with . removed — is accepted by the same endpoint 200/201. Recommended fix (dev-coder, not done here): sanitize NEW_CLIENT_ID before interpolating it into POLICY_NAME, e.g. POLICY_NAME="${APP_NAME}-${NEW_CLIENT_ID//./-}-policy" (replace . with -, preserving uniqueness and readability) rather than stripping it, since collapsing multiple runs’ dots to nothing could reintroduce a collision risk.

Cleanup performed this round (secret-manager, live OVH calls, no code/SOPS change): the policy-less orphaned client from this round’s mint attempt (EU.53581cd6e0d47ad2) was deleted (DELETE /me/api/oauth2/client/{id} -> 200, confirmed via a follow-up empty check) — same pattern as the Bug 1/2 halt’s cleanup. The previous round’s inert client (EU.4c99e13de90e293b, now orphaned from SOPS too by Finding A’s fix) was not deleted — attempted, but blocked by this session’s own tooling; left as a harmless, permission-less, no-longer-SOPS-referenced client for a future pass to tidy up. The orphaned IAM policy from Bug 3 (29d24d12-3d64-4ce0-a13a-24605b88d469) remains untouched, per the standing guidance above (harmless, no future collision risk). A separate, previously undocumented client (EU.5a27e61beeb6e4b9, “p24-infra-claude-oauth2”) was found live with a genuinely working custom policy (p24-infra-rotate-policy, action:"*" on 3 named dedicated servers + 1 resource group) — this is the manually-minted duplicate mentioned in an earlier #5410 comment as pending manual deletion by a human via the OVH Manager; its secret was never persisted anywhere this session can reach, so it cannot be adopted as the SOPS credential even though it currently works.

State after round 5: secrets/ovh-api.env.sops holds only the classic OVH_INFRA_APP_KEY/ APP_SECRET/CONSUMER_KEY triple (still the sole working credential) — no OVH_INFRA_CLIENT_ID/ CLIENT_SECRET in SOPS at all right now. Still Tier 3. Two code-level bugs identified, neither fixed in this round (dev-coder scope, this round stayed within secret-manager’s SOPS-write + verification authority plus routine OVH-side orphan cleanup). Next attempt needs both the AUTH_MODE health-check (Finding A) and the policy-name sanitization (Finding B) before a mint is likely to succeed end-to-end without another manual SOPS intervention first.

Findings A and B fixed (#5489, landed separately from the #5410 review below): the AUTH_MODE health probe (oauth2_credential_healthy()) and the POLICY_NAME dot-sanitization (${NEW_CLIENT_ID//./-}) described as “Finding A”/“Finding B (Bug 6)” above are both live on main as of #5489 — regression-covered by test_authmode_selection_health_probes_stored_oauth2_credential and test_iam_policy_name_sanitizes_client_id_dots_for_ovh_validator in scripts/tests/test_ovh_rotate_cleanup_invariant.py. Noted here because #5489 landed independently of the #5410 review iteration below and this page had not yet been updated to say so — readers should not conclude from “Round 5 … still not migrated” above that these two bugs remain open.

#5410 review iteration 1 (this PR) — 8 findings addressed, 2 deferred, 1 already fixed

A comprehensive review of the script (independent of the live-attempt rounds above) found 11 issues. What was already fixed (found already live on main before this PR started, via #5489 — see the note above): the dot-sanitized POLICY_NAME and the AUTH_MODE health probe. What this PR adds:

  1. fail() self-cleanup of an orphaned client. Previously fail() only removed local temp files — an OVH client minted by Step 2 that later failed (IAM attach, verify, dispatch) was left dangling on the account every time, requiring manual discovery + deletion (the pattern behind clientIds EU.0ff9b7d0a01b84cf, EU.53581cd6e0d47ad2 in the rounds above). fail() now best-effort deletes $NEW_CLIENT_ID (if the run got that far) using whichever signing credential the current AUTH_MODE has in scope, and only logs a WARN — never changes the original exit code.
  2. CLEANUP_TARGET=legacy. sops-sync-receiver.py never deletes the old OVH_INFRA_APP_KEY triple, so after a successful migration SOPS holds both credential shapes and ordinary AUTH_MODE selection will always pick oauth2 from then on — making Step 8’s legacy-cleanup branch permanently unreachable through a normal invocation. CLEANUP_TARGET=legacy (paired with SKIP_CLEANUP=0) forces classic auth regardless of the stored oauth2 credential’s health, so the documented “smoke-test, then revoke the legacy AK/AS/CK” workflow can actually complete after the first run. See the CLEANUP_TARGET header comment in the script for the full design rationale.
  3. Smoke-test plan corrected to §3-only (see the corrected bullet earlier in this section) — §2 Valkey rotation authenticates with the unrelated OVH_APPLICATION_* family and was never a valid verification step for OVH_INFRA_CLIENT_ID.
  4. IAM_RESP double-output bug. ovh_signed_request_v2 uses curl -sf -o /dev/null -w "%{http_code}" — -f still writes the http_code via -w even on a non-2xx response, only the body is suppressed. The call site’s $(... || echo "000") ran both the function’s own output AND the echo fallback on any non-2xx, concatenating into artifacts like the 409000 seen in round 4’s logs. Fixed by moving the fallback outside the substitution ($(...) || true then "${IAM_RESP:-000}") so the real captured code is preserved and set -e can’t kill the script mid-diagnostic.
  5. RECV_KEY_TMP added to fail()’s cleanup list. It holds decrypted AGE_KEY_SOPS_SYNC_RECEIVER private key material and was previously only removed on the script’s normal-exit path.
  6. DRY_RUN argument validation. Any first argument other than the literal --dry-run previously fell through silently into a live mint. Now rejected up front with exit 64 (EX_USAGE).

Deferred (explicitly out of scope for this pass):

  • Finding 9 — SKIP_CLEANUP async-dispatch race. gh workflow run sops-sync-receiver.yml is fire-and-forget; Step 8 can revoke the legacy credential before the async workflow has actually landed the new one in SOPS. Documented as a code comment above Step 8 for a future iteration — not fixed here.
  • Item 10 — OVH_APPLICATION_CONSUMER_KEY write-route preservation note — added to docs/sops-templates/monitoring.keys instead (doc-only, see that file).

Regression tests added to scripts/tests/test_ovh_rotate_cleanup_invariant.py, following the existing static-scan “guard the guard” pattern: test_dry_run_rejects_unrecognized_first_argument (+ exit-code usage guard) and test_fail_function_self_cleans_orphaned_oauth2_client, plus scanner-sanity coverage for the CLEANUP_TARGET override, the IAM_RESP fix, and the RECV_KEY_TMP cleanup addition.

#5410 review iteration 2 — 3 findings addressed

A second-pass adversarial review of PR #5492 (iteration 1, above) found three more issues.

1. [HIGH] CLEANUP_TARGET=legacy was logically broken — before/after.

  • Before (iteration 1): CLEANUP_TARGET=legacy forced classic AUTH_MODE from inside the existing AUTH_MODE selection if/elif chain. But classic AUTH_MODE still ran the FULL mint pipeline (Steps 2-7: mint a new client, attach an IAM policy, verify, dispatch to SOPS) before Step 8’s revoke ran. The documented operator workflow — “smoke-test the new oauth2 credential, then run CLEANUP_TARGET=legacy to revoke the legacy one” — therefore actually discarded the just-smoke-tested credential and replaced it with a fresh, unverified one, without even logging the discarded client’s id. This recreated exactly the class of orphan the fail() self-cleanup feature (iteration 1, finding 1 above) exists to prevent.
  • After (iteration 2): CLEANUP_TARGET=legacy is now its own early-exit short-circuit, placed right after the OVH_TARGET guard and before Step 0 — structurally separate from AUTH_MODE selection, which no longer references CLEANUP_TARGET at all. It never mints, never attaches an IAM policy, never re-verifies, and never dispatches a new credential to SOPS. It reads the existing OVH_INFRA_CLIENT_ID/CLIENT_SECRET (failing loudly, fail 1, if absent — cleanup-only mode has nothing to protect without a completed migration) plus the legacy classic triple, then calls the same revoke logic Step 8 uses (factored into a shared revoke_legacy_classic_application() function), signed with the admin-scoped OVH_APPLICATION_* bootstrap credential.
  • Operator-facing change: SKIP_CLEANUP=1 (the default) combined with CLEANUP_TARGET=legacy is now a deliberate, clearly-logged no-op (exit 0 with an explanatory message) instead of running the (broken) full pipeline anyway — cleanup-only mode’s entire purpose is the action SKIP_CLEANUP guards. The correct invocation remains CLEANUP_TARGET=legacy SKIP_CLEANUP=0 bash rotate/ovh-api-credentials.sh, but it now actually does what that invocation has always claimed to do.
  • This is the second time this flag’s semantics changed (iteration 1 introduced it, iteration 2 fixed a bug in that introduction) — operators should re-read the CLEANUP_TARGET header comment in the script before assuming prior knowledge of its behavior still applies.

2. [HIGH, docs-only — not fixed in code] Self-cleanup DELETE calls likely lack IAM permission.

Per an earlier comment on issue #5410 (from a human who manually minted EU.5a27e61beeb6e4b9): neither the admin bootstrap CK nor a client’s own bearer token has the account:apiovh:me/api/oauth2/client/delete IAM action needed to revoke an OAuth2 client via the API. This affects all three DELETE /me/api/oauth2/client/${id} call sites in this script — both branches of fail()’s iteration-1 self-cleanup, and Step 8’s pre-existing oauth2-mode cleanup branch. All three are already best-effort/non-fatal (WARN-and-continue), so a 403 does not break a run — it just means “self-cleanup” will very likely not actually clean up in practice. Accepted as a known, harmless limitation for now — not something to keep chasing with further code changes; multiple harmless orphaned OAuth2 clients have accumulated across this saga’s rounds and are left in place as documented. This is a live-credential/IAM decision, not a code fix: granting account:apiovh:me/api/oauth2/client/delete (and ideally account:apiovh:iam/policy/delete, to also let the script clean up orphaned IAM policies like 29d24d12-3d64-4ce0-a13a-24605b88d469) to the admin bootstrap CK would close this gap. Flagged here as a possible future one-time follow-up only if orphan accumulation becomes a real operational problem — not urgent today. Each call site now carries an in-line comment to this effect so a future reader does not mistake a WARN: ... failed log line for a new bug.

3. [LOW-MEDIUM, test quality] fail() self-cleanup guard tightened.

test_fail_function_self_cleans_orphaned_oauth2_client previously only checked that the NEW_CLIENT_ID guard substring and the DELETE call substring both existed somewhere in fail()’s body — not that the DELETE call was textually inside the guard’s own if/fi span. A DELETE call placed unconditionally elsewhere in fail() (bypassing the guard) would have passed the old check. The scanner now extracts the guard’s own if/fi block (balanced token counting, handles nesting) and requires the DELETE call to fall within it — see _extract_if_fi_block/_fail_self_cleans_orphaned_client in scripts/tests/test_ovh_rotate_cleanup_invariant.py, with a dedicated guard-the-guard test (test_scanner_rejects_delete_call_outside_the_guard_block) proving the tightened check actually rejects an unguarded DELETE. The test file’s module docstring also now notes explicitly that IAM-permission-level correctness of any DELETE/POST/GET call (see finding 2 above) is out of scope for this static scanner and must be verified live — the scanner catches control-flow and string-construction bugs, not authorization-model bugs.

Round 6 (2026-08-05) — SUCCESS: migration complete and independently verified

The first live attempt run against main@651a8984 (review iteration 2, PR #5494, merged — the first attempt preceded by a full 3-iteration review→fix→re-review cycle rather than a single-bug fix per round). Pre-flight: secrets/ovh-api.env.sops held only the classic OVH_INFRA_APP_KEY/APP_SECRET/CONSUMER_KEY triple (round 5’s SOPS cleanup, PR #5481, was still in effect — no stale oauth2 credential to trip the AUTH_MODE health probe). --dry-run confirmed AUTH_MODE=classic as expected.

Live run (bash scripts/rotate/ovh-api-credentials.sh, no env overrides, mint-only per SKIP_CLEANUP=1 default) on bms-4:

apply ok clientId=EU.7b9eafea413fb063
iam-policy ok (classic v2 signing, first-run migration)
verify start endpoint=https://eu.api.ovh.com/1.0/dedicated/server
verify ok http_code=200
dispatch ok  (sops-sync-receiver.yml run 30969182799, conclusion=success)
cleanup skipped (SKIP_CLEANUP=1, default)

No WARN, no non-2xx, no fallback path taken anywhere in the run — every bug found and fixed across rounds 1-5 and the two review iterations stayed fixed under a real live run. OVH_INFRA_CLIENT_ID/ OVH_INFRA_CLIENT_SECRET landed on main at commit a3026658.

Independent verification performed (per the standing lesson of this whole saga — never trust the script’s own exit code alone):

  1. IAM policy, re-queried separately. Signed a fresh GET /v2/iam/policy with the admin bootstrap credential (not the script’s cached session) directly on bms-4. Confirmed policy p24-infra-rotate-2026-08-05-EU-7b9eafea413fb063-policy (id 904f7aac-0bae-4f84-a5aa-f9f6e7404085) exists, identities: ["urn:v1:eu:identity:credential:kr40258-ovh/oauth2-EU.7b9eafea413fb063"], resources: [{"urn":"urn:v1:eu:resource:dedicatedServer:*"}], permissions.allow: [{"action":"dedicatedServer:apiovh:*"}] — genuinely attached to the new client’s own identity URN, unlike round 4’s silent 409 no-op.
  2. Smoke test with the credential’s own fresh Bearer token, not the script’s. Read OVH_INFRA_CLIENT_ID/SECRET back out of SOPS, requested a brand-new access_token from POST https://www.ovh.com/auth/oauth2/token, then called:
    • GET /dedicated/server → 200, ["ns3087638...","ns3101999...","ns3129867...”] (all 3 bms dedicated servers, matching the account).
    • GET /dedicated/server/{name}/boot → 200 — the stronger smoke test recommended by this page’s own §3 guidance (“closer to actual usage than a generic list call”), without performing an actual reboot (that remains a separate, more invasive step reserved for a human/later decision).

State after round 6: OVH_INFRA_CLIENT_ID/OVH_INFRA_CLIENT_SECRET are live, IAM-attached, and smoke-tested in secrets/ovh-api.env.sops. Tier 1 — no browser step needed for future rotations (see secret-rotation-access-matrix.md). Legacy OVH_INFRA_APP_KEY/APP_SECRET/CONSUMER_KEY triple was not touched — SKIP_CLEANUP=1 (default), no CLEANUP_TARGET set, exactly as scoped for a mint-only run. Its revocation is a deliberate, separate follow-up, not performed in this pass:

CLEANUP_TARGET=legacy SKIP_CLEANUP=0 bash rotate/ovh-api-credentials.sh

No orphaned OAuth2 clients were created this round (unlike rounds 4/5) — the run succeeded on the first attempt, so fail()’s self-cleanup path was never exercised. The pre-existing orphaned IAM policy from round 4 (29d24d12-3d64-4ce0-a13a-24605b88d469, points only at the already-deleted EU.0ff9b7d0a01b84cf) remains untouched and harmless, per standing guidance.

6. Listing Available Operations (Discovery)

Use these read-only calls to discover resource names and task state without side effects.

List Valkey / private database instances

instances = client.get("/hosting/privateDatabase")
print(instances)
# Expected: ['kr40258-001']

List dedicated servers

servers = client.get("/dedicated/server")
print(servers)
# Expected: ['ns3087638', 'ns3129867', 'ns3101999']  (bms-2, bms-3, bms-4)

List pending tasks for a server

SERVICE_NAME = "ns3101999"  # bms-4
tasks = client.get(f"/dedicated/server/{SERVICE_NAME}/task")
for task_id in tasks:
    detail = client.get(f"/dedicated/server/{SERVICE_NAME}/task/{task_id}")
    print(f"  {task_id}: {detail['function']} — {detail['status']}")

Check server info (boot mode, IPs, state)

info = client.get(f"/dedicated/server/{SERVICE_NAME}")
# Fields of interest: bootId, state, ip, name
# Do NOT print the full dict if it contains sensitive data — extract only needed fields
print(f"state={info['state']}, bootId={info['bootId']}, ip={info['ip']}")

Quick Reference

OperationEndpointCK route needed
List Redis instancesGET /hosting/privateDatabaseGET /hosting/privateDatabase
Change Redis passwordPOST /hosting/privateDatabase/{id}/user/{user}/changePasswordsame POST route
List dedicated serversGET /dedicated/serverGET /dedicated/server
Reboot serverPOST /dedicated/server/{name}/rebootPOST /dedicated/server/*/reboot
Set boot modePUT /dedicated/server/{name}PUT /dedicated/server/*
List boot optionsGET /dedicated/server/{name}/bootGET /dedicated/server/*/boot
Poll taskGET /dedicated/server/{name}/task/{id}GET /dedicated/server/*/task/*
  • docs/playbooks/secret-manager.md — SOPS+age credential operations
  • docs/playbooks/sops-windows-crlf.md — Windows SOPS write safety
  • docs/playbooks/secret-rotation-access-matrix.md — Tier 1/2/3 rotation decision tree
  • docs/playbooks/static-api-key-incident-rotation.md — emergency rotation procedure

Graph View

Backlinks