p24-infra Docs

whatsup-wa-app-api-operational-gaps

7 min read

Playbook: p24-wa-a / whatsup-android-chat-puller — CF Worker operational gaps

Covers three related, recurring-risk gaps discovered together during a 2026-08-19/20/27/28 multi-day session testing the whatsup-android-chat-puller multi-device redundancy feature (#162/#163) end-to-end. All three concern the p24-wa-app-api Cloudflare Worker and its Supabase project (mwkqmgadqnkkihjdeqsi, the shared Ecotrans Supabase project).

Gap 1 — p24-wa-app-api has no CI deploy; merged code silently doesn’t run

Symptom: A merged PR to whatsup-android-chat-puller’s dev branch touching infra-src/wa-app-api/** does not take effect on the live Worker. Bugs believed fixed keep reproducing identically after merge, because the deployed Worker is still running old code.

Confirm:

gh api repos/radieu/whatsup-android-chat-puller/contents/.github/workflows --jq '.[].name'
# No workflow here deploys infra-src/wa-app-api — build-apk.yml/release.yml only build the APK.

Fix — manual deploy required after every merge touching infra-src/wa-app-api/:

# On bms-4 (as claude-runner — has gh auth + the repo checkout):
ssh root@54.36.123.110 "su - claude-runner -c '
  cd /tmp/<scratch-dir>/whatsup-android-chat-puller
  git fetch origin dev && git reset --hard origin/dev
  cd infra-src/wa-app-api && npm install
'"

Then deploy (from Windows, credentials extracted fresh in the same PowerShell call per this repo’s safe-extraction pattern — see docs/playbooks/sops-windows-crlf.md sibling patterns):

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$decrypted = sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\cloudflare.env.sops
$cfToken = ($decrypted | Where-Object { $_ -like "CF_WORKERS_API_TOKEN=*" }) -replace '^[^=]*=',''
# cloudflare.env.sops is "created, not yet distributed" per this repo's CLAUDE.md §Secrets —
# if the token above is empty/stale, fall back to the duplicated copy in monitoring.env.sops
# (same key name, still the live source until PR-D/PR-G land).
 
# Build a base64-encoded remote script (never pass secrets as a bare SSH CLI arg — see
# feedback_ssh_command_quoting_base64 pattern) that sets, inside the script itself:
#   $env:CLOUDFLARE_API_TOKEN  = "<value of $cfToken above>"
#   $env:CLOUDFLARE_ACCOUNT_ID = "acc5b2c956dd488265003364e2c7f84a"   # see note below — not a secret
# then runs, from infra-src/wa-app-api on the synced bms-4 checkout:
#   npx wrangler deploy
# Transfer the base64 blob over SSH, decode + execute remotely, then wipe the temp script file
# (truncate -s 0 -- "$f" && rm -f -- "$f" — shred is blocked by the disk-destruction guard, #7826).

Why --account-id (or CLOUDFLARE_ACCOUNT_ID env var) is required explicitly: both available CF tokens (CF_WORKERS_API_TOKEN, CF_API_TOKEN) lack the User Details:Read permission needed for wrangler’s interactive /memberships account lookup — wrangler tail fails outright for this reason (no workaround found); wrangler deploy/wrangler secret put succeed once the account ID is passed explicitly, skipping that lookup. (acc5b2c956dd488265003364e2c7f84a is a Cloudflare account identifier, not a credential — it carries no access on its own without a valid API token, same class as an AWS account ID; already referenced openly elsewhere in this repo’s playbooks.)

Prevention (not yet done): add a deploy-wa-app-api.yml GH Actions workflow, needs: test, triggered on push to dev/main touching infra-src/wa-app-api/**, running wrangler deploy with CLOUDFLARE_API_TOKEN from a GH Secret. File a whatsup-android-chat-puller issue for this if not already tracked.

Gap 2 — WASABI_ADMIN_ACCESS_KEY/WASABI_ADMIN_SECRET_KEY missing from SOPS entirely

Symptom: /provision-storage and /ingest (S3 admin operations) fail; push-secrets.sh’s REQUIRED_KEYS list and wrangler.toml’s doc comment both list these keys as sourced from secrets/whatsup.env.sops, but a full key-name scan of that file shows they were never actually present — the Worker’s live secret was set once via a bare wrangler secret put and never captured into SOPS.

Confirm (safe — key names only):

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\whatsup.env.sops |
  ForEach-Object { $_.Split('=')[0] } | Select-String -Pattern "WASABI_ADMIN"
# Empty result = the gap is still present.

Fix — mint fresh credentials for the existing p24-wasabi-admin IAM identity (do NOT create a new IAM user — this one already carries the correct WasabiAdministratorAccess managed policy):

import boto3, os
iam = boto3.client('iam', endpoint_url='https://iam.wasabisys.com',
    aws_access_key_id=os.environ['WASABI_ROOT_ACCESS_KEY'],   # true account-root key, from Wasabi Console
    aws_secret_access_key=os.environ['WASABI_ROOT_SECRET_KEY'],
    region_name='us-east-1')
# Confirm the identity actually exists before minting a key against it — a typo'd UserName
# fails with NoSuchEntity, not a silent no-op:
iam.get_user(UserName='p24-wasabi-admin')
# p24-wasabi-admin already has 1 active key (secret unknown/unrecoverable) + WasabiAdministratorAccess.
# Wasabi allows 2 keys/user — create a second rather than a new user:
resp = iam.create_access_key(UserName='p24-wasabi-admin')
# Write resp['AccessKey']['AccessKeyId'] / ['SecretAccessKey'] straight to SOPS — never print.

Store as WASABI_ADMIN_ACCESS_KEY / WASABI_ADMIN_SECRET_KEY in secrets/whatsup.env.sops (scripts/sops-set.ps1), then wrangler secret put both into the Worker (same base64/SSH pattern as Gap 1).

Where to get a Wasabi account-root key if none in SOPS works: console.wasabisys.com → Access Keys → Create New Access Key → owner Root Account (not a sub-user). Root keys are shown once; the CSV download is a live secret — read its two column values into env vars without ever printing them, then delete the CSV file immediately (rm -f, not PowerShell Remove-Item — that cmdlet has hit a sandbox 'e:' is blocked false-positive in this environment on unrelated paths; rm -f via the Bash tool has been reliable).

Known-stale root keys as of 2026-08-20 (do not assume these work — verify with iam.list_users() before relying on them): WASABI_ACCESS_KEY/WASABI_SECRET_KEY and P24_INFRA_WASABI_ACCESS_KEY/P24_INFRA_WASABI_SECRET_KEY in secrets/monitoring.env.sops both returned InvalidAccessKeyId — consistent with the known recurring Wasabi-key-drift pattern (#3366 and related). Re-verify before use rather than assuming SOPS is current.

Gap 3 — IAM_ENC_KEY value in SOPS doesn’t decrypt the live-encrypted data

Symptom: /device-config returns 500. The Worker’s service_get_user_iam_keys-equivalent RPC (p24_wap_get_user_iam_keys) throws Wrong key or corrupt data even with a syntactically correct call — meaning the pgcrypto symmetric key the Worker currently has (from wrangler secret put IAM_ENC_KEY, set independently at some unknown past point) does not match what’s in SOPS secrets/whatsup.env.sops, and neither necessarily matches what originally encrypted the existing rows.

This key cannot be recovered by inspection — pgcrypto has no reveal mechanism and CF Worker secrets are write-only. Do not attempt to “figure out” the right value; treat any confirmed mismatch as requiring rotation + re-encryption (or, if data is already unrecoverable by any available key, deletion + fresh re-provisioning).

Full incident rotation: docs/playbooks/encryption-key-rotation.md (this key is one of the four IAM_ENC_KEY-class Tier 3 static keys covered there — read it before touching this key at all, especially its blast-radius warning: rotation invalidates every currently-provisioned user’s Wasabi IAM keys).

Safe pattern for testing whether an old-key value actually works, without risking a leak (2026-08-20 incident, p24-infra#6561): create a SECURITY DEFINER re-encryption function FIRST, with no secret values in its own DDL, confirm it exists (safe schema query), THEN test it with two deliberately-wrong dummy strings before ever using a real key — confirms the function’s error path does not echo argument values (PL/pgSQL exceptions from this function only show the static function body in CONTEXT, never the bound parameters, which is what makes a subsequent real-value call safe). Never hand-embed a raw secret value into ad-hoc SQL sent through the Supabase Management API /database/query endpoint outside of this pattern — a syntax or name-resolution error (e.g. a typo’d function name) WILL echo the full query text, including any embedded literal, back in the API response.

If the drift is confirmed and old data is unrecoverable: see the resolution steps logged in dev_r_rotation_log (secret_name=IAM_ENC_KEY, 2026-08-28) — delete the affected p24_wap_user_storage rows, clear each real user’s stale Wasabi IAM access key (wa-app-{userId} sub-user, iam.delete_access_key), then call /provision-storage again per user to mint fresh keys under the new IAM_ENC_KEY. /provision-storage’s own idempotency guard (listAccessKeys short-circuit) means the stale-key-deletion step is mandatory — re-calling the endpoint without it just returns provisioned_existing and does nothing.

Follow-up already landed independently (PR 166, whatsup-android-chat-puller): a dual-key read (IAM_ENC_KEY then IAM_ENC_KEY_OLD fallback) was added to p24_wap_get_user_iam_keys callers, making future rotations non-disruptive for rows still encrypted under a prior key. Confirm this pattern is used for any future IAM_ENC_KEY rotation instead of the full stop-the-world re-encrypt-or-delete procedure above.

Escalation

If any of these recur and this playbook’s steps don’t resolve it, escalate via a critical-labeled issue in radieu/p24-infra (per docs/playbooks/static-api-key-incident-rotation.md) — do not attempt ad-hoc credential recovery beyond what’s documented here.

Graph View