p24-infra Docs

testing-standards

4 min read

Playbook — Testing Standards

Source of truth for the dev-tester role and all test-related decisions in the p24-infra ecosystem.

Core Policy: No Database Mocks

Rule: Tests must run against a real (test) database. No in-memory fakes, no mock objects substituting for database connections.

Why this rule exists: In Q1 2025, mock-based tests passed while a production migration failed — the mock did not enforce the schema constraints that a real DB does. The failure was caught in production. This rule prevents that class of defect.

How to comply: Use a dedicated test database (Supabase test project, local Postgres with test schema, or the subsystem’s test fixtures). Never pass a mock or stub where a real DB connection is expected.

Test Runner by Subsystem

SubsystemLanguageTest runnerCommand
monitoring/exporters/queue-exporter/Pythonpytestpytest monitoring/exporters/queue-exporter/tests/
audit-engine/Pythonpytestcd audit-engine && pytest
infra-src/waha-router/TypeScriptvitestcd infra-src/waha-router && npm test
portal/TypeScriptvitestcd portal && npm test
portal/ E2ETypeScriptPlaywrightcd portal && npx playwright test

Run the correct test runner for the subsystem being changed. Never run all runners for a single-subsystem change.

Coverage Expectations

SubsystemTarget line coverageNotes
queue-exporter80%Core metrics functions must be covered
audit-engine75%Action handlers + scheduler critical path
waha-router80%HMAC verification + routing logic mandatory
portal60%API routes + auth helpers; UI components lower priority

Coverage is a floor, not a goal. A 100%-covered test suite that never makes a real DB call is worse than a 60%-covered suite that tests the integration boundary correctly.

Test Types and When to Use Each

Unit tests

  • Test a single function or class with real dependencies (no mocks)
  • Use for: pure computation, data transformation, validation logic
  • Do NOT use for: database operations, HTTP calls, file I/O (use integration tests)

Integration tests

  • Test a component against its real dependencies (real DB, real API endpoint)
  • Use for: database queries, Supabase RPC calls, exporter metric output
  • Setup: real test DB credentials from environment variables; never hardcode

E2E tests (Playwright — portal only)

  • Test user flows against a running dev server
  • Cover: auth flow, key CRUD operations, critical dashboard paths
  • Do NOT: test every UI state; focus on paths that, if broken, block a user from core tasks

Writing New Tests

When adding new logic, write tests before or alongside the implementation:

  1. Identify the real dependencies (DB table, external API, file)
  2. Write the test against those real dependencies using test credentials from env
  3. Run the test against a real test environment before committing
  4. Never commit a test that passes only because a dependency is mocked

Test file naming

  • Python: tests/test_[module_name].py
  • TypeScript: [filename].test.ts co-located, or tests/[filename].test.ts

Environment for tests

All subsystems read test credentials from environment variables. Use:

# For local development
export SUPABASE_URL=...       # test project URL
export SUPABASE_SERVICE_KEY=... # test project service key

Never hardcode credentials in test files. Never commit .env.test files to git.

What Counts as a Failing Test

A test failure blocks a PR merge. Failures include:

  • Test assertion failed
  • Test errored (uncaught exception, DB connection refused, missing env var)
  • Test runner exited non-zero

Flaky tests: If a test fails intermittently due to timing or ordering issues, fix it. Do not mark it as skipped or increase timeout without root-cause analysis.

CI/CD Integration

All PRs targeting main run tests via GitHub Actions. The workflow:

  1. Sets up Python 3.12 / Node 22 per subsystem
  2. Reads test DB credentials from GH Secrets
  3. Runs the relevant test suite
  4. Fails the PR check if coverage drops below the threshold

Local test runs use the same commands as CI. If it passes locally but fails in CI, the difference is almost always environment (missing env var, different DB state).

Adding a New Subsystem to Testing

When a new component is added to p24-infra:

  1. Create a tests/ directory in the subsystem root
  2. Add a pytest.ini (Python) or vitest.config.ts (TypeScript) config
  3. Add the test command to the CI workflow (.github/workflows/)
  4. Update this playbook with the new subsystem row in the table above
  5. Set an initial coverage target in this playbook

Graph View

Backlinks