p24-infra Docs

ai-processing-register-workflow

6 min read

AI-Logs Register — Email-to-Record Pipeline Architecture

Register: AI-Logs (67597cb44eefb1002c9847d4 in regRecords/registries) Process: aiProcessingLog (process.configs collection, 675abf9acb6b3800253342ae) Org (officeId): 5c752bbca20da35ca1b99083 Email entry point (RETIRED 2026-07-11, see §0): rechnung-AI@integrations-eu.pinbox24.com (Mailgun inbound, domain integrations-eu.pinbox24.com) Last updated: 2026-08-05 (§0 added — #3752; body below otherwise unchanged since 2026-07-02)

0. Current live path (2026-08-05) — read this first

The email pipeline described in §1 below was decommissioned 2026-07-11. Ingestion for this exact register moved to:

Google Drive (production folder, per rechnungen/gutshrifts/standard category)
      │  polled hourly by a scheduleTrigger
      ▼
n8n workflow MUjApqruo6H88ebw ("W4 Direct API Ingestion (Rechnungen/Gutshrifts/Standard)"), bms-4
  Redis singleton lock (w4ingest:lock:rechnungen, #4008) → Google Drive search/download
      │
      ├─ POST /api/auth  (Pinbox24 JWT — credential currently named "TEMP - bridge to
      │     CF Worker 3826", i.e. mid-migration to the p24-auth CF Worker's token-mint
      │     path; observed live 2026-08-05, not yet fully cut over)
      ├─ POST /api/reg/67597cb44eefb1002c9847d4/new     (same body shape as §2 below)
      ├─ POST /api/offices/files/upload
      ├─ PUT  /api/reg/67597cb44eefb1002c9847d4/{id}    (link, same as §1's file-first step)
      └─ POST /api/reg-process/{id}/318                 (advance — the real pipeline's
            equivalent of "process engine kicks off" in §1)
              ↓
        MongoDB w4_db.regRecords — same record model as §2 below
              ↓
        Process engine kicks off: aiProcessingLog (unchanged from §1)

Everything downstream of record creation — the record model (§2), process activities (§3), duplicate-check logic — is unchanged by this cutover; only the trigger changed (email → Drive-polling n8n workflow). Discovered and documented during the #3752 active-liveness-probe investigation (triggering root-cause: #5440). The exact request/response shapes above were confirmed by reading the LIVE workflow via the n8n REST API (read-only), not guessed.

An active synthetic liveness probe for this live path now exists (Leg 1: direct-API, bypassing n8n/Drive; Leg 3: an isolated n8n webhook canary) — see docs/mailgun-e2e-probe-operations.md. Both legs write a tagged _synthetic: true record and deliberately stop before the reg-process/.../318 advance call shown above.

The mailgun-pipeline-exporter service name (bms-1) is now a historical artifact of the retired email leg — not renamed, see the ops doc above for why.

1. Full Pipeline (HISTORICAL — email leg, retired 2026-07-11; kept for incident-archaeology reference)

Gmail/Email client
      │  (real PDF attachment)
      ▼
Mailgun MX (integrations-eu.pinbox24.com)
      │  Inbound route → webhook POST
      ▼
p4-ms-mailgun (mailgun-prod_mailgun_1 on bms-1)
  PORT: internal, exposed as mailgun-api.w4.pinbox24.com via nginx-proxy
  │
  ├─ POST /api/v3/storage/:officeId/mailgunFileHandler → s3-v2-v42-prod
  │     (file-first: upload BEFORE creating register record)
  │     ← { success: true, result: { _id, originalName, size } }
  │
  └─ POST /api/reg/:regid/new → v42-prod (w4 main backend)
        body doc includes: recordMainDocument, originalName, fileSize (all from step above)
              ↓
        MongoDB w4_db.regRecords — new record created
              ↓
        Process engine kicks off: aiProcessingLog

2. Record Model — Key Fields

After the pipeline runs correctly, a register record in w4_db.regRecords has:

FieldValueSet by
regIdObjectId(67597cb44eefb1002c9847d4)API on creation
officeId5c752bbca20da35ca1b99083config
recordData.recordMainDocumentObjectId string of the files collection docmailgun patch (file-first upload)
recordData.originalNamee.g. PROCEDURA OPERACYJNA...v8.pdfmailgun patch (from s3-v2 response)
recordData.fileSizee.g. "215899" (string!)mailgun patch — MUST be string for mqFilter
recordData.aiProc"pdf"integration config
recordData.aiProcInfo"docs_ai.waitingForProcessing"integration config
recordData.costCategory"Rechnung"integration config
recordData.duplicateCountnumber (set by activity 296)process engine
recordData.duplicate"N" or "Y" (set by activity 298/300)process engine

The files collection doc (in w4_db.files) linked via recordMainDocument holds the actual S3 path and metadata.

3. Process Activities — aiProcessingLog

Activities are stored as JSON in w4_db.process.configs. To inspect:

// Inside v42-prod container:
const db = mongoose.connection.db;
const cfg = await db.collection('process.configs').findOne({ _id: new ObjectId('675abf9acb6b3800253342ae') });
console.log(JSON.stringify(cfg.actionsList, null, 2));

Activity 296 — check duplicates (type: trigger)

"filter_by_mq_filter_and_update": {
  "keyName":    { "value": "duplicateCount", "optionType": "defaultText" },
  "mqFilter":   { "value": "originalName='##originalName##' AND fileSize='##fileSize##'", "optionType": "defaultText" },
  "sourceRegId": { "value": "regId", "optionType": "formField" }
}
  • Queries regRecords where originalName and fileSize match the current record’s values (substituted from recordData)
  • sourceRegId → formField → reads regId from current record (the AI-Logs register itself)
  • Sets recordData.duplicateCount = count of matching records

Self-match behaviour: a newly created record always matches itself → duplicateCount ≥ 1. For a unique file, exactly 1 match (itself) is expected and correct.

Why fileSize must be a string: the mqFilter string fileSize='##fileSize##' uses string comparison. If stored as number (e.g. 215899), MongoDB strict type match fails → 0 results → false duplicate.

Activity 297 — if potential duplicate (type: switch)

key "1"        → activity 298 (no duplicate — self-match only)
##isEmpty##    → activity 299 (zero matches — shouldn't happen for valid records)
##isNotEmpty## → activity 299 (more than 1 match → real duplicate)

Activity 298 — no duplicate (type: trigger)

Sets recordData.duplicate = "N" via update_register_record. Process continues to activity 71 — Document passed to docRegistration.

Activity 300 — duplicates to confirm (type: state)

Human review queue. Reached when duplicateCount ≠ 1 (either 0 = file metadata missing, or >1 = real duplicate).

Activity 71 — Document passed to docRegistration

AI processing begins: the system reads recordMainDocument to fetch the PDF from Wasabi S3 and processes it. Status 72 (“Error to manage main documents”) means the PDF was unreachable or invalid — check s3-v2 connectivity and file integrity.

4. s3-v2 API Contract

All responses from s3-v2-v42-prod are wrapped:

{ "success": true, "result": <actual_value> }
{ "success": false, "result": "<error_message>" }

Source: storage.controller.js:109 — res.json({ success: true, result: yield helper(req.body) }).

Any caller that reads the response MUST unwrap: const data = response.result; — not response directly.

The mailgunFileHandler endpoint specifically returns (after the 2026-07-02 patch):

{ "success": true, "result": { "_id": "ObjectId hex string", "originalName": "filename.pdf", "size": 215899 } }

5. Integration Config (integrationsEmails collection)

The integration config for rechnung-AI@integrations-eu.pinbox24.com:

// Query:
db.collection('integrationsEmails').findOne({ emailAddress: /rechnung-AI/i })
// Key fields:
{
  emailAddress: "rechnung-AI@integrations-eu.pinbox24.com",
  officeId: "5c752bbca20da35ca1b99083",
  actionDef: [{
    type: "registerinsert",
    regData: { regid: "67597cb44eefb1002c9847d4" },  // AI-Logs register
    doc: { aiProc: "pdf", aiProcInfo: "docs_ai.waitingForProcessing", costCategory: "Rechnung" }
  }]
}

6. Known Issues and Tech Debt

IssueImpactFix
Runtime patches lost on container restartFull regression to status 300 for all emailsPermanent GitLab MR (see playbook)
No smoke test automationSilent failure goes undetectedn8n daily test workflow
Process config in MongoDB onlyHard to version-control or audit changes to activitiesExport to git as JSON
files collection doc named 7ZwMu8NZP.pdf (random)Filename not preserved in old recordsFixed in new records via originalName field
Status 72 error for non-PDF or corrupted PDFsExpected — not a pipeline bugValidate PDF before upload

7. Improvement Proposals

A. Persistent volume-mount patches (quick win)

Same approach as v42-prod memory fix: bind-mount the patched JS files into containers so they survive recreate:

# In /root/mailgun-prod/docker-compose.yml
volumes:
  - /root/mailgun-prod/patches/integration.helper.js:/app/helper/integration.helper.js:ro
 
# In /root/s3v2-prod/docker-compose.yml
volumes:
  - /root/s3v2-prod/patches/mailgunFileHandler.helper.js:/app/dist/apps/storage/mailgunFileHandler.helper.js:ro

Steps: copy patched files to host patch dirs, add volume mounts, recreate containers.

B. Export process config to git

# On bms-2 (MongoDB PRIMARY):
mongoexport --db w4_db --collection process.configs \
  --query '{"_id": ObjectId("675abf9acb6b3800253342ae")}' \
  --out /tmp/aiProcessingLog-config.json

Commit to docs/pinbox24/process-configs/aiProcessingLog.json. Enables diff-based auditing of activity changes.

C. Smoke test workflow (n8n or cron)

After any mailgun/s3-v2 deploy: send a test email with a datetime-unique filename, wait 30s, query MongoDB, verify originalName/fileSize/recordMainDocument all set and processStatusId != 300. Alert to Discord if regression.

D. Permanent GitLab MR

The cleanest path: commit TypeScript fixes to both GitLab repos, rebuild images, deploy via docs/playbooks/pinbox24-bms1-manual-deploy.md. This removes all hotfix durability risk.

Graph View

Backlinks