Overview — what is this system?
Outside companies send us customer records (an ID, a name, an email). This system receives those records, saves them into our customer table, and gives operators screens to watch the work happen and fix anything that gets stuck.
Post-office flow
- A sender drops a letter in the mailbox — a third party calls our API.
- The clerk stamps it received — the API stores an ingest record and answers accepted within a fraction of a second. The sender never waits for delivery.
- A courier delivers it immediately — the fast-lane worker writes the customer into the customer table, usually in under a second.
- Every 2 minutes a supervisor checks for letters nobody picked up — the sweeper re-queues anything the courier missed.
- Once a night an auditor counts everything — aggregate-stats records totals for the Overview page.
Guarantees
- Nothing is lost. Every accepted request is stored first. Every accepted request ends as DONE or FAILED — never silently dropped.
- Newest wins. If two updates for the same customer race, the newer request wins and the older one is skipped as stale.
Words used in this guide
| Word | Meaning |
|---|---|
| Customer | The current truth: one row per externalId with name and email. |
| Ingest (inbox row) | One accepted request, with a status: PENDING, PROCESSING, DONE, FAILED. |
| Queue | A waiting line of jobs. customer-fast-lane = deliver now, customer-cron = scheduled work. |
| Job | One unit of work in a queue (for example: process one ingest row). |
| Cron | Work on a schedule: the 2-minute sweeper and the nightly stats. |
| Job run (log) | A history entry saying a job finished or failed, with how long it took. |
Running locally
pnpm dev starts everything. Fixed ports: dashboard on 3000, API on 3001, this docs app on 3002.
Health screen
Open the real Overview first whenever something feels wrong. If PENDING or FAILED is growing, go to Ingests. If failed jobs are growing, go to Queues. If a card says unavailable (Redis down?), the queue system is unreachable — the inbox cards still work.
Example view — not live dataFast lane
waiting: 2 active: 0 delayed: 0 completed: 120 failed: 1
Cron
waiting: 0 active: 0 delayed: 2 completed: 48 failed: 0
Inbox
PENDING: 2 PROCESSING: 0 DONE: 118 FAILED: 1
Latest stats
{
"customers": 300000,
"pending": 2,
"failed": 1
}