Deviopath › Node.js

Background Job Queue With Retries

Hard · Async Patterns

Background job queues for Node.js (BullMQ, Agenda...) run jobs with priorities, limited concurrency and automatic retries. Simulate one with a simulated clock.

Write runQueue(options, jobs). options is { concurrency, maxAttempts, retryDelay }. Each job is { id, priority, duration, fails, at }: it is added at time at, every attempt takes duration (>= 1), and its first fails attempts fail (later attempts succeed).

Process time moments in increasing order. At each moment t:

1. Handle attempts that finish at t, in worker order (workers are numbered from 0). Success: log "<t> done <id>". Failure: if the job has made fewer than maxAttempts attempts, log "<t> fail <id> (retry at <t + retryDelay>)" and the job becomes ready again at t + retryDelay. Otherwise log "<t> dead <id>" and move it to the dead-letter list.

2. Every idle worker (lowest index first) takes the best ready job (ready time <= t): highest priority first, then the earliest ready time, then the earliest position in jobs. Log "<t> start <id>#<attempt>" (attempts count from 1).

A running job is never interrupted, even if a higher priority job arrives. Return { log, completed, deadLetter }, where completed and deadLetter list job ids in the order they finished.

Next: DataLoader Batching