Deviopath › Node.js

Retry With Exponential Backoff

Hard · Reliability

Network calls from Node.js fail in temporary ways (ECONNRESET, ETIMEDOUT, DNS EAI_AGAIN, HTTP 503...). A good client retries them with exponential backoff and jitter and gives up on errors that will never succeed.

Write retrySchedule(outcomes, options) that simulates the retries on a simulated clock in milliseconds (no real timers). outcomes[k - 1] describes attempt k: either { code, ms? } (a network error) or { status, retryAfter?, ms? } (an HTTP response with an optional Retry-After header). ms (default 0) is how long the attempt takes before its result arrives. If the list runs out, the attempt returns { status: 200, ms: 0 }. options is { baseDelay, maxDelay, maxAttempts, jitter, timeout?, deadline?, startTime? }.

- Attempt 1 starts at time 0. An attempt that starts at at ends at at + ms, and the next attempt starts at end + wait.

- Timeout: if timeout is given and ms > timeout, the client aborts the attempt at at + timeout, and its result is ETIMEDOUT whatever the server would have answered.

- A status from 200 to 299 means success. Retryable failures: codes ECONNRESET, ETIMEDOUT, EAI_AGAIN, and statuses 429 and 503. Anything else is fatal: stop immediately.

- If a retryable failure happens on attempt maxAttempts, stop: the retries are exhausted.

- The wait before attempt k + 1: if the response is 429/503 with retryAfter, follow it. Retry-After is either a number of seconds (wait retryAfter * 1000 ms) or an HTTP date like "Wed, 21 Oct 2015 07:28:00 GMT" (wait until that moment: Date.parse(retryAfter) - (startTime + end), but never less than 0; startTime is the real time in ms since 1970 at simulated time 0). Otherwise use "full jitter": Math.floor(jitter[k - 1] * Math.min(maxDelay, baseDelay * 2 ** (k - 1))). (jitter stands in for Math.random() so results are deterministic.)

- If deadline is given and the next attempt would start after it (end + wait > deadline), stop instead of waiting.

Return { outcome, attempts, waits }:

- outcome: "success", "fatal", "exhausted" or "deadline".

- attempts: { attempt, at, result } for every attempt made, where result is the error code, or "HTTP <status>".

- waits: the waits that were actually scheduled, in order.

Next: HTTP Range Requests