Deviopath › Node.js

Circuit Breaker

Hard · Reliability

A circuit breaker stops a Node.js service from hammering a dependency that is down (the opossum package is a popular implementation). It has three states:

- CLOSED: calls go through. Count consecutive failures; a success resets the count. When the count reaches failureThreshold, switch to OPEN.

- OPEN: every call is rejected immediately ("fail fast", error CircuitOpen). Once now - openedAt >= resetTimeout, switch to HALF_OPEN. This check happens lazily at the start of every method call, using that call's now.

- HALF_OPEN: allow trial calls, but at most halfOpenMax may be in progress at once (a call is in progress from request until its result is recorded). Extra calls are rejected. Any failure switches back to OPEN (with openedAt = now). After successThreshold successes, switch to CLOSED.

Every state change resets all counters (failures, successes, calls in progress) and calls onStateChange(from, to, now).

Implement CircuitBreaker(options, onStateChange) where options is { failureThreshold, resetTimeout, halfOpenMax, successThreshold }:

- request(now) returns null when the call is rejected, otherwise a ticket (any non-null value you choose).

- record(now, ticket, ok) reports the result of an allowed call. If the circuit changed state after that ticket was handed out, the result is stale and must be ignored completely.

- getState(now) returns the current state.

The driver runBreaker(options, ops) is already written. Ops are ["call", now, id], ["ok", now, id], ["fail", now, id] (results are only reported for allowed calls) and ["state", now]. It returns a log.

Next: Access Log Statistics