Deviopath › Node.js

Writable Stream Backpressure

Hard · Streams

Backpressure is how Node.js streams stop a fast producer from filling memory when the consumer (a disk, a socket) is slower. writable.write(chunk) always accepts the chunk, but returns false once the internal buffer reaches highWaterMark. A well-behaved producer then stops writing and waits for the "drain" event.

Simulate it. Complete SimWritable(highWaterMark, drainPerTick) (the event methods on, once and emit are already written) and produce(stream, chunks):

- write(chunk) (a string): add chunk.length to buffered, then return buffered < highWaterMark. When it returns false, remember that a drain is needed.

- tick(): the consumer removes up to drainPerTick characters from the buffer (buffered never goes below 0). If the buffer is now empty and a drain is needed, clear that flag and emit "drain", unless end() was already called (real Node does not emit "drain" while ending). After that, if end() was called and the buffer is empty, emit "finish".

- end(): marks the stream as ending. If the buffer is already empty, emit "finish" right away. "finish" is emitted at most once.

- produce(stream, chunks): write the chunks in order. When write returns false and chunks are left, stop and continue only after the next "drain" (use once). After writing the last chunk, call end() immediately, even if that write returned false.

The driver runBackpressure(highWaterMark, drainPerTick, chunks) is already written. It logs every write (with its return value and buffered), end, every tick, drain and finish, and ticks until "finish" (at most 30 ticks).

Next: Multipart Form Parser