Deviopath › Node.js

NDJSON Stream Parser

Hard · Streams

NDJSON (newline-delimited JSON) stores one JSON value per line. It is popular for logs and data exports because it can be processed as a stream, line by line, without loading the whole file.

A readable stream gives you raw bytes in chunks of any size, so a line, and even a single multi-byte UTF-8 character like é or 🚀, can be split across chunks. Implement NdjsonParser(onRecord, onError, maxLineLength):

- write(chunk) receives a Uint8Array. Decode it as UTF-8 without breaking characters that are split between chunks (hint: new TextDecoder() and decode(chunk, { stream: true })). A byte order mark at the very start is ignored (TextDecoder does that for you).

- Every complete line (ending in \n) is handled right away, before write returns. A \r right before the \n is removed.

- Lines are numbered from 1, counting every line, including blank ones.

- Blank lines (empty or only whitespace) are skipped.

- Otherwise parse the line with JSON.parse and call onRecord(value). If parsing fails, call onError({ line, error: "Invalid JSON" }) and continue.

- Line length limit: a client that never sends a newline must not be able to fill the server's memory. A line may have at most maxLineLength characters (.length of the decoded text, not counting the \n but counting a \r before it). As soon as a line is known to be longer, even before its \n arrives, call onError({ line, error: "Line too long" }) once, throw away what you buffered, and ignore the rest of that line up to its \n. The line still counts in the numbering.

- end() flushes the decoder and handles the last line if the input did not end with \n.

The driver parseNdjson(chunks, maxLineLength) is already written. Chunks are passed as arrays of byte values; it returns { records, errors, afterWrite }, where afterWrite is the total number of callbacks (onRecord plus onError) received after each write call.

Next: Chunked Transfer Decoder