Streaming CSV Parser
Hard · Streams
Large CSV files are parsed in Node.js as a stream (like csv-parse), because they do not fit in memory. The difficulty: a chunk can end anywhere, in the middle of a field, a quoted value, an escaped "" pair or a \r\n line break.
Implement CsvParser({ header }) with write(chunk) and end(). Both return an array of the records completed during that call.
- Fields are separated by , and records end at \n. Outside quotes, \r characters are ignored, so both \n and \r\n line endings work.
- A field that starts with " is quoted: it ends at the next " that is not followed by another ". Inside it, "" means one ", and ,, \n and \r are ordinary characters. A " in the middle of an unquoted field is an ordinary character.
- A line with no characters (ignoring \r) is skipped. a,,b has an empty middle field, and a line containing just "" is one empty field.
- A record is returned as soon as its \n is seen. end() returns the last record if the input did not end with a line break. If the input ends inside a quoted field, end() throws Error("Unterminated quoted field").
- Without header, records are arrays of strings. With header: true, the first record gives the column names and is not returned; the others become objects. Missing fields become "", and extra fields are ignored.
The driver parseCsvStream(chunks, options) is already written. It returns { rows, perWrite }, where perWrite counts the records returned by each write and by end, or { error, rows } if something throws.