Deviopath › Python

Lazy Generator Pipeline

Hard · Generators

Generators produce values one at a time, only when asked. Chaining them builds a pipeline that can handle a huge file while holding one line in memory, and that stops reading as soon as it has enough. itertools.islice(iterable, n) takes at most n items without pulling more.

Each line has the form "name,score". Write three generators and chain them:

- parse(lines) yields (name, score) tuples. Skip blank lines and lines that start with # (after stripping spaces). Strip spaces around the name; the score is an int.

- passing(records, min_score) yields the records with score >= min_score

- first_passing(lines, min_score, n) returns an iterator of the names of the first n passing records

Everything must stay lazy: do not build lists of all lines or records.

The driver run_pipeline(lines, min_score, n) is already written. It feeds the lines through a counting generator and returns the names plus pulled, how many lines your pipeline actually read. A lazy pipeline stops right after the line with the nth match.

Next: Memoize with Stats