Generator and Iterator

An iterator supplies successive values through iteration and eventually signals exhaustion. A generator function containing yield creates a generator iterator when called. Its body runs as the consumer asks for values, suspending at each yield and retaining the state needed to continue.

This changes when work happens. A function that returns a list completes that list before returning. A generator can open a file and parse a record only when iteration starts, so exceptions may arise in the consuming loop. After an iterator is exhausted, iterating over it again does not replay earlier values; create a new generator when another pass is needed.

Incremental production helps memory only when the whole processing path avoids retaining all records. Summing a small field can keep a running total. Converting the generator to a list, storing every distinct key, or reading one enormous record can still require substantial memory. A generator is not automatically asynchronous or parallel, and laziness alone does not make a computation faster.

A suspended generator can also retain an open file. Breaking the consumer’s loop does not itself close a still-live generator. If early termination is expected, arrange deterministic cleanup, such as contextlib.closing around a generator that owns the file. Exhaustion or explicit closure then allows its internal cleanup to run; forced process termination has different guarantees.

Reference: Python generator iterator methods. For worked examples, see Python for Data Engineers.