Python Generators and Iterators – Memory-Efficient Data Processing

When processing large datasets — log files with billions of rows, streaming sensor data, large model training corpora — loading everything into memory is impossible. Python generators and iterators let you process data one item at a time, using only the memory needed for a single element. This guide covers generators from basics to advanced data science patterns.

The Problem with Lists

# This loads ALL data into memory — crashes on large files
data = [process(line) for line in open("huge_file.csv")]

# Check memory usage
import sys
my_list = list(range(10_000_000))
print(f"List: {sys.getsizeof(my_list) / 1e6:.1f} MB")  # ~80 MB

my_gen = range(10_000_000)  # generator
print(f"Generator: {sys.getsizeof(my_gen)} bytes")      # 48 bytes

Generator Functions with yield

def read_csv_chunks(filepath, chunk_size=1000):
    '''Yields one chunk of rows at a time.'''
    import pandas as pd
    chunk_iter = pd.read_csv(filepath, chunksize=chunk_size)
    for chunk in chunk_iter:
        yield chunk

# Process a 10 GB file with constant memory
total_rows = 0
for chunk in read_csv_chunks("huge_dataset.csv", chunk_size=5000):
    processed = chunk.dropna()
    total_rows += len(processed)

print(f"Processed {total_rows:,} rows")

Generator Expressions

# List comprehension (eager — loads all)
squares_list = [x**2 for x in range(1_000_000)]

# Generator expression (lazy — computes on demand)
squares_gen  = (x**2 for x in range(1_000_000))

# Can be used anywhere an iterator is expected
total = sum(x**2 for x in range(1_000_000))  # no intermediate list!
max_val = max(abs(x) for x in data)

yield from – Delegating to Sub-Generators

def chain_files(*filepaths):
    '''Iterate over multiple files as if they were one.'''
    for path in filepaths:
        with open(path) as f:
            yield from f  # delegates to file iterator

def flatten(nested):
    '''Flatten arbitrarily nested lists.'''
    for item in nested:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item

print(list(flatten([1, [2, [3, 4]], 5])))  # [1, 2, 3, 4, 5]

itertools – The Power Tools

import itertools

# Infinite counters
counter = itertools.count(start=0, step=1)
first_10 = list(itertools.islice(counter, 10))

# Batching
def batch(iterable, size):
    it = iter(iterable)
    while True:
        chunk = list(itertools.islice(it, size))
        if not chunk: break
        yield chunk

for mini_batch in batch(training_data, size=32):
    model.train_step(mini_batch)

# All combinations and permutations
from itertools import combinations, product
param_grid = list(product([0.01, 0.1, 1.0], [50, 100, 200], ['relu', 'tanh']))
print(f"Grid search combinations: {len(param_grid)}")

# Chaining iterables
all_data = itertools.chain(train_data, val_data, test_data)

Custom Iterators with __iter__ and __next__

class DataBatchIterator:
    '''Iterate over a dataset in shuffled mini-batches.'''
    def __init__(self, X, y, batch_size=32, shuffle=True):
        self.X          = X
        self.y          = y
        self.batch_size = batch_size
        self.shuffle    = shuffle

    def __iter__(self):
        import numpy as np
        indices = np.arange(len(self.X))
        if self.shuffle:
            np.random.shuffle(indices)
        for start in range(0, len(indices), self.batch_size):
            batch_idx = indices[start:start + self.batch_size]
            yield self.X[batch_idx], self.y[batch_idx]

    def __len__(self):
        return (len(self.X) + self.batch_size - 1) // self.batch_size

for X_batch, y_batch in DataBatchIterator(X_train, y_train, batch_size=64):
    train_step(X_batch, y_batch)

Generator Pipelines for ETL

def read_lines(filepath):
    with open(filepath, encoding='utf-8') as f:
        yield from f

def parse_json(lines):
    import json
    for line in lines:
        try:
            yield json.loads(line.strip())
        except json.JSONDecodeError:
            pass

def filter_valid(records):
    for record in records:
        if record.get('event_type') and record.get('user_id'):
            yield record

def enrich(records):
    for record in records:
        record['processed_at'] = datetime.now().isoformat()
        yield record

# Compose the pipeline
pipeline = enrich(filter_valid(parse_json(read_lines("events.jsonl"))))

# Consume it — runs lazily, constant memory
for record in pipeline:
    insert_to_db(record)

When to Use Generators

Use generators when your data doesn’t fit in memory, when you’re streaming data (API responses, Kafka, file tails), when you want to compose data transformations as a pipeline, or when you’re implementing a custom training data loader. Don’t use generators when you need random access to elements, when you need to iterate multiple times over the same data, or when the dataset comfortably fits in memory (lists are simpler and easier to debug).

Conclusion

Generators are Python’s mechanism for lazy evaluation — computing values on demand rather than all upfront. For data science, they enable processing datasets of unlimited size with constant memory. The pattern of composing generator functions into a pipeline is elegant and performant. Master yield, generator expressions, and itertools, and you’ll never again crash your machine trying to process a 50 GB log file.