Python Decorators and Generators: Advanced Patterns

Python is beloved for its readability, but its true power lies in how it allows developers to reshape the language itself. Two of the most potent features for writing elegant, idiomatic Python are Decorators and Generators.

Most developers know the basics: decorators add @timing logs to functions, and generators use yield to save memory. But when you push these features into advanced territory, they shift from convenient syntax to architectural pillars.

In this article, we will explore advanced patterns for both, and see what happens when we combine them to write truly Pythonic code.

Part 1: Advanced Decorator Patterns

A decorator is just a function that takes a function and returns a new function. But as requirements grow, the basic pattern breaks down.

Pattern 1: Parameterized Decorators (The "Three-Nest" Problem)

What if your @retry decorator needs to know how many times to retry? Or your @timeout decorator needs the number of seconds? You need a way to pass arguments to the decorator itself.

This requires three levels of nesting: the parameter factory, the decorator, and the wrapper.

import functools
import time

# Level 1: Takes the decorator parameters
def retry(max_retries=3, delay=1):
    # Level 2: Takes the target function
    def decorator(func):
        # Level 3: Takes the function's arguments
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(max_retries):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    print(f"Attempt {attempt + 1} failed: {e}")
                    time.sleep(delay)
            raise RuntimeError(f"Function {func.__name__} failed after {max_retries} retries")
        return wrapper
    return decorator

# Usage:
@retry(max_retries=5, delay=2)
def call_flaky_api():
    # ...

Visualizing the call stack:

graph TD
    A["@retry(max_retries=5)"] -->|Returns| B[decorator function]
    B -->|Wraps| C[call_flaky_api]
    C -->|Calls| D[wrapper function]
    D -->|Executes logic & calls| E[Original call_flaky_api logic]

Pattern 2: Class-Based Decorators (Stateful Wrappers)

When your decorator needs to maintain state (like counting calls, rate limiting, or caching), nested functions become ugly. Use a class that implements the __call__ method.

import functools
import time

class RateLimiter:
    def __init__(self, max_calls, period):
        self.max_calls = max_calls
        self.period = period
        self.calls = []

    def __call__(self, func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            now = time.time()
            # Remove calls outside the current time period
            self.calls = [c for c in self.calls if c > now - self.period]

            if len(self.calls) >= self.max_calls:
                raise Exception("Rate limit exceeded!")

            self.calls.append(now)
            return func(*args, **kwargs)
        return wrapper

@RateLimiter(max_calls=5, period=60)
def send_notification(user):
    # Guaranteed not to spam more than 5 times a minute
    pass

Part 2: Advanced Generator Patterns

Generators don't just save memory; they fundamentally change how you structure data flows by enabling lazy evaluation.

Pattern 1: Composable Data Pipelines (ETL)

The most powerful use of generators is chaining them together to process streaming data. Because each step yields one item at a time, you can process gigabytes of data using nearly zero RAM.

graph LR
    A[Raw Log File] -->|read_lines| B(Parse Generator)
    B -->|yield dict| C(Filter Generator)
    C -->|yield valid| D(Transform Generator)
    D -->|yield JSON| E[Output]

import json

# Step 1: Read lazily
def read_logs(filepath):
    with open(filepath, 'r') as f:
        for line in f:
            yield line.strip()

# Step 2: Parse lazily
def parse_dicts(lines):
    for line in lines:
        try:
            yield json.loads(line)
        except json.JSONDecodeError:
            continue

# Step 3: Filter lazily
def filter_errors(records):
    for record in records:
        if record.get('level') == 'ERROR':
            yield record

# The Pipeline: Memory stays flat regardless of file size!
lines = read_logs('server.log')
records = parse_dicts(lines)
errors = filter_errors(records)

for error in errors:
    send_alert(error)

Pattern 2: yield from (Delegating to Sub-Generators)

Often, you have a generator that needs to yield all the items from another generator. Instead of writing a for loop, Python 3.3+ introduced yield from.

This is incredibly useful for flattening nested structures or tree traversal.

# Flattening a matrix without creating intermediate lists
def flatten(matrix):
    for row in matrix:
        # "Yield every item from this sub-iterator"
        yield from row 

matrix = [[1, 2], [3, 4], [5, 6]]
print(list(flatten(matrix))) # [1, 2, 3, 4, 5, 6]

Pattern 3: Async Generators (Streaming I/O)

Since Python 3.6, you can use yield inside async functions. This is the ultimate pattern for streaming data from network requests or databases without blocking the event loop.

import asyncio
import aiohttp

async def fetch_paginated_api(url):
    page = 1
    async with aiohttp.ClientSession() as session:
        while True:
            async with session.get(f"{url}?page={page}") as response:
                data = await response.json()

                if not data['items']:
                    break # No more data

                # Yield items one by one to the async consumer
                for item in data['items']:
                    yield item

                page += 1

# Consumer
async def main():
    async for item in fetch_paginated_api('https://api.example.com/users'):
        print(item)

The Intersection: @contextmanager

The most elegant pattern in Python is where Decorators and Generators collide: the contextlib.contextmanager decorator.

Writing a class with __enter__ and __exit__ methods for context managers (the with statement) is verbose. Instead, you can write a generator that yields exactly once, and decorate it. The code before the yield is the __enter__, and the code after is the __exit__.

from contextlib import contextmanager
import time

@contextmanager
def timer(label):
    """A context manager to time a block of code."""
    # __enter__ logic
    start = time.time()
    print(f"[{label}] Starting...")

    try:
        # Suspend execution and hand control back to the 'with' block
        yield
    finally:
        # __exit__ logic (guaranteed to run, even on exceptions)
        elapsed = time.time() - start
        print(f"[{label}] Finished in {elapsed:.4f}s")

# Usage:
with timer("Data Processing"):
    # Do expensive work here
    time.sleep(1.5)

Visualizing the Control Flow:

sequenceDiagram
    participant Code as with block
    participant CM as @contextmanager
    Code->>CM: Calls timer()
    CM->>Code: Runs setup (start timer)
    CM-->>Code: Yields control
    Note over Code: Executes inside the 'with' block
    Code->>CM: Block finishes (or raises Exception)
    CM->>CM: Runs finally block (calculate time)

Conclusion

Decorators and Generators are Python's way of giving you metaprogramming superpowers without leaving the readable syntax Python is known for.

• Use Parameterized/Class Decorators to encapsulate cross-cutting concerns (retry logic, rate limiting, auth checks) into clean, reusable @ syntax.
• Use Generator Pipelines to process massive datasets with O(1) memory complexity.
• Use @contextmanager to replace verbose classes with elegant, readable with blocks.

Stop thinking of decorators as just "wrappers" and generators as just "lists that save memory." They are structural tools that separate the what from the how, allowing you to write code that scales in both performance and maintainability.