Measure First and Design the Pipeline

Measure First and Design the Pipeline

The capstone ingests records from multiple sources, validates and transforms them, and writes to a pluggable sink. Architecture starts with workload evidence, capacity limits, and interface contracts rather than an assumption that async is always faster.

Module: Module 8: Production Capstone: Typed Async Ingestion

Core mental model

This lesson is built around four connected ideas. Read them as a decision framework, then prove each one with the walkthrough.

• Profile or time the sequential baseline before choosing concurrency.
• Separate source, parser, transformer, and sink behind small typed protocols.
• Define queue capacity and worker limits from downstream constraints.
• Specify ordering, retry, duplicate, and partial-failure semantics up front.

Walkthrough

from typing import AsyncIterator, Protocol

class Source(Protocol):
    def __aiter__(self) -> AsyncIterator[bytes]:
        ...

class Sink(Protocol):
    async def write(self, record: "Record") -> None:
        ...

class Clock(Protocol):
    def monotonic(self) -> float:
        ...

# source -> bounded raw queue -> parse workers
#        -> bounded record queue -> sink workers

The protocols keep infrastructure replaceable in tests. Two bounded queues isolate stage rates and make backpressure visible. Capacity and concurrency belong in configuration with safe limits.

Hands-on lab

Implement a sequential baseline over fixture data, record throughput and peak memory, then write an architecture decision record with interfaces, capacities, failure policy, and an async hypothesis.

What to watch for

• Optimizing without a baseline cannot prove improvement.
• Unbounded concurrency can overwhelm the sink while appearing fast in a toy test.
• Undefined duplicate and ordering semantics become production data bugs.

Engineering checklist

• Connect the lesson's mental model to the choices made in the walkthrough.
• Test the normal case, an empty or boundary case, and one invalid case.
• Keep external input, side effects, and reusable logic in clearly separated layers.
• Prefer the clearest correct implementation before optimizing or generalizing it.

Completion check

You can implement the lab without copying the example, explain the core concepts in your own words, and show tests or terminal output that demonstrate the expected and failure paths.