PCIe Overview and Architecture

Learning objectives

• Explain the core mental model behind PCIe Overview and Architecture
• Apply PCIe Overview and Architecture within What PCIe Is and Why It Exists
• Identify important boundaries, trade-offs, and failure modes
• Produce concrete evidence from the practice exercise

Related: Physical Layer | Transaction Layer | Data Link Layer | PCIe Generations | PCIE Index

Why PCIe Exists

PCI (Peripheral Component Interconnect, 1992) was a parallel shared bus: all devices sat on the same set of address, data, and control wires. Every device listened to every transaction. Only one device could use the bus at a time. As CPU and memory speeds climbed through the 1990s, the shared bus became the bottleneck, not because the devices were slow, but because they had to take turns.

PCI-X (1998) stretched the parallel bus to 64 bits at 133 MHz, buying time, but the fundamental problem remained: adding more devices made every device slower, because the bus arbitration overhead and signal integrity limits of a long parallel bus could not keep up.

PCIe (PCI Express, ratified 2002) replaced the shared parallel bus with a switched point-to-point serial fabric. Instead of one bus with many devices, PCIe is a network of dedicated links. Every device gets its own private, full-duplex connection to a switch or Root Complex.

No sharing. No bus-wide arbitration. Adding a device adds connectivity and switching structure rather than forcing all devices onto one shared highway.

The same software-facing configuration model from PCI was largely preserved, which is why operating systems and drivers could evolve from PCI to PCIe without throwing away the entire ecosystem.

A Useful Mental Model

AXI4 often feels like an on-chip transport protocol. PCIe feels more like a packet-switched network inside a machine.

That means three mental moves help:

1. think in packets, not read/write wires
2. think in topology, not shared bus ownership
3. think in layered responsibilities, not one monolithic protocol block

If you carry the old PCI bus mental model too far into PCIe, many rules seem arbitrary. Once you think "small network fabric," the protocol gets much easier to reason about.

PCIe Topology

The Three Roles

Every element in a PCIe fabric plays one of three roles:

Root Complex (RC) - The host-side origin of the PCIe fabric. The Root Complex connects the CPU and memory subsystem to the PCIe fabric. It contains the PCIe port controller and root ports for downstream links.

Switch - A packet-forwarding device that fans out one upstream port into multiple downstream ports. A switch does not consume TLPs for application logic. It reads routing information and forwards packets to the right port.

Endpoint (EP) - The ultimate source or destination of transactions. GPUs, NICs, NVMe SSDs, and PCIe-attached ASICs are endpoints.

Why This Topology Scales Better

In PCI, every device was forced onto one shared electrical conversation.

In PCIe:

• each link is local
• each port negotiates its own width and speed
• switching happens with packet forwarding rules

Analogy:

PCI was like one long office meeting where only one person could speak at a time. PCIe is like a building full of point-to-point phone calls routed through a switchboard.

A Typical Topology

         +------------------------------+
         |       CPU / Root Complex     |
         |  (memory controller + PCIe)  |
         +------+----------+------------+
                |          |
             x16 link    x4 link
                |          |
         +------v------+  +--v------+
         |  GPU (EP)   |  | Switch  |
         |  discrete   |  | x4 up   |
         +-------------+  | x1 down |
                          +--+--+---+
                             |  |
                           NVMe NIC
                            EP   EP

Each link is independent. The GPU's x16 link runs independently of what the NVMe SSD is doing. The switch arbitrates traffic internally instead of making all devices contend on one global bus.

The Three-Layer Architecture

PCIe is intentionally layered:

+----------------------------------------+
|        Transaction Layer (TL)          |  <- semantics, routing, ordering
+----------------------------------------+
|        Data Link Layer (DLL)           |  <- reliability, ACK/NAK, LCRC
+----------------------------------------+
|         Physical Layer (PHY)           |  <- signaling, lanes, LTSSM
+----------------------------------------+

Transaction Layer

Creates and consumes TLPs. This is where read, write, configuration, completion, and message semantics live.

See Transaction Layer and TLP Structure.

Data Link Layer

Makes one hop reliable using:

• sequence numbers
• LCRC
• ACK/NAK
• replay from a retry buffer

See Data Link Layer and Flow Control.

Physical Layer

Moves bits across the link using:

• differential signaling
• lane training
• SerDes
• link-state machinery
• equalization

See Physical Layer and PCIe Generations.

Why Layering Matters

Layering keeps responsibilities separate.

For example:

• if a bit is corrupted on one hop, the Data Link Layer handles it
• if a TLP is routed to the wrong destination, that is a Transaction Layer issue
• if the link never reaches L0, that is a Physical Layer problem

This separation is one of the most useful debugging tools in PCIe.

When a system fails, ask:

1. did the link come up?
2. did packets move reliably across the hop?
3. did the right transaction semantics happen at the endpoint?

How a Transaction Flows End-to-End

Scenario: the host issues a 64-byte memory read from an NVMe SSD.

Step 1 - Request

1. The Root Complex Transaction Layer creates a Non-Posted Memory Read TLP.
2. The Data Link Layer adds sequence tracking and LCRC.
3. The Physical Layer serializes the packet onto the link.
4. A switch, if present, forwards the packet based on routing information.
5. The endpoint receives the TLP and interprets the request.

Step 2 - Completion

1. The endpoint creates a Completion with Data TLP.
2. The TLP is routed back through the fabric.
3. The Root Complex matches the completion to the original request using Requester ID and Tag.

This flow is one of the most important PCIe ideas:

• reads are request/completion pairs
• writes are often posted and do not require a completion
• routing and reliable delivery happen beneath the software-visible transaction

Key Terminology Quick Reference

Common Pitfalls

Pitfall 1: Confusing a switch port with an endpoint

A switch port has config space and looks device-like to software, but it is fundamentally a routing element, not an application endpoint.

Pitfall 2: Thinking PCIe is a shared bus with serial wires

It is not. The topology and the forwarding rules are different enough that the old bus mental model causes mistakes.

Pitfall 3: Mixing up the layers

• link bring-up problems are PHY issues
• replay and ACK/NAK issues are DLL issues
• ordering and completion issues are TL issues

Pitfall 4: Forgetting that reads and writes behave differently

Reads require completions. Many writes are posted and complete differently from the requester's perspective.

Best Next Notes

• Transaction Layer for what the packets mean
• TLP Structure for field-by-field decoding
• Data Link Layer for ACK/NAK and replay
• Physical Layer for LTSSM and signaling
• Configuration Space for how software discovers and controls devices

Summary

PCIe replaced the shared PCI bus with a layered, packet-switched, point-to-point fabric. Once you think of it as an in-system network with dedicated links, switches, and transaction packets, the architecture becomes much easier to understand and the rest of the protocol topics fit naturally into place.

Practice lab

Draw or encode one legal transaction trace for PCIe Overview and Architecture. Annotate fields, channel events, ordering points, and completion conditions; then construct one illegal or adversarial trace and define the checker that should catch it.

Review questions

1. What problem does PCIe Overview and Architecture solve, and what assumptions does it rely on?
2. Which boundary or failure case is easiest to miss, and how would you expose it?
3. What alternative design would you consider, and what trade-off would change the decision?
4. What artifact, trace, test, or metric proves that your implementation is correct?

Completion evidence

• A working artifact, annotated trace, or reproducible experiment
• At least one normal case and one deliberately failing or boundary case
• A concise explanation of the design choice and its trade-offs
• Saved output showing how correctness was evaluated