PCIe Addressing and Routing

Learning objectives

• Explain the core mental model behind PCIe Addressing and Routing
• Apply PCIe Addressing and Routing within Addressing, Routing, and Switches
• Identify important boundaries, trade-offs, and failure modes
• Produce concrete evidence from the practice exercise

Related: TLP Structure | Transaction Layer | Configuration Space | Enumeration | PCIE Index

The Three Routing Mechanisms

PCIe defines exactly three ways a TLP can be routed to its destination. The routing mechanism used by a TLP is determined by its Fmt and Type fields in the TLP header.

1. Address Routing

Used by: Memory Read, Memory Write, I/O Read, I/O Write

The TLP header contains a physical address (32-bit or 64-bit). Each switch port compares the address in the TLP against its programmed memory window and I/O window to decide which output port the TLP should be forwarded to.

A switch's memory window is configured during enumeration (see Enumeration): the BIOS sets the Memory Base and Memory Limit registers in each bridge's Type 1 config header. When a TLP arrives at a switch:

1. The switch checks if the address falls within the upstream port's memory window → forwards upstream
2. If not, it checks each downstream port's memory window in turn → forwards to the first match
3. If no window matches → the switch is the final destination (impossible for a normal switch) or the TLP is dropped with UR

32-bit vs 64-bit addressing:

• 3-DW headers carry a 32-bit address (4-byte aligned, bits [1:0] are implicitly 00)
• 4-DW headers carry a 64-bit address, required for BARs mapped above 4 GB

A 64-bit address where the upper 32 bits are all zero is legal but wastes a DW — senders targeting addresses < 4 GB should use 3-DW headers for efficiency.

2. ID Routing

Used by: Configuration Read/Write, Completion, some Messages

The TLP header contains a Completer ID (Bus:Device:Function, 16 bits) identifying the target device. Each switch port compares the target Bus number against its programmed bus number range (primary/secondary/subordinate — see Enumeration).

A switch's routing decision for ID-routed TLPs:

1. If target Bus == the downstream port's secondary bus → forward downstream to that port (the target is directly connected)
2. If target Bus is between secondary and subordinate of a downstream port → forward downstream (the target is below that port's switch/bridge)
3. If target Bus < the upstream port's secondary bus → forward upstream (the target is above this switch)
4. If no match → UR (bus number not in any range)

This is why the Subordinate Bus Number in a bridge's Type 1 header must be set to the highest bus number in the subtree below that bridge during enumeration. If it's set too low, TLPs for buses just below the limit won't route correctly.

Configuration Type 0 vs Type 1:

• Type 0 means "this device is directly connected to the receiving bridge's downstream bus" — the bridge terminates it locally.
• Type 1 means "this device is further downstream" — each bridge reduces the hierarchy by one level until the target bus is reached, at which point it becomes Type 0.

3. Implicit Routing

Used by: Messages

Messages don't carry a destination address or ID for routing purposes. Instead, the routing direction is encoded in the Type[2:0] sub-field of the TLP Type field:

• Routed to Root Complex (000): Switches forward these upstream unconditionally. PME, ERR_*, and interrupt assertion messages use this.
• Broadcast from Root Complex (011): Switches forward these downstream on all active ports. Power management turn-off uses this.
• Local (100): The switch terminates this message locally; it is not forwarded.
• Addressed (001): Uses the address field for routing — rare.
• ID routed (010): Uses the Completer ID field for routing.

Bus:Device:Function (BDF) Addressing

Every PCIe function (a sub-unit of a device) has a unique 16-bit identifier:

┌──────────────┬─────────────┬──────────────────┐
│  Bus [15:8]  │ Device [7:3]│  Function [2:0]  │
│  (8 bits)    │  (5 bits)   │   (3 bits)       │
└──────────────┴─────────────┴──────────────────┘

• Bus: 0–255 (256 possible bus numbers)
• Device: 0–31 (32 device numbers per bus)
• Function: 0–7 (8 functions per device)

The combination is written as BB:DD.F in Linux (e.g., 03:00.0 = Bus 3, Device 0, Function 0). The BDF is also called the Routing ID (RID) in the PCIe spec.

Requester ID and Completer ID in TLPs

• Requester ID: The BDF of the device that generated the request. Placed in the TLP header by the requesting device. The completer echoes this back in the completion so the switch knows where to route it.
• Completer ID: The BDF of the device generating the completion (or configuration target). Used for ID routing.

How Switches Forward TLPs

A switch is a multi-port bridge. Internally, it has a crossbar or shared buffer that connects all ports. Each port has an independent link (independent ACK/NAK, independent flow control).

When a TLP arrives on the ingress port:

1. The switch's TL receives and validates the TLP (checks LCRC, dequeues from DLL)
2. The TL looks up the routing table (memory windows or bus number ranges) to determine the egress port
3. The TL checks egress flow control credits — does the egress port have enough credits from the downstream device? If not, the TLP is buffered in the switch
4. When credits are available, the TL forwards the TLP to the egress port's DLL, which stamps a new sequence number and LCRC for the next hop

This per-hop credit check is critical: a switch never forwards a TLP to a port that doesn't have the credits to receive it. This prevents deadlock and ensures backpressure propagates upstream.

Peer-to-Peer Transactions

A peer-to-peer transaction is a Memory Read or Memory Write from one endpoint to another endpoint that shares a common upstream switch.

  Root Complex
       │
   ┌───▼───┐
   │Switch │
   └┬─────┬┘
    │     │
   EP-A  EP-B

EP-A writes to EP-B's BAR (address 0x8000_0000, which is in EP-B's memory window).

The TLP path:

1. EP-A → Switch (ingress port B, address routing)
2. Switch: address 0x8000_0000 falls in EP-B's window → egress port C
3. Switch → EP-B

Critical constraint: The switch must have peer-to-peer routing enabled (a bridge may be allowed to route TLPs between downstream ports without forwarding them to the upstream port). Not all switches support this, and it is controlled by the PCI Bridge Control register's Enable Peer-to-Peer bit.

If peer-to-peer routing is not enabled, the switch cannot route the TLP from downstream to downstream — the TLP would have to go up to the Root Complex and back down, which requires the Root Complex's address space to include EP-B's BAR (it does, since the RC is the root of all address assignment) and the RC to route it back. This is inefficient but works.

ARI — Alternative Routing-ID Interpretation

Standard PCIe limits a device to 8 functions (Function numbers 0–7 within Device:Function fields). This is a problem for SR-IOV: a single physical NIC can present hundreds of Virtual Functions. 8 per device × 32 devices per bus = 256 functions per bus — insufficient.

ARI (Alternative Routing-ID Interpretation) removes the Device/Function boundary, treating the 8-bit Device+Function field as a single flat 8-bit Function number, giving 256 functions per bus number — all under a single device slot.

ARI changes:

• The Completer ID's Device field must be 0 for ARI-capable functions (all function numbers are in the Function field)
• Switches must support ARI forwarding (negotiate via ARI capability registers) to route TLPs with Function numbers > 7 correctly
• The OS/firmware enables ARI in both the endpoint and the upstream bridge before SR-IOV enumeration

See Enumeration for how this interacts with bus enumeration, and Configuration Space for the ARI capability registers.

Common Pitfalls

PITFALL 1: Sending a 64-bit MWr to an address < 4 GB
It works but wastes 4 bytes of link bandwidth per TLP (the extra DW for the
upper 32 address bits, which are all zero). Use 3-DW headers for addresses
that fit in 32 bits. Hardware that only decodes 32-bit addresses may not
respond to 4-DW TLPs correctly.

PITFALL 2: Incorrect Subordinate Bus Number blocks routing
During enumeration, if a bridge's Subordinate Bus Number is left at its default (0)
or set too low, TLPs for buses beyond that limit will not be routed correctly.
The BIOS/OS must set Subordinate Bus Number to cover all buses in the subtree.

PITFALL 3: Assuming completions need no routing setup
Completions are ID-routed using the Requester ID. The switch must have bus
number ranges configured to forward completions back to the requester. If bus
numbers are wrong, completions vanish silently — manifesting as completion timeouts.

PITFALL 4: Peer-to-peer performance expectations
Peer-to-peer through a switch consumes ingress and egress port bandwidth.
A peer-to-peer write from EP-A to EP-B at 8 GB/s consumes 8 GB/s from
EP-A's upstream link AND 8 GB/s from EP-B's upstream link (both connect to the
same switch). Plan switch bandwidth accordingly.

Related

• TLP Structure — Routing-relevant fields in TLP headers (Fmt, Type, Completer ID, address)
• Enumeration — How bus number ranges and memory windows are configured
• Configuration Space — BDF assignment and ARI capability
• Transaction Layer — Overview of TLP types and when each routing mechanism is used
• PCIE Index — Full vault index

Practice lab

Draw or encode one legal transaction trace for PCIe Addressing and Routing. Annotate fields, channel events, ordering points, and completion conditions; then construct one illegal or adversarial trace and define the checker that should catch it. Add an operational constraint such as concurrency, recovery, security, latency, or cost, and defend the resulting design trade-off.

Review questions

1. What problem does PCIe Addressing and Routing solve, and what assumptions does it rely on?
2. Which boundary or failure case is easiest to miss, and how would you expose it?
3. What alternative design would you consider, and what trade-off would change the decision?
4. What artifact, trace, test, or metric proves that your implementation is correct?

Completion evidence

• A working artifact, annotated trace, or reproducible experiment
• At least one normal case and one deliberately failing or boundary case
• A concise explanation of the design choice and its trade-offs
• Saved output showing how correctness was evaluated