An Under-The-Hood View of PCIe 3.0 Link Training (Part II)

Figure 1: A diagrammatic view of the
PCIe 3.0 dynamic link training process

Our last post in this series began examining the recovery.equalization process of PCIe 3.0 dynamic link training, beginning with Phases 0 and 1 of the process (Figure 1). Next, we will move on to take a closer look at Phases 2 and 3, where we'll see what can happen with devices in which the algorithms are not up to par. Namely, issues such as packet errors, dropped packets, and link retraining at lower data rates than 8 Gb/s.

In Phase 2, we have the add-in card requesting different TxEQ settings from the system and also tuning its own RxEQ. Here, things are happening both at the protocol and electrical levels, and they are closely related.

Figure 2: Viewing the PCIe 3.0 protocol traffic
between system board and add-in card

To begin with the protocol-level happenings, we use PCIe protocol-analysis software running on the oscilloscope to view the signal traveling between the system board and the add-in card. In Figure 2, the first thing to look at in protocol-analyzer traces is that letter "R," which represents the root complex in the PCIe channel, and the arrows that tell you which way the packet is moving relative to the root complex. In the example in Figure 2, we see a message coming from the end point (the add-in card) to the root complex (the Tx/Rx on the motherboard).

Figure 3: The Equalization Control portion of the
protocol-analysis view gives us more information

The next piece of the protocol-analyzer trace that's of interest to us is Equalization Control, which orients us inside the recovery.equalization state diagram of Figure 1. When you view these traces on the oscilloscope, hovering with the cursor over the item of interest will bring up more data. If we hover over the Eq Control box shown in Figure 3, it brings up the yellow dialog at the top of the image.

Figure 4: The add-in card begins negotiation with the
system board for a TxEQ setting

The software tells us that we are in Phase 2 of the Equalization Control process. It also reports a value of 2070. The third number, 7, is the Tx equalization preset value. The last value is "Use Preset," and the dialog says "no," which means we're not using this preset but instead we will use cursors. So, in Phase 2 as noted, the add-in card will request different TxEQ settings from the system. The system responds, and then the add-in card will tune its RxEQ.

Next, the add-in card requests not a preset, but rather a cursor, with specific cursor coefficients (Figure 4):

• Pre-Cursor = 6
• Cursor = 16
• Post-Cursor = 2

Figure 5: The root complex responds to the end point
confirming the change in its TxEQ settings

It's telling the root complex to physically set its FIR filter in the TxEQ with these specific characteristics. These correspond to the TxEQ block in the root complex, which we can determine from the packet in play using the protocol-analysis software. We have the upstream port (the add-in card) requesting specific TxEQ settings from the downstream port (the root complex on the system board) by specifying a desired preset or, in this case, cursor values.

In the next packet, a message comes from the system to the add-in card to confirm that it has changed its TxEQ settings to the requested cursor settings (Figure 5). It also has to have changed electrically. These cursor settings of 6-16-2 correspond to the physical filter in the system.

Figure 6: Applying the TxEQ settings requested by the add-in
card prepares the signal for the channel's characteristics

So let's have a look from the electrical point of view (Figure 6). The output from the root complex exhibits a clean eye diagram. We want to apply some filtering to it, and we want to apply the specific filtering requested by the add-in card. So the root complex sets these cursor coefficients in its filter and the output of the TxEQ block looks rather distorted. But it's distorted in that way deliberately in anticipation of the channel's characteristics. 

The interplay between the electrical and physical layers are what is important here. We need these things to happen concurrently, so that the transmitter implements the physical changes to its filtering and it is creating the proper signal, and we need the protocol-layer confirmation sent to the add-in card alerting it of the change. 

Figure 7: After TxEQ filtering, the effects of the channel,
and RxEQ filtering, the receiver sees a good-looking eye

Now that the root complex has responded to the add-in card's request with a confirmation of the filtering changes, the add-in card looks at the signal it's sent after it's passed through the channel and hit the receiver. At this point, it's a decent-looking eye diagram (Figure 7). Now the add-in card tunes its RxEQ settings to open the eye even more, and you can see the effects of that RxEQ in the last eye diagram at right.

It's all about timing these interactions correctly. As soon as the add-in card sees the confirmation from the system board saying it has changed its filter settings, it will begin to tune its RxEQ settings. If this process is not synced up properly, the tuning on the RxEQ side will not match what's happened on the TxEQ side, and you end up with packet errors and a poor bit-error rate.

Figure 8: A second round of TxEQ and RxEQ adjustments
yields an even cleaner eye diagram at the receiver inputs

Once the add-in card has tuned its RxEQ and is satisfied with the results, it will send yet another request to the root complex for new settings. This time it asks for a pre-cursor of 2, a cursor of 17, and a post-cursor of 5. The system responds to confirm this second change, and at the same time it physically changes its TxEQ settings to apply the cursor values. Again, after receiving the confirmation, the add-in card tunes its RxEQ settings in response to the adjusted signal to open the eye as much as possible.

This process repeats some number of times, the number being specific to the implementation. Some devices request only a single preset while others request a variety of presets and cursor values. Still others request every possible cursor combinations in search of perfection. In any case, once the iterations are complete, the equalization process transitions to Phase 3.

Figure 9: In shifting to Phase 3, the add-in card sends its final
acknowledgement to the root complex to confirm TxEQ is
now set to its liking

A new packet from the add-in card signals that it has moved to Phase 3 (Figure 9). In that packet, the add-in card reflects the cursor settings that it wants the system board's transmitter to use from this point forward. Meanwhile, the add-in card has also settled on its RxEQ setting, but it doesn't need to report that information back to the system board because it's set on the add-in card itself.

Phase 3 is essentially the same process as Phase 2, but in reverse. Now we have the root complex on the system board requesting different TxEQ settings from the end point on the add-in card, and the root complex tunes its own RxEQ. After however many iterations of the process are called for in the implementation, the recovery.equalization process concludes.

So what has been accomplished in this process? In Phase 2, we arrived at an optimal TxEQ setting for the root complex and an optimal RxEQ setting for the end point. In Phase 3, the root complex set its RxEQ while the end point set its TxEQ.

Bear in mind that this is a process gone through for this specific channel. If you were to plug that add-in card to a different system board, it would in all likelihood come up with different settings. Even in the same system, if you were to reseat the card, or the ambient temperature changes, it will again come up with different settings.

The end result is a bit-error rate of less than 10^-12, which ensures a robust link without a lot of packet errors, re-transmissions of data, or retraining at a lower data rate. This is a requirement of PCIe 3.0 compliance, and this is what dynamic link equalization is all about.

Previous posts in this series:

The Hows and Whys of PCIe 3.0 Dynamic Link Equalization
PCIe 3.0 Dynamic Link EQ: De-Emphasis, Preshoot, Cursors, and Presets
An Under-The-Hood View of PCIe 3.0 Link Training (Part I)