xilinx answer 72702 ultrascale and ultrascale+ pcie ......figure 3 legacy interrupt timing diagram...

© Copyright 2020 Xilinx

Xilinx Answer 72702 – Interrupt Debug Guide 1

Xilinx Answer 72702

UltraScale and UltraScale+ PCIe Interrupt debug guide

Important Note: This downloadable PDF of an Answer Record is provided to enhance its usability and readability. It is important to note that Answer Records are Web-based content that are frequently updated as new information becomes available. You are reminded to visit the Xilinx Technical Support Website and review (Xilinx Answer 72702) for the latest version of this Answer Record.

Introduction

This document provides the steps involved to initiate Legacy, MSI and MSI-x interrupts with different IP cores targeting UltraScale and UltraScale+ devices. (Xilinx Answer 58495) provides the theoretical background and the steps to initiate Legacy and MSI interrupts with the 7 Series Integrated block IP core.

Legacy Interrupts

Steps to initiate Legacy interrupts

The following steps are involved in generating Legacy interrupts: 1. Make sure the ‘Interrupt Disable’ bit in the Command Register is not set in the endpoint 2. Enable AXISTEN_IF_ENABLE_RX_MSG_INTFC in the IP GUI 3. Drive the interrupt ports

All of the steps mentioned above are applicable to the Integrated Block for PCIe IP in UltraScale/UltraScale+. With the XDMA and AXI-Bridge subsystem IP cores, step 2 is done by default as these IP cores use the Integrated Block for PCIe IP hierarchically and message interface options are set in this IP.

‘Interrupt Disable’ bit in Command Register

The ‘Interrupt Disable’ bit is bit10 in the Command Register of the configuration space. To make sure Legacy interrupts are not disabled by the PCI express Integrated Block in the FPGA, this bit needs to be set to 1’b0.

Figure 1 Command register in Type 0 configuration space

https://www.xilinx.com/support/answers/72702.html




In a hardware design, this step can be implemented in the driver running on the host machine (or) using a hardware module driving configuration management interface. In simulation, the testbench in the Root Port model that comes with the generation of the IP, can be updated using the TSK_TX_TYPE0_CONFIGURATION_WRITE task. For example:

AXISTEN_IF_ENABLE_RX_MSG_INTFC

Legacy interrupts are generated as a message TLP with the message type set as “Assert_INTA” (or) “Deassert_INTA”. To enable transmission of these messages, the “Config TX message Interface” and “Receive Message Interface” options need to be enabled. This option is enabled in the IP configuration GUI by default. Make sure that this option is not deselected.

Drive the interrupt ports

After the steps mentioned in the previous section are completed, the design is ready to generate the interrupts. Refer to the product guide of the target IP for the timing diagram of the interrupt signaling. It is also shown below for reference. The figure below shows the timing diagram for the “Integrated block for PCI express” IP from (PG156). The same timing diagram is applicable for both UltraScale and UltraScale+ devices.

Figure 2 Legacy Interrupt timing diagram for Integrated Block for PCIe IP (Ref: PG156, April 4, 2018)

The figure below shows the timing diagram for “DMA/Bridge subsystem” in “DMA” mode targeting UltraScale and UltraScale+ devices and “AXI-Bridge” mode when targeting UltraScale+ devices. For an AXI-Bridge subsystem IP targeting UltraScale devices, please refer to Table 1 for equivalent signals.



Figure 3 Legacy Interrupt timing diagram for “DMA/Bridge subsystem” IP (Ref:PG194, July 16,2019)

Things to note: Integrated block for PCI express IP

1. The interrupt input ports are required to be synchronous to user_clk (or) axi_aclk domain based on the target IP. 2. Only one Legacy interrupt can be enabled at any given time. The individual bits of cfg_interrupt_int is mapped as

mentioned in the table below. Port Interrupt

cfg_interrupt_int[0] INTA

cfg_interrupt_int[1] INTB

cfg_interrupt_int[2] INTC

cfg_interrupt_int[3] INTD

3. The interrupt process in the IP gets triggered by sampling the cfg_interrupt_int as high but not on the rising edge.

DMA/Bridge Subsystem IP

1. There is no mapping present between usr_irq_req and INTA/INTB/INTC/INTD. With any number of interrupts, all of the interrupt requests will be the same as the interrupt option chosen in the IP GUI.

2. The interrupt is triggered on the rising edge of usr_irq_req input. “AXI-Bridge Subsystem” IP

1. With the “AXI-Bridge subsystem” IP, names of the ports are different. The table below shows the mapping of this signal to the signal names shown in the timing diagram above.

Signal with AXI-Bridge subsystem IP Equivalent signal with “DMA/Bridge subsystem” IP

intx_msi_request usr_irq_req

intx_msi_grant usr_irq_ack

Table 1 Equivalent signals between AXI-Bridge subsystem IP and DMA/Bridge sub system IP

2. There is no mapping present between Intx_msi_request and INTA/INTB/INTC/INTD. With any number of interrupts, all of the interrupt’s requests will be same as the interrupt option chosen in the IP GUI.

3. The interrupt is triggered on the rising edge of Intx_msi_request input.

Verify interrupt is sent

With the Legacy Interrupt process, a message TLP is transmitted towards the host on assertion and de-assertion of the interrupt. The figure below shows a protocol analyzer capture of these two messages transmitted by the end point.



Figure 4 Protocol Analyzer capture showing message TLP’s when Legacy interrupt is initiated

The same message TLP’s can be observed at the PIPE interface between the PCI express integrated block and the GT’s. Refer to the “Descrambling data at PIPE interface” section of this document for steps to descramble data at the PIPE interface. Note: The port names in different IP cores will be different. The Interrupt assertion can be either edge triggered (or) level triggered. Please refer to the product guide of the IP being used.

MSI interrupts

Steps to initiate MSI interrupts

The following steps are involved in generating MSI interrupts:

1. Write to the msi_enable bit in the message control register in configuration space 2. Unset the mask bits in the configuration space and in the IRQ Block User Interrupt Enable Mask (0x04)

register 3. Set the Bus Master enable bit in the command register of the end point configuration space 4. The host must program the Message Data field as the interrupt payload for this vector 5. The host must program the Address register as the interrupt destination address for this vector 6. Map each interrupt input port to a unique interrupt vector

7. Drive the ports of the IP as described in the respective product guides

All of the steps mentioned above are applicable to XDMA, AXI-Bridge subsystem IP cores. Step 2 “Unset the mask bit in IRQ Block User Interrupt Enable Mask (0x04) register” and step 6 “Map each interrupt input port to a unique interrupt vector” are not applicable to the Integrated Block for PCIe IP.

Message Control Register

The figure below shows the configuration space related to MSI interrupt capability.

Figure 5 MSI capability in configuration space (Ref: PG213, June 24, 2019)

The MSI enable bit is bit0 of the Message Control Register shown in the figure below. To enable MSI capability, this bit needs to be set to 1’b1.



Figure 6: Message Control Register in MSI capability space

Note: the Offset address to the Message Control register is different in UltraScale and UltraScale+ devices. With UltraScale, this register is at byte offset 90h. With UltraScale+, this register is at byte offset 48h. When more than one interrupt vector is enabled, the “Multiple Message Enable” bits need to be set. The figure below shows the description of these bits from the PCIe local bus specification v3.0.

Figure 7: Multiple Message Enable bit description (Ref: PCI LOCAL BUS SPECIFICATION, REV. 3.0)

In a hardware design, this step can be implemented in the driver running the host machine (or) using a hardware module driving the configuration management interface. In simulation, the testbench in the Root Port model can be updated using TSK_TX_TYPE0_CONFIGURATION_WRITE task. For example:



What to expect after successfully executing this step? The msi_enable signal will be asserted HIGH What to check if expected observations are not found? Readback the MSI control register contents to confirm that the msi_enable bit is set correctly

o Simulation – TSK_TX_TYPE0_CONFIG_READ o Hardware – “setpci” command Ex: “setpci -s 06:00.0 04.l”

Note: Configuration space offset is different with different IPs. Confirm that the correct offset is used in the design based on the target device. Mask bits in the configuration space and in the IRQ Block User Interrupt Enable Mask (0x04) register.

Unset the Mask bits

In the configuration space, MSI capability space has “Mask bits”. The mask bits must be set to 1’b0 for interrupts to be sent to the root port. The figure below provides the address format to be used when accessing the IRQ registers in the XDMA BAR space, memory map for AXI-Bridge subsystem IP, and “DMA/Bridge subsystem” in AXI-Bridge mode. To access IRQ Block registers, the Target field will be “2” and channel field will be “0”. So, the offset to the IRQ Block User Interrupt Enable Mask register is 0x2004.

Figure 8 PCIe to DMA Address Format (Ref: PG195, June 20, 2019)

Note: the number of Target fields supported with XDMA and AXI-Bridge subsystem IP cores are different but the IRQ registers are mapped to the same address offset. The figure below shows the bit description of the IRQ Block User Interrupt Enable Mask register. To ensure that the MSIx interrupt is not masked, the corresponding mask bit of this register needs to be set to 1’b1.

Figure 9: IRQ Block User Interrupt Enable Mask register (PG195, June 20, 2019)

If the mask bit is not set and usr_irq_req is asserted, the “IRQ Block User Interrupt Pending” register (offset: 0x2048) shows that the interrupt is in a pending status.



In a hardware design, this step can be implemented in the driver running in the host machine. In simulation, the testbench in the Root Port model can be updated using the TSK_TX_TYPE0_CONFIGURATION_WRITE task for writes to the configuration space of the end point and TSK_TX_BAR_WRITE to write to IRQ block registers in PCIe to the DMA BAR as shown in the example below: Example:

With the AXI-Bridge subsystem IP, IRQ registers are accessible only through the AXI_CTRL interface. The M_AXI interface of the IP can be connected back to the AXI_CTRL interface through an interconnect. This enables the driver to access the IRQ registers through a memory write sent to the M_AXI interface. In simulation, TSK_TX_MEMORY_WRITE_32 can be used to access IRQ registers in end point from Root Port model. Example:

Bus Master enable bit in command register

The Bus master enable bit is part of the command register in the configuration space. Figure 1 provides a bit description of command register. In general, memory space enable and I/O space enable bits will be set HIGH during enumeration. If not, please set these bits also. To ensure that the memory write TLP is transmitted to the host when MSI-X interrupt is asserted, the Bus Master Enable bit needs to be set. In a hardware design, this step can be implemented in the driver running the host machine (or) using a hardware module driving configuration management interface. In simulation, the testbench in the Root Port model can be updated using the TSK_TX_TYPE0_CONFIGURATION_WRITE task as shown in the example below: Example:

Message Data field

Figure 5 shows the configuration space related to MSI interrupt capability. When MSI interrupt is initiated by the user, the Endpoint sends a memory write TLP with “Message Data” register contents as data payload of the TLP. RP must write specific data value to this register in MSI capability to recognize the source of the MSI interrupt. This is essential when multiple end points are connected to the same RP. The default value of this register is 16’h0. Note: the message data register needs to be written every time the PCIe link is enumerated by the RP. In a hardware design, this step can be implemented in the driver running the host machine (or) using a hardware module driving the configuration management interface. In simulation, the testbench in the Root Port model can be updated using the TSK_TX_TYPE0_CONFIGURATION_WRITE task.



Example:

Message Address

Along with the Message data, RP must write to Message Address field of the MSI capability space of the end point configuration space. In a production design, this step can be implemented in the driver running in the host machine. The driver can allocate host memory to which MSI interrupts can be destined. The address to this memory location is programmed as “Message address” in the Endpoint MSI capability. Another method is to use a hardware module driving the configuration management interface. In simulation, the testbench in the Root Port model can be updated using the TSK_TX_TYPE0_CONFIGURATION_WRITE task as shown in the example below Example:

Default value of this register is 16’h0. Note: the message address register needs to be written every time the PCIe link is enumerated by the RP.

Unique interrupt vector

All of the IPs targeting UltraScale and UltraScale+ allow the user to enable more than one MSI interrupt. In this case, each of the interrupt input needs to be mapped to a unique interrupt vector. Based on the interrupt vector assigned, the message data field will be updated with the memory write TLP. If the Interrupt vector “5” is used to send the MSI interrupt, the message data field is incremented by “5” and used as the data payload of memory write TLP associated with the MSI interrupt. With the XDMA IP, usr_irq_req input will be of multiple bits when more than one MSI interrupt is enabled. Each of the usr_irq_req bits need to be mapped to a unique vector by writing to the IRQ register space (IRQ Block User Vector Number at offset 0x2080 – 0x208C). In addition, each of the interrupt sources needs to be mapped to different usr_irq_req input bit in the user logic. With the Integrated block for PCIe IP, the cfg_interrupt_int input will be of multiple bits. Only the mapping of different interrupt sources to different bits of this input is required in the user logic. With the AXI-Bridge subsystem IP, intx_msi_request is always a 1-bit signal. The user needs to set the msi_vector_num input based on the interrupt source.

Drive Interrupt Ports

After the steps mentioned in the previous sections are completed, the design is ready to generate interrupts. Refer to the product guide of the target IP for the timing diagram of the interrupt signaling. The figure below shows the timing diagram for initiating MSI interrupts when using the “Integrated block for PCI express” IP.



Figure 10: MSI interrupt timing diagram for “Integrated block for PCI express” IP (PG213 (v1.3) June 24, 2019)

The figure below shows the timing diagram for the “DMA/Bridge subsystem” in “DMA” mode targeting UltraScale and UltraScale+ devices and “AXI-Bridge” mode when targeting UltraScale+ devices. For the AXI-Bridge subsystem IP targeting UltraScale devices, please refer to Table 1 for equivalent signals.

Figure 11 MSI interrupt timing diagram for “DMA/Bridge subsystem” IP (PG195 (v4.1) June 20, 2019)


1. The interrupt process in the IP gets triggered by sampling the cfg_interrupt_int as high but not on the rising edge. 2. Even though cfg_interrupt_int is kept asserted HIGH, only one interrupt gets asserted

DMA/Bridge subsystem IP The interrupt is triggered on the rising edge of usr_irq_req input “AXI-Bridge subsystem” IP.

1. With the “AXI-Bridge subsystem” IP, the names of the ports are different. Table 1 shows the mapping of this signal to the signal names shown in the timing diagram above.

2. The interrupt is triggered on the rising edge of usr_irq_req input

Note: The interrupt input ports are required to be synchronous to the user_clk (or) axi_aclk domain based on the target IP.

MSI-X Interrupts

Steps for Enabling MSI-X Interrupt

The following steps are involved in generating MSI-X interrupts: 1. The MSI-X Enable bit must be set in the MSI-X Control register to enable MSI interrupt operation

2. The Function Mask in MSI-X Control register must be unset



3. Set the Bus Master enable bit in the command register of the Endpoint

4. The Mask bit in the Vector Control register in the MSI-X Table must be unset to enable the vector

5. Unset the mask bit in the IRQ Block User Interrupt Enable Mask (0x04) register

6. Map each interrupt input port to a unique interrupt vector

7. RP must program Message Data field as the interrupt payload for this vector

8. RP must program the Address register as the interrupt destination address for this vector

9. Drive the ports of the IP as described in the respective product guides

Things to note:

• Steps 1, 2 and 3 can be done by writing to the configuration management interface or by the driver.

• Steps 4, 5 and 6 involve writing to the BAR space.

• Steps 7 and 8 are expected to be taken care of during enumeration in hardware by the driver.

• All of the steps mentioned above are applicable to XDMA and AXI-Bridge subsystem IP cores.

• Only step 5 “Unset the mask bit in IRQ Block User Interrupt Enable Mask (0x04) register” is not applicable to Integrated Block for PCIe IP.

MSI-X Enable and Function Mask Bits in MSI-X Control Register

The MSI-X control register is part of the configuration space. The figure below shows a snippet of MSI-X capability.

Figure 12 MSI-X capability in configuration space (Ref: PG156 April 4, 2018)

Note: the Offset address to the Message Control register is different in UltraScale and UltraScale+ devices. With UltraScale, this register is at byte offset B0h. With UltraScale+, this register is at byte offset 60h. Bit 15 is the MSI-X enable bit and Bit 14 is the function mask bit. To enable the MSI-X function, the MSI-X enable bit must be set and the function mask bit must be unset.

Figure 13: Message control register in MSI-X capability space



After setting the bits in the message control register using the configuration management interface (or) Type0 configuration access, msix_enable output signal from XDMA IP will be asserted HIGH. In a production design, this step can be implemented in the driver running the host machine (or) using a hardware module driving the configuration management interface. In simulation, the testbench in the Root Port model can be updated using the TSK_TX_TYPE0_CONFIGURATION_WRITE task as shown in the example below Example:

What to expect after successfully executing this step? The Msix_enable signal will be asserted HIGH What to check if expected observations are not found? Readback the register contents to confirm that the corresponding bits are set correctly Simulation – TSK_TX_TYPE0_CONFIG_READ Hardware – “setpci” command can be used to read the configuration space registers Ex: setpci -s 06:00.0 04.l Note: Configuration space offset is different with different hardblocks. Confirm that the correct offset address is used.

Bus Master Enable bit in command register

The bus master enable bit is part of the command register in the configuration space. Figure 1 provides the bit description of the command register. In general, memory space enable and I/O space enable bits will be set HIGH during enumeration. If not, please also set these bits. To ensure that the memory write TLP is transmitted to the host when the MSI-X interrupt is asserted, the Bus Master Enable bit needs to be set. In a production design, this step can be implemented in the driver running the host machine (or) using a hardware module driving the configuration management interface. In simulation, the testbench in the Root Port model can be updated using the TSK_TX_TYPE0_CONFIGURATION_WRITE task. Example:

Mask Bit in Vector Control Register

The MSI-X table looks like the figure below. Bit 0 of the vector control register is the mask bit.



Figure 14: MSI-X Table Structure (Ref: PCI LOCAL BUS SPECIFICATION, REV. 3.0)

The BIR/PBA BIR table shown in Figure 12 MSI-X capability in configuration space (Ref: PG156 April 4, 2018) points to the BAR space in which the MSI-X table and Pending bit array registers will reside. If the Table BIR field in figure above is “2”, it means that the MSIx Table will be mapped to BAR2 in the host memory. The table offset field in the same configuration space register will provide the offset from the base address of BAR2 space. With the XDMA IP, the MSIx table is internal to the core. And the offset to the MSIx table is 0x8000.

Mask bit in IRQ register space

Figure 8 provides the address format to be used when accessing the registers in XDMA BAR space. To access IRQ Block registers, the Target field will be “2”, channel field will be “0”. So, the offset to IRQ Block User Interrupt Enable Mask register is 0x2004.

Figure 9 shows the bit description of the IRQ Block User Interrupt Enable Mask register. To ensure that the MSIx interrupt is not masked, the corresponding mask bit of this register needs to be set to 1’b1. If the mask bit is not set and usr_irq_req is asserted, the “IRQ Block User Interrupt Pending” register (offset: 0x2048) shows the interrupt is in a pending status.

Unique Interrupt Vector

All of the IPs targeting UltraScale and UltraScale+ allow the user to enable more than one MSI-x interrupt. Each of the interrupt input needs to be mapped to a unique interrupt vector. Each MSI-x vector has a message data field. Based on the interrupt vector assigned, the message data field of that vector will be used with the memory write TLP. With the XDMA IP, usr_irq_req input will be of multiple bits when more than one MSI-x interrupt is enabled. Each of the usr_irq_req bits need to be mapped to unique vectors by writing to IRQ register space. In addition, each of the interrupt sources needs to be mapped to different usr_irq_req input bit in the user logic. With the Integrated block for PCIe IP, the cfg_interrupt_int input will be of multiple bits. Only the mapping of different interrupt sources to different bits of this input is required in the user logic. With the AXI-Bridge subsystem IP, intx_msi_request is always a 1-bit signal. The user needs to set the msi_vector_num input based on the interrupt source.

Message Data field

Figure 14 shows the MSI-x table in the BAR space assigned to the MSI-x interrupt capability. When MSI-x interrupt is initiated by the user, the end point sends a memory write TLP with the “Message Data” register contents of the corresponding



vector as a data payload of the TLP. The RP must write a specific data value to this register in the MSI-x vector table to recognize the source of the MSI interrupt. Note: the message data register needs to be written every time PCIe link is enumerated by the RP. In a hardware design, this step can be implemented in the driver running the host machine. In simulation, the testbench in the Root Port model can be updated using TSK_TX_MEMORY_WRITE_32 task. Example:

Message Address

Along with Message data, RP must write to Message Address field of vector table with the end point. In a production design, this step can be implemented in the driver running the host machine. The driver can allocate host memory which MSI-x interrupts can be destined to. The address to this memory location is programmed as “Message address” in the end point MSI-x capability. In simulation, the testbench in the Root Port model can be updated using TSK_TX_MEMORY_WRITE_32 task. Example:

Default value of this register is 16’h0. Note: the message data register needs to be written every time PCIe link is enumerated by the RP.

Drive Interrupt Ports

After the steps mentioned in the previous section are completed, the design is ready to generate the interrupts. Refer to the product guide of the target IP for the timing diagram of the interrupt signaling. The below figure shows the timing diagram for “Integrated block for PCI express” IP.

Figure 15 MSI-X interrupt timing diagram for “Integrated block for PCI express” IP (Ref: PG213, June 24, 2019)

The below figure shows the timing diagram for the “DMA/Bridge subsystem” in “DMA” mode targeting UltraScale and UltraScale+ devices and the “AXI-Bridge” mode when targeting UltraScale+ devices. For the AXI-Bridge subsystem IP targeting UltraScale devices, please refer to Table 1 for equivalent signals.



Figure 16 MSI-X interrupt timing diagram for “DMA/Bridge subsystem” IP (Ref: PG195, June 20, 2019)


1. The interrupt process in the IP gets triggered by sampling the cfg_interrupt_int as high but not on the rising edge. 2. Even though cfg_interrupt_int is kept asserted HIGH, only one interrupt gets asserted

DMA/Bridge subsystem IP The interrupt is triggered on the rising edge of the usr_irq_req input “AXI-Bridge subsystem” IP

1. With the “AXI-Bridge subsystem” IP, the names of the ports are different. Table 1 shows the mapping of this signal to the signal names shown in the timing diagram above.

2. The interrupt is triggered on the rising edge of the usr_irq_req input

Note: The interrupt input ports are required to be synchronous to the user_clk (or) axi_aclk domain based on the target IP.

MSI-x related features with different IP cores

UltraScale Integrated Block for PCI express The MSI-X vector table and the MSI-X Pending Bit Array need to be implemented as part of the user logic, by claiming a BAR aperture. AXI Bridge for PCI express Gen3 Subsystem (Targeting UltraScale devices)

• The MSI-X vector table and the MSI-X Pending Bit Array need to be implemented as part of the user logic, by claiming a BAR aperture.

• The MSI-x table needs to be implemented outside of the IP. The “Internal” option is not present.

DMA/Bridge Subsystem for PCI Express in “DMA” mode (Targeting UltraScale/UltraScale+ devices)

• The MSI-X vector table and the MSI-X Pending Bit Array is implemented as part of the core. This is applicable for both UltraScale and UltraScale+ devices.

• The MSI-x table is always implemented as “Internal”. The “External” option is not available. UltraScale+ Integrated Block for PCI express The MSI-X feature has different options.

1. MSI-X External -- The MSI-X vector table and the MSI-X Pending Bit Array need to be implemented as part of the user logic, by claiming a BAR aperture.



2. MSI-X Internal -- The MSI-X vector table and the MSI-X Pending Bit Array are implemented as part of the core 3. MSI-X AXI4-Stream -- Applications that need to generate MSI/MSIX interrupts with traffic class bits not equal to 0

or address translation bits not equal to 0 must use the RQ interface to generate the interrupt (memory write descriptor)

DMA/Bridge Subsystem for PCI Express in “AXI-Bridge” mode (Targeting UltraScale+ devices) MSI-X feature has different options.

1. MSI-X External -- The MSI-X vector table and the MSI-X Pending Bit Array need to be implemented as part of the user logic, by claiming a BAR aperture. Note: the MSI-X External option can only be set when “Advanced” mode is selected in the Basic Tab of the IP GUI.

2. MSI-X Internal -- The MSI-X vector table and the MSI-X Pending Bit Array are implemented as part of the core

How to confirm interrupt is sent

All of the interrupts get transmitted to the host in the form of a TLP. Legacy interrupts are sent as message TLP’s, MSI/MSI-x interrupts are sent as memory write TLP’s with data. The data with the memory write TLP is the “message data” field in the MSI capability space (or) in the vector table with MSI-x capability. Simulation At the end point side, TLP’s related to the interrupts can be observed at the PIPE interface between the PCI express integrated block and the GT’s. Analyzing the TLP’s at this interface will require disabling of the scrambling inside the IP. Refer to the “Descrambling data at PIPE interface” section of this document for steps to descramble data at the PIPE interface. At the root port side, TLP’s related to the interrupts can be observed on m_axis_cq interface. Hardware With the XDMA IP in “DMA” mode (or) “Bridge” mode, IRQ registers help to know the status of the interrupts. Also, cfg_interrupt_* ports at the hardblock interface can be probed to understand the interrupt status. Refer to the “Descrambling data at PIPE interface” section of this document for steps to descramble data at the PIPE interface. . After this, capture the pipe interface signals when the interrupt is asserted. Figure 4 shows the protocol analyzer captures of the two messages transmitted by the end point in response to assertion and de-assertion of Legacy interrupts. In case of MSI/MSI-x, the interrupt will be seen as a memory write TLP with data. In Linux, you can run the command “cat /proc/interrupts/” to trace the interrupts received by the host.



Column 1: IRQ number Column 2 and 3: # of interrupts per CPU (variable # of columns depends on how many CPUs your system has) Column 4: Type of interrupt Column 5: type of irq for example: "edge" irq Column 6: Name of device

Case Study

Issue 1:

Reading the Table offset from Configuration space does not show a value 0x8000. Solution: If you are using the IP from Vivado 2018.1, please apply the patch from (Xilinx Answer 71169)

Issue 2:

Consecutive reads (or) writes from the configuration space using the configuration management interface makes PCIe unresponsive. Solution: Keeping cfg_mgmt_rd_en (or) cfg_mgmt_wr_en asserted continuously could cause this issue. Per the timing diagram of configuration management interface mentioned in Figure 18 cfg_mgmt_wr_en must be de-asserted after cfg_mgmt_rd_wr_done is pulsed. A new write should be initiated only after this.

Issue 3:

When more than one MSI interrupt is enabled with the “AXI Bridge Subsystem” IP targeting UltraScale devices, msi_vector_width is always 0 Solution: When more than one interrupt vector is enabled, the “Multiple Message Enable” bits need to be set. Refer to Figure 7: Multiple Message Enable bit description (Ref: PCI LOCAL BUS SPECIFICATION, REV. 3.0).




Issue 4:

When initiating an interrupt multiple times, the interrupt is not acknowledged for all instances. For example, if the interrupt is initiated 10 times (pulse on usr_irq_req or intx_msi_request), it is acknowledged (pulse on usr_irq_ack or intx_msi_grant) by the IP 5 times. Solution: usr_irq_req or intx_msi_request needs to be asserted synchronous to user_clk (or) axi_aclk based on the IP. Otherwise, the IP does not identify that the interrupt is initiated by the user.

Recommendations and Gotchas

Initiate multiple interrupts at once With the XDMA IP, usr_irq_req and usr_irq_ack are multi-bit signals. As a result, it is possible to send interrupts from multiple sources at once. However with the AXI-Bridge subsystem IP and the Integrated block for PCIe IP, this option is not available. Keep a note of the Byte address Vs Dword address for configuration space In Simulation, the TSK functions provided with the IP example design accept “Byte address”. With the cfg_mgmg_interface, the address field needs to be driven with “Dword address” When using the “setpci” command on the hardware “Byte address” needs to be used to access the registers in the configuration space. Note: the offset to the registers in the configuration space are not same across different devices What can cause msi_fail assertion? This can occur when the host is allocated 1 interrupt, but the user is trying to send the second interrupt. In this case asserting an interrupt using cfg_interrupt_int[0] will result in a pulse on cfg_interrupt_sent but using cfg_interrupt_int[1] will result in a pulse on cfg_interrupt_fail. How to know which interrupt mode is active? If Legacy/MSI/MSI-x interrupts are enabled together in a design, the figure below shows which interrupt mode is active based on the status of msi_enable and msix_enable outputs.

Special considerations



From (PG156) v4.4 April 4, 2018 and PG213 v1.3 June 24,2019 The user logic requires ordering to be maintained between a request TLP and MSI/MSI-X TLP signaled through the MSI Message interface. In this case, the user logic must wait for the sequence number of the requester request to appear on the pcie_rq_seq_num[3:0] output before signaling MSI or MSI-X on the MSI Message interface. Check how many interrupts are allocated by host during enumeration The figure below shows example lspci output in which MSI interrupts are enabled, 1 interrupt is requested, and 1 interrupt is allocated by the host. If the end point requests 4 interrupts and the host allocates 1 interrupt, you will find “Count=1/4”.

AXI Bridge for PCI express Gen3 Subsystem (Targeting UltraScale devices) MSI_Vector_Width output port can be monitored to know the number of interrupts allocated by the host. This output is a 3-bit signal, the figure below shows the mapping of this output to the number of interrupts allocated.

Figure 17 MSI Vectors enabled in Message Control Register



UltraScale/UltraScale+ Integrated Block for PCI express

The DMA/Bridge Subsystem for PCI Express IP in “DMA” (or) “AXI-Bridge” mode does not have a port to specify number of MSI interrupts allocated by the host. How to send interrupts to different PF’s and VF’s? This is defined in this table by setting cfg_interrupt_msi_function_number to the correct value.

How to acknowledge the interrupt? From (PG194) v3.0 July 16,2019 and (PG195) v4.1 June 20,2019: After a usr_irq_req bit is asserted, it must remain asserted until the corresponding usr_irq_ackbit is asserted and the interrupt has been serviced and cleared by the Host. The user_irq_ack assertion indicates that the requested interrupt has been sent to the PCIe block. This will ensure that the interrupt pending register within the IP remains asserted when queried by the Host's Interrupt Service Routine (ISR) to determine the source of interrupts. You must implement a mechanism in the user application to know when the interrupt routine has been serviced. This detection can be done in many different ways depending on your application and your use of this interrupt pin. This typically involves a register (or array of registers) implemented in the user application that is cleared, read, or modified by the Host software when an Interrupt is serviced.



Appendix

Configuration Management Interface Access

To access PCIe configuration space, Configuration Management Interface option can be enabled. The timing diagram for initiating read and write access through this interface can be found in (PG213)/(PG156) product guides. Also, these product guides need to be referred to in order to know the offset address for different configuration space registers. Only these documents will have this information. Product guide documents for other IP cores for example XDMA/AXI Bridge subsystem will not have this information. To ensure proper access with the Configuration Management Interface, cfg_mgmt_wr_en (or) cfg_mgmt_rd_en needs to be released along with the cfg_mgmt_rd_wr_done signal as shown in the figure below.

Figure 18 Configuration Management Interface timing diagram

Note: the configuration space with Integrated block for PCIe in different devices can have different offset address to MSI/MSI-x capability. Please refer to the product guide of the target Integrated block for PCIe. The address value to be given for the cfg_mgmt_addr must be DWORD address. The address values mentioned in the product guide is a Byte address. You will need to divide the value by 4 to get the DWORD. The DWORD address of the Command/Status Register in the PCI Configuration Space Header is 01h. (The byte address is 04h.) In the hardware test, add the configuration management interface to the VIO and drive them to write to the required registers in the configuration space. Make sure that the timing requirement mentioned above is taken care. In Simulation, write/read to the configuration space can be taken care of using TSK_TX_TYPE0_CONFIGURATION_READ and TSK_TX_TYPE0_CONFIGURATION_WRITE. Example code for how to use this task function can be found in the test bench generated with the IP example design:

Using the “set pci” command to access configuration space

“setpci” is another method to read/write to the configuration space registers. The commands below are one example of the different ways to use this command.

setpci -s 06:00.0 04.b --> read byte



setpci -s 06:00.0 04.w --> read word setpci -s 06:00.0 04.l --> read Dword setpci -s 06:00.0 04.w = <value> --> write to config space “-s” → slot “06:00.0” → b:d.f → Bus:Device.Function “04” → offset address in configuration space.

Note: The address offset mentioned above is “Byte Address”

JTAG2AXI Master to access the AXI-Lite register space

With the XDMA and AXI-Bridge subsystem IP cores, MSI/MSI-x interrupts can be mapped to different vectors and they can be masked selectively. These controls are available in the registers implemented with the IP. These registers can be accessed using the AXI-Lite Slave (or) AXI-Lite control interface available with these two IPs. Note: selective masking of the interrupts is not present with the Integrated block for PCIe IP cores. This is only present with XDMA, AXI-Bridge subsystem IP cores. The JTAG2AXI master IP can be used to access the registers. Refer to (Xilinx Answer 71322) for guidance on how to use this IP to access registers through the AXI-Lite interface. In simulation, access to the BAR space can be done using TSK_TX_BAR_READ and TSK_TX_BAR_WRITE. Example code to know how to use this task function can be found in the test bench generated with the IP example design.

Descrambling data at PIPE interface

Analyzing the TLP’s at the PIPE interface will require disabling of the scrambling inside the IP. Based on the rate at which the link is operating, different options are required to descramble the data.

1. At Gen1/Gen2 rates -- DEBUG_PL_DISABLE_SCRAMBLING in the IP top files need to be set to “TRUE” to ensure that the data on PIPE interface is not scrambled.

2. At Gen3 rate – Enable the “Descrambler for Gen3” option in the IP GUI. Refer to (Xilinx Answer 72471) for guidance about using this feature.

References

(PG213): UltraScale+ Devices Integrated Block for PCI Express v1.3 (PG156): UltraScale Devices Gen3 Integrated Block for PCI Express v4.4 (PG194): AXI Bridge for PCI Express Gen3 Subsystem v3.0 (PG195): DMA/Bridge Subsystem for PCI Express v4.1 PCI Local Bus Specification Revision 3.0 (Xilinx Answer 58495) Xilinx PCI Express Interrupt Debugging Guide



https://www.xilinx.com/support/documentation/ip_documentation/pcie4_uscale_plus/v1_3/pg213-pcie4-ultrascale-plus.pdf

https://www.xilinx.com/support/documentation/ip_documentation/pcie3_ultrascale/v4_4/pg156-ultrascale-pcie-gen3.pdf

https://www.xilinx.com/support/documentation/ip_documentation/axi_pcie3/v3_0/pg194-axi-bridge-pcie-gen3.pdf

https://www.xilinx.com/support/documentation/ip_documentation/xdma/v4_1/pg195-pcie-dma.pdf

https://members.pcisig.com/wg/PCI-SIG/document/download/8237




Revision History

09/20/2020 - Initial release

xilinx answer 72702 ultrascale and ultrascale+ pcie ......figure 3 legacy interrupt timing diagram...

Documents