capi reference card - openpower foundationopenpowerfoundation.org/.../capi-ref-card-v1-1/capi...apr...

www.openpowerfoundation.org

http://openpowerfoundation.orghttp://www.openpowerfoundation.org

CAPI Reference Card April 2, 2019 Version 1.1

OpenPOWER Foundation iiWorkgroup Notes

Non-standard Track

CAPI Reference Card:Accelerator Work Group OpenPower Foundation

Version 1.1 (April 2, 2019)Copyright © 2017, 2018 OpenPOWER Foundation

All capitalized terms in the following text have the meanings assigned to them in the OpenPOWERIntellectual Property Rights Policy (the "OpenPOWER IPR Policy"). The full Policy may be found at theOpenPOWER website or are available upon request.

This document and translations of it may be copied and furnished to others, and derivative works thatcomment on or otherwise explain it or assist in its implementation may be prepared, copied, published,and distributed, in whole or in part, without restriction of any kind, provided that the above copyright noticeand this section are included on all such copies and derivative works. However, this document itself maynot be modified in any way, including by removing the copyright notice or references to OpenPOWER,except as needed for the purpose of developing any document or deliverable produced by an OpenPOW-ER Work Group (in which case the rules applicable to copyrights, as set forth in the OpenPOWER IPRPolicy, must be followed) or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by OpenPOWER or itssuccessors or assigns.

This document and the information contained herein is provided on an "AS IS" basis AND TO THEMAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, THE OpenPOWER Foundation AS WELLAS THE AUTHORS AND DEVELOPERS OF THIS STANDARDS FINAL DELIVERABLE OR OTHERDOCUMENT HEREBY DISCLAIM ALL OTHER WARRANTIES AND CONDITIONS, EITHER EXPRESS,IMPLIED OR STATUTORY, INCLUDING BUT NOT LIMITED TO, ANY IMPLIED WARRANTIES, DUTIESOR CONDITIONS OF MERCHANTABILITY, OF FITNESS FOR A PARTICULAR PURPOSE, OFACCURACY OR COMPLETENESS OF RESPONSES, OF RESULTS, OF WORKMANLIKE EFFORT, OFLACK OF VIRUSES, OF LACK OF NEGLIGENCE OR NON-INFRINGEMENT.

OpenPOWER, the OpenPOWER logo, and openpowerfoundation.org are trademarks or registeredtrademarks of OpenPOWER Foundation, Inc., registered in many jurisdictions worldwide. Other company,product, and service names may be trademarks or service marks of others.

Abstract

The document provides a detailed focus on the base design requirements of an FPGA based accelera-tor card utilizing CAPI 2.0. The document is not a static document it will be filled with additional informa-tion over time. The OpenPower members are invited to add information, ideas and approaches to thedocument. The Appendix has an open structure and implementations like adding a new FPGA or addinga common function block can be introduced at any time. Please provide your input to the OpenPow-er Accelerator Workgroup first for discussion. With a signoff, the new item will added to the currentdocument.

This document is a Work Group Note owned by the Accelerator Workgroup (AWG) and handledin compliance with the requirements outlined in the OpenPOWER Foundation Work Group(WG) Process document. It was created using the Master Template Guide version 1.0.0.Comments, questions, etc. can be submitted to the public mailing list for this document at.

Acknowledgement to members of the workgroup for their contributions

https://members.openpowerfoundation.org/document/dl/596


OpenPOWER Foundation iiiWorkgroup Notes

Non-standard Track

Table of ContentsPreface ......................................................................................................................................... vii

1. Conventions ...................................................................................................................... vii2. Document change history ................................................................................................ viii

1. Introduction ................................................................................................................................ 12. Reference Documents ............................................................................................................... 3

2.1. PCI-SIG® Card Electromechanical Specification (CEM) .................................................. 32.2. PCI-SIG Base Specification ............................................................................................ 32.3. OpenPower Compliance .................................................................................................. 32.4. OpenPower Coherent Accelerator Interface Architecture ................................................. 32.5. OpenPower Ready™ ...................................................................................................... 42.6. Field Replaceable Unit (FRU) Service Interface (FSI) ...................................................... 42.7. IPMI and MCTP .............................................................................................................. 4

3. FPGA and Core Components .................................................................................................... 53.1. XILINX® part definition ................................................................................................... 53.2. PSL Physical Pins ........................................................................................................... 53.3. Minimum of Target Devices (Base Version) ..................................................................... 63.4. Extended Implementation ................................................................................................ 73.5. IPMI support ................................................................................................................... 83.6. Card Control Subsystem Blocks .................................................................................... 103.7. Core - Function Definition ............................................................................................. 123.8. On Board additional I2C ................................................................................................ 13

4. Infrastructure/Pervasive ............................................................................................................ 154.1. VPD .............................................................................................................................. 154.2. Configuration Flash ....................................................................................................... 154.3. Card Clock Generator or Clock Buffer ........................................................................... 164.4. DIP Switches ................................................................................................................. 164.5. Programming Interface .................................................................................................. 16

5. Power Supply ........................................................................................................................... 185.1. I2C Control Interface ..................................................................................................... 185.2. Current Monitor ............................................................................................................. 185.3. Power Consumption ...................................................................................................... 18

6. Thermal .................................................................................................................................... 206.1. Passive Heat-sink - No Fan .......................................................................................... 20

7. Human Factors ........................................................................................................................ 217.1. LED Color Codes .......................................................................................................... 217.2. Programmer Access ...................................................................................................... 21

8. Pre-Defined I/O ........................................................................................................................ 228.1. Network or Other Interfaces .......................................................................................... 22

9. Mechanism to unlock a 'Bricked' Card ..................................................................................... 239.1. Interface ........................................................................................................................ 239.2. Common Procedure ...................................................................................................... 24

A. Design Guidelines ................................................................................................................... 25A.1. FPGA Implementation Guidelines ................................................................................. 25A.2. Future Updates ............................................................................................................. 31A.3. VPD Structure ............................................................................................................... 32A.4. Basic IPMI commands .................................................................................................. 32A.5. Reference Card - Hardware Details .............................................................................. 32A.6. Schematic Detatils ........................................................................................................ 34


OpenPOWER Foundation ivWorkgroup Notes

Non-standard Track

B. OpenPOWER Foundation overview ......................................................................................... 36B.1. Foundation documentation ............................................................................................ 36B.2. Technical resources ...................................................................................................... 36B.3. Contact the foundation .................................................................................................. 37


OpenPOWER Foundation vWorkgroup Notes

Non-standard Track

List of Figures1.1. Common Card Functions ........................................................................................................ 23.1. Base Version ........................................................................................................................... 73.2. Extended Version .................................................................................................................... 83.3. IMPI Block Diagram ................................................................................................................ 93.4. System view Future Card Management (CAPI and OpenCAPI) ............................................. 103.5. Future Card Management (CAPI and OpenCAPI) - Base Version .......................................... 113.6. Future Card Management (CAPI and OpenCAPI) - Extended Version ................................... 114.1. JTAG Footprint ...................................................................................................................... 179.1. Bricked Card - Base ............................................................................................................. 249.2. Bricked Card - Extended ....................................................................................................... 24A.1. KU15P Pin-out ...................................................................................................................... 25A.2. From PCIe edge connector to FPGA .................................................................................... 26A.3. VU3P Pin-out ........................................................................................................................ 26A.4. From PCIe edge connector to FPGA .................................................................................... 27A.5. VU33P Pin-out ...................................................................................................................... 28A.6. From PCIe edge connector to FPGA .................................................................................... 29A.7. ZU19EG Pin-out ................................................................................................................... 30A.8. From PCIe edge connector to FPGA .................................................................................... 31A.9. CAPI Accelerator Reference Card VPD - Structure ............................................................... 32A.10. Block Diagram .................................................................................................................... 33A.11. High Level Description of the SMBus Card Management: ................................................... 34A.12. SMBus Card Management: ................................................................................................. 35


OpenPOWER Foundation viWorkgroup Notes

Non-standard Track

List of Tables4.1. Example ................................................................................................................................ 155.1. Card Power Estimate (Example) ........................................................................................... 197.1. LED List - Example ............................................................................................................... 21A.1. IMPI Commands ................................................................................................................... 32


OpenPOWER Foundation viiWorkgroup Notes

Non-standard Track

Preface1. ConventionsThe OpenPOWER Foundation documentation uses several typesetting conventions.

NoticesNotices take these forms:

Note

A handy tip or reminder.

Important

Something you must be aware of before proceeding.

Warning

Critical information about the risk of data loss or security issues.

ChangesAt certain points in the document lifecycle, knowing what changed in a document is important. Inthese situations, the following conventions will used.

• New text will appear like this. Text marked in this way is completely new.

• Deleted text will appear like this. Text marked in this way was removed from the previous versionand will not appear in the final, published document.

• Changed text will appear like this. Text marked in this way appeared in previous versions but hasbeen modified.

Command promptsIn general, examples use commands from the Linux operating system. Many of these are alsocommon with Mac OS, but may differ greatly from the Windows operating system equivalents.

For the Linux-based commands referenced, the following conventions will be followed:

$ prompt Any user, including the root user, can run commands that are prefixed with the $prompt.

# prompt The root user must run commands that are prefixed with the # prompt. You can alsoprefix these commands with the sudo command, if available, to run them.


OpenPOWER Foundation viiiWorkgroup Notes

Non-standard Track

Document linksDocument links frequently appear throughout the documents. Generally, these links include a textfor the link, followed by a page number in parenthesis. For example, this link, Preface [vii],references the Preface chapter on page vii.

2. Document change historyThis version of the guide replaces and obsoletes all earlier versions.

The following table describes the most recent changes:

Revision Date Summary of Changes

December 5, 2018 • V1.1: Updated appendix add two additional pinouts

June 4, 2018 • V1.0: Minor changes based on AWG feedback ready for vote

May 28, 2018 • V1.0 - pre3: Minor changes based on AWG feedback

May 8, 2018 • V1.0 - pre2: Significant editorial updated from workgroup review

November 29, 2017 • V1.0 - pre1: Initial conversion from word document


OpenPOWER Foundation 1Workgroup Notes

Non-standard Track

1. IntroductionThe market offers many PCI Express® form-factor FPGA Accelerators. These cards come with acomplete implementation of PCI Express functions and sub-functions. There are many different carddesigns and functional implementations. Each one is unique and may require a unique FPGA imageto support the implementation's function blocks. Supporting a new card requires the Power ServiceLayer (PSL) be adapted to the new hardware. The effort to port the PSL and provide support for thenew environment can be high.

To limit variations in the hardware implementations and reduce the CAPI support costs theOpenPOWER Accelerator workgroup is publishing this document as a workgroup note. Thisdocument will define a baseline for the CAPI Accelerator Card design called the CAPI ReferenceCard. The document outlines the minimum content needed for any future PCI Express based CAPIAccelerator card. Ultimately, the intent is to limit the number of different PSL implementations but notlimit the functionality of future designs.

CAPI Reference Card Highlights

1. A common platform description of the core functions1 of a CAPI card.

The software stack2 will handle the defined core functions independent of the card design andindependent of application or vendor.

2. The Workgroup Note defines what must be implemented in the following important areas:

A. PSL (CAPI 2.0) implementation targeting the pinout of three different FPGA part numbersincluding the external devices to support the CORE Functions1 of the card

B. Pointers to Low Profile form factor PCI Express card Specifications according to PCI-SIG®and pointers to specifications that are relevant to the OpenPower Foundation.

C. Power sub-system aware service infrastructure description

• Interaction with the Thermal subsystem• Interaction with the Power subsystem• Interaction with the Pervasive subsystem

D. Common Interface definition including PCI Express and CAPI for

• High Speed• Low Speed• I/Os

E. Human Interface (HID)

F. Common card recovery procedure (how to recover from a bricked card)

3. The following choices are up to the designer?

• Component choice for most of the core functions1.

1Core function in the context of hardware implementation is describing a specific high-level circuitry forming a specific function.2Software stack in the context of the document is covering Operating System, hypervisor, and firmware interfaces to the CAPI card.



Non-standard Track

• Application specific system blocks as an add-on to the core of the design reference (above)

The core functions that are common to all cards are depicted in Figure 1.1, “Common CardFunctions” [2].

Figure 1.1. Common Card Functions



Non-standard Track

2. Reference DocumentsThe following chapter provides quick access to reference documents that are available on the WEB.PCI-SIG® https://pcisig.com/ provides comprehensive specifications to design PCIe based add-incards and motherboards. The specification that covers the whole mechanical, physical and electri-cal constrains is the CEM spec and additional details on implementing PCI Express® to the systemis covered by the Base Spec. The OpenPower WEB site will provide further information about thesystem platform with P9 and the compliance within the ecosystem. You can find this information atthe following link: https://openpowerfoundation.org/technical/resource-catalog/ .

You may need to be a member of PCI-SIG and OpenPower to be able to receive all the information.Many of the links in chapter 2 will only work if you have signed up for membership. This documentwill provide bookmarks within the various specifications to prevent you from going through all thedocumentation pages to find the one item that needs to be answered.

2.1. PCI-SIG® Card ElectromechanicalSpecification (CEM)The main focus of the CEM is the evolutionary strategy to implement PCI Express into systems.

The specification can be found here: https://pcisig.com/specifications/pciexpress/

2.2. PCI-SIG Base SpecificationThe PCI-SIG Base Specification describes the PCI Express architecture, interconnect attributes andthe programming interface.

The specification can be found here: https://pcisig.com/specifications/pciexpress/

2.3. OpenPower ComplianceThe OpenPOWER Compliance Work Group will define OpenPOWER key interfaces that need tobe compliant in a standard specification of compliance and define how to measure and documentcompliance pre-silicon and post-silicon. The work group product may be found at: https://openpowerfoundation.org/technical/resource-catalog/ searching on "compliance".

2.4. OpenPower Coherent Accelerator InterfaceArchitectureThis document defines the Coherent Accelerator Interface Architecture (CAIA) for the IBM®POWER8® and follow-on systems with CAPI 2.0. The information contained in this document allowsvarious CAIA-compliant accelerator implementations to meet the needs of a wide variety of systemsand applications. Compatibility with the CAIA allows applications and system software to migratefrom one implementation to another with minor changes. http://openpowerfoundation.org/wp-content/uploads/resources/hwarch-caia-spec/content/ch_preface.html

https://pcisig.com/https://openpowerfoundation.org/technical/resource-catalog/https://pcisig.com/specifications/pciexpress/https://pcisig.com/specifications/pciexpress/https://openpowerfoundation.org/technical/resource-catalog/https://openpowerfoundation.org/technical/resource-catalog/http://openpowerfoundation.org/wp-content/uploads/resources/hwarch-caia-spec/content/ch_preface.htmlhttp://openpowerfoundation.org/wp-content/uploads/resources/hwarch-caia-spec/content/ch_preface.html



Non-standard Track

2.5. OpenPower Ready™The OpenPOWER Ready Workgroup provides leadership and management of all aspects of theOpenPOWER Ready program. Key aspects are the OpenPOWER Ready Definition and Criteria,review of OpenPOWER Ready requests, and outreach to encourage participation.

Information about the OpenPOWER Ready program is available here: https://openpowerfoundation.org/technical/openpower-ready/

2.6. Field Replaceable Unit (FRU) ServiceInterface (FSI)The FSI interface is a serial data link that operates in half duplex mode. Devices communicate inmaster/slave mode where the master device initiates a data transfer. FSI is a point-to-point two wireinterface which is capable of supporting distances of up to 4 meters at up to 166 MHz bus frequency.

Key features are ease of use, easy scalability, robustness, and support for virtualisation and thetunneling of interrupts and DMA control signals across the interface. FSI is superior to similarindustry standard interfaces in these and many other features including speed, distance, data protec-tion, and address range.

FSI interfaces have been used successfully for many years in IBM servers to attach IBM FlexibleSupport Processors to CPUs and IBM ASICs.

Information about the FSI Specification is available here: https://openpowerfoundation.org/?resource_lib=field-replaceable-unit-fru-service-interface-fsi-openfsi-specification

2.7. IPMI and MCTPThe Intelligent Platform Management Interface (IPMI) is a set of computer interface specificationsfor an autonomous computer subsystem that provides management and monitoring capabilitiesindependently of the host system's CPU, firmware and operating system. IPMI defines a set ofinterfaces used by system administrators for out-of-band management of computer systems andmonitoring of their operation. For example, https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/platform-management-fru-document-rev-1-2-feb-2013.pdf provides away to manage a computer that may be powered off or otherwise unresponsive by using a networkconnection to the hardware rather than to an operating system or login shell.

The Management Component Transport Protocol (MCTP) ( http://www.dmtf.org/sites/default/files/standards/documents/DSP0237_1.0.0.pdf ) is a protocol defined by the DMTF Platform ManagementComponent Intercommunications sub-team of the DMTF Pre-OS Workgroup. MCTP is designed tosupport communications between different intelligent hardware components that make up a platformmanagement subsystem that provides monitoring and control functions inside a managed system.

https://openpowerfoundation.org/technical/openpower-ready/https://openpowerfoundation.org/technical/openpower-ready/https://openpowerfoundation.org/?resource_lib=field-replaceable-unit-fru-service-interface-fsi-openfsi-specificationhttps://openpowerfoundation.org/?resource_lib=field-replaceable-unit-fru-service-interface-fsi-openfsi-specificationhttp://www.dmtf.org/sites/default/files/standards/documents/DSP0237_1.0.0.pdfhttp://www.dmtf.org/sites/default/files/standards/documents/DSP0237_1.0.0.pdf



Non-standard Track

3. FPGA and Core Components3.1. XILINX® part definitionTo reduce the effort in porting the PSL (Power Service Layer) the reference design currently targetsonly three different types of FPGAs. Based on XILINX® roadmap newer parts will be introducedover time and the specification will be updated accordingly. Because of this restriction to three sizesthe time consuming part of fine tuning to get placement, routing, and timing closure of the designcompleted is minimized.

Note

All future updates to the FPGA types will be found in the Section A.1, “FPGA Implemen-tation Guidelines” [25].

3.1.1. Small FPGADefinition in this document: Big enough to support PSL plus smaller AFUs.

Details on the supported package and further specific information on the FPGA being used can befound at Section A.1, “FPGA Implementation Guidelines” [25].

3.1.2. Medium FPGADefinition in this document: Extend the capability of the above component and provide mid-sizeapplications

3.1.3. Large FPGADefinition in this document: Small enough to fit on a low profile PCI Express card not violating thePCISIG CEM. Fan-out of the BGA and the physical size of the package as well as the low profilecard are limiting factors.

3.2. PSL Physical PinsFrom the above defined FPGA sizes a more detailed description of the pin-out will be provided inSection A.1, “FPGA Implementation Guidelines” [25] in each of the FPGA type sub-sections.

3.2.1. PCIE x8 basic pin outPCIe x8 edge connector to FPGA RX/TX pins (Subset of x16) can be found here:

KU15P: Section A.1.1, “KU15P” [25]

VU3P: Section A.1.2, “VU3P” [26]




Non-standard Track

ZU19EG: Section A.1.4, “ZUG19EG” [30]

3.2.2. PCIE x16 basic pin outPCIe x16 edge connector to FPGA RX/TX pins can be found here:

KU15P: Section A.1.1, “KU15P” [25]



ZU19EG: Section A.1.4, “ZUG19EG” [30]

3.2.3. I2C / SMBus at PCIe edge Connector and on CardPCI-SIG defines the two SMBus pins (clock and data) at the PCI Express edge connector locationB5 and B6 as "optional"; meaning there is no definition for a particular target device connected onthe add-in card to those pins. However, for the CAPI Reference Card these pins are defined as must have and connected to at least a minimum amount of defined devices (block diagrams below)to support the CAPI card system management function-block. An advanced managment functionusing a micro controller might be implemented. It is important that this SMBus interface is working atSystem Power Standby for the components in red below. The electrical specification of this interfacecan be downloaded here: http://smbus.org/specs/ .

Important

The designer should avoid overloading the SMBus. Assess the number of direct connect-ed devices to this interface.

3.3. Minimum of Target Devices (Base Version)One important device attached to the SMBus/I2C interface is the serial EEPROM Vital ProductData (VPD). It holds the card hardware configuration and unique identification information. Thedetailed information which needs to be contained in the VPD PROM will be explained later in thisdocument in Section A.3, “VPD Structure” [32]. Optional devices for temperature, current andvoltage reading may be connected via this interface, but is not mandatory to do so.

http://smbus.org/specs/



Non-standard Track

Figure 3.1. Base Version

Note

The devices in the circle do not need to be reachable at standby. The components shownin blue are mandatory for a Base Version implementation to provide the 'unbrick' capabil-ity.

3.4. Extended ImplementationIt is possible to use a single micro-Controller instead of using the discrete set of components shownin Figure 3.1, “Base Version” [7]. In this system the VPD will point to the micro-Controller baseaddress. The micro-Controller will use SMBUS protocol to communicate with the BMC or hostprocessor enabling additional features and functions..



Non-standard Track

Figure 3.2. Extended Version

Note

The components shown in blue support the 'unbrick' mechanism. From the high levelsoftware the procedure will use the same commands to 'unbrick' as the base versionabove.

3.5. IPMI supportFrom the system management software level, all CAPI 2.0 cards should provide the same commonfeatures. To provide this capability a reduced set of IPMI commands will be introduced to providesystem management capabilities. You will find the command set at Section A.4, “Basic IPMIcommands” [32]. These commands are implemented using the PCIe edge connector SMBusinterface.

Figure 3.3, “IMPI Block Diagram” [9] depicts the management system structure from the IPMIspecification.

The design point has changed for current and future OpenPower Systems. The SMBus is directlyconnected to the host processor. The Baseband Management Controller (BMC) is connected to thehost processor via I2c or FSI. For information about FSI see Section 2.6, “Field Replaceable Unit(FRU) Service Interface (FSI)” [4].

The block-diagram shown in Figure 3.3, “IMPI Block Diagram” [9] is incomplete as it shouldshow the host processor as part of the control and management subsystem. For the next generationof Power processors (Power10 and follow-on) an internal circuit enables the capability to route theI2C bus through the processor's root complex into the PCIe - subsystem to provide control capabili-ties into the PCIe subsystem via SMBus while the system is in 'Standby'.



Non-standard Track

Figure 3.3. IMPI Block Diagram

Note

BMC only block diagram. Newer Power systems provide the PCIe signals plus thesideband signals from the Root Complex of the processor. Seen Figure 3.4, “Systemview Future Card Management (CAPI and OpenCAPI)” [10].

The diagram in Figure 3.3, “IMPI Block Diagram” [9] shows the current implementation based inthe IPMI SPEC.

Connected to the BMC or Service Processor, the PCI Management Bus will be connected to each ofthe PCIe add-In cards in the chassis. The system board will manage the connectivity to each of thecards.

The block-diagram shown in Figure 3.4, “System view Future Card Management (CAPI andOpenCAPI)” [10] depicts the future card management for the CAPI and OpenCAPI cards. Thepath to manage and service the FPGA accelerator is the SMBus into the card. The first target deviceon this bus will be the VPD PROM. it will hold the content that is reflecting the hardware implemen-tation on the accelerator card. See Section 3.2.3, “I2C / SMBus at PCIe edge Connector and onCard” [6].



Non-standard Track

Figure 3.4. System view Future Card Management (CAPI and OpenCAPI)

Note

The function blocks depicted inside the CAPI 2.0 card belong to the core functionsdefined for an OpenCAPI card and provide system management capabilities.

3.6. Card Control Subsystem BlocksThe following function blocks connected via the red lines are mandatory for the CAPI card designand apply to both the base- and the extended version. Additionally, you also need to provide functionblocks (light red) to report temperature, current and status of the card. The base addresses arestored to the VPD but the logical implementation of those blocks are up to the designer.



Non-standard Track

Figure 3.5. Future Card Management (CAPI and OpenCAPI) - Base Version

Figure 3.6. Future Card Management (CAPI and OpenCAPI) - Extended Version



Non-standard Track

NoteThe Block Diagrams in Figure 3.5, “Future Card Management (CAPI and OpenCAPI) -Base Version” [11] and Figure 3.6, “Future Card Management (CAPI and OpenCAPI)- Extended Version” [11] show both implementations.

It is left to the designer to determine which implementation is used. The version chosenmust be reflected in the VPD content. The correct pointer to the control device (I/Oexpander or uController) must be store at a defined address for the different configura-tions.

The structure and content of the VPD is defined and will be explained in detail later inthis document.

In addition to the 'Core' designs depicted above, application specific functions and pin-out maybe included in Section A.2, “Future Updates” [31] of this document at a later date. Commonlyused pin-outs can be found in Section A.1, “FPGA Implementation Guidelines” [25]. Examples ofadditional functions which could to be included at a later date are:

1. DRAM Pins (one SODIMM)

2. Application Interface (one SFP or QSFP)

DRAM and Network interface are not required for all applications hence there is no need to addthem to the 'Core'. But the support interface for both functions shall be accessible under thesame base address if DRAM and/or SFP is used in the application. For further details, pleasesee Section 3.8, “On Board additional I2C” [13] in this document.

3. Additional features suggested by actively participating OpenPOWER Foundation AcceleratorWorkgroup members.

3.7. Core - Function Definition1. VPD with a fixed 7-bit address (0x50)2. uController (implementation dependent)3. I/O Expander (implementation dependent)4. I2C Isolation5. Thermal6. Clock7. Power8. LED9. FPGA10. Configuration Flash

Note: Please find an implementation example in Section A.3, “VPD Structure” [32]. Theimplementation details like pin-out, etc. might vary based on physical implementation.

Note

Example of SMBus Address table:

Isolation: PCA9515 or equivalent



Non-standard Track

GPIO: PCF8574 or equivalent PCA9538 (Adr. 0x40)VPD: 24C64 (Adr. 0x50)uController: (Adr. 0x9C)

A detailed address list can be found in the MCTP specification starting from page 40.

3.8. On Board additional I2CSmall Form-Factor Pluggable (SFP) type high speed interconnect modules use a managementinterface to read the current status of the module or to modify its setting. The SFP cages use I2Cinterfaces to communicate. It is good practice to openly define the additional I2C Bus implementationwith addresses in the hardware manual that accompanies the CAPI card. This also applies to:

1. Power Subsystem

2. Thermal Sensors (if not attached to the edge connector)

3. Clock

4. Other functions that provide information at runtime.

3.8.1. I/O Control and StatusThe I/O control subsystem depends on the target application. There are network interface moduleswith I2C interface (SFP, QSFP etc.) to provide specific parameters to the high speed interface. Theremight exist a storage device like NVME that will need to be connected to SMBus to read status orprovide information from its internal SEEPROM.

This is an example of an application driven feature and is not part of the Reference Card Document.If OPF members are willing to open their implementation, there is room at the appendix to providesuch details in the future. The appendix will be worked on by the AWG and published when releasedby the OpenPower Accellerator WG Web site. This will increase the library of functions that areavailable to the OpenPower members and will reduce effort and possible false implementations toothers.

3.8.2. EnvironmentalThe basic environmental function set may be represented by one thermal sensor but can be extend-ed to have more thermal measurement probe points in the system. It might be necessary to provideadditional thermal measurement points but this up to the application design point. There is onesensor mandatory on the CAPI card. The designer has the choice of the following measurementpoints:

1. FPGA Die Temperature2. Air Intake Temperature3. Air Exhaust Temperature

The final design point may cover more temperature probe points but only one of the above ismandatory to report thermal status of the FPGA CAPI card.

The final configuration can be retrieved from the VPD where the hardware configuration is stored. Toaccess the FPGA Die temperature, it is recommended to use a temperature sensor on the SMBus



Non-standard Track

that measures the internal diode of the FPGA. For air temperature sensors it is recommended toelevate the sensor off the surface of the PCB to reduce the influence of the PCB temperature.

The designer should select the best possible location to minimize the measurement error.

This workgroup note recommends using the LM75 which is an industry standard temperature sensordevice. This note does not require a specific device and there are many register and pin compatibledevices available. If a device that is not register compatible to the LM75 is used, the designer shouldprovide a reference outlining the differences. The register memory map for this alternate device maybe documented in the vendor's board hardware reference guide.

This delta can be coded into a table to translate the current sensor output to a LM75 compatible part.The details are shown in Section A.3, “VPD Structure” [32].



Non-standard Track

4. Infrastructure/Pervasive4.1. VPDAs mentioned before in Section 3.2.3, “I2C / SMBus at PCIe edge Connector and on Card” [6], inthe CAPI Reference Design the VPD will hold important information about the FPGA card system.The following chapter will provide the details of the content that will be stored in the VPD. TheOpenPOWER Accelerator Workgroup has provided a detailed spreadsheet that demonstrates theVPD content with function length and type. This spreadsheet may be downloaded at the followinglink https://openpowerfoundation.org/wg/aclwg/document/1576

4.1.1. Minimal ContentThe absolute minimum content of the VPD is detailed in the VPD Content Spreadsheet. Forimplementation see the 'Required' column marked with 'x' for minimal content considerations.

The VPD architecture was adapted from the PCI Local Bus specification. The definition can befound at Appendix I (page 315 here: https://pcisig.com/specifications/pciexpress/ ). The table atSection A.3, “VPD Structure” [32] shows the structure of the data and the length of each field aswell as the field definition. The additional fields will be explained in the following chapter.

4.1.2. I2C Base Addresses in VPDThe accelerator card VPD I2C base address is pre-defined at: 0x50. Other I2C accessible deviceswith their Base Address that are additionally implemented shall be stored to the VPD. The followingis a short example on how the base address can be retrieved.

More details can be found at the Section A.3, “VPD Structure” [32].

Table 4.1. Example

Length(Bytes)

Value (IBM example) Format

2 "V5" (I/O Expander I2C Device Address) ASCII

1 1 Binary

1 0x21 Binary

4.2. Configuration FlashThe configuration flash belongs to the FPGA and is part of the CAPI card infrastructure.

It is up to the designer to select which Flash device is implemented for the card design. The cardhardware manual should provide details. (for example: size and partitioning) A defined section in theFlash will be used to store the 'recovery image' in the Base - Configuration and shall be protectedfrom overwrite.

It it is good practice to define the address space in the hardware manual for the card.

https://openpowerfoundation.org/wg/aclwg/document/1576https://pcisig.com/specifications/pciexpress/



Non-standard Track

4.2.1. Default User Image Location and Fallback locationfor card recoveryTo enable recovery of an image after a faulty programming session or a broken update event theadd-in card can be restored to the 'factory- or fallback- image'. At a minimum the structure should be:

1. Base Address and size of Image 1 (factory)2. Base Address + factory image and size of Image 2 (user)

4.3. Card Clock Generator or Clock BufferIn addition to the 100 MHz PCIe clock, a programmable Clock generator may be added to the designto provide a reference clock for the predefined fabric . The I2C base address can be read backfrom the VPD if the designer provided such information. It might be necessary to define a specif-ic command to or from the Clock - Tree. In this case the field in the VPD must provide the baseaddresss of the clocking device or devices.

4.3.1. Status Register (read only)The Clock generator's status and settings can be read via the I2C interface. The register addressesfor the status and settings are stored in the VPD. The device specific registers should be describedin the CAPI card hardware manual. Alternately, a link to the card datasheet from the manufacturer'swebsite location may be provided.

4.4. DIP SwitchesIn the case of specific functions enabled by jumpers or switches the card documentation must definethe purpose and the default setting in the card user guide.

4.5. Programming InterfaceThe accelerator card must have a JTAG interface to program the FPGA. The pins or the PCB solderpads are defined based on the XILINX JTAG adapter pin out. See Section 4.5.2, “JTAG Pin assign-ment / Footprint” [17] in this document.

4.5.1. On Board - Module (optional)To provide an easy way to program the FPGA an additional USB to JTAG converter module may beimplemented. A USB to JTAG Module is not part of the Reference Design but the following minimumrequirements should be implemented in case such a USB programing interface has been provided.

• The module must have the same 'pin-out' for the JTAG interface as defined by the XILINXwebsite. The signals and the associated pins are defined in Section 4.5.2, “JTAG Pin assign-ment / Footprint” [17] below.

• The programming module must be supported by XILINX - Vivado development suite.

On Module USB Connector:To reduce the number of USB cable types, it is recommended that designers use Micro USB. Thepreferred location of the USB female connector should be towards the tailstock of the host system

http://www.xilinx.com/products/design-tools/vivado.html



Non-standard Track

to provide easy access for users. If real estate restrictions will not provide enough space at the cardbracket other implementations might solve this requirement. One possible solution could be accessfrom the top of the card so that the programming interace can be reached when the host system topcover is removed.

Connector Interface:If the USB programming module has more pins for voltages and control signals than the previouslydefined interface in

Section 4.5.2, “JTAG Pin assignment / Footprint” [17]; then the predefined pins remain untouchedand the additional pins will extend the footprint. In this case the new pin-out is a superset of thedefinition above while keeping the pin numbers in the subset the same.

4.5.2. JTAG Pin assignment / FootprintA possible JTAG Interface implementation is defined in the XILINX documentation. Implementing theexact same header that is been used by XILINX might not be practicable in some designs so In caseof a different header or connector for the JTAG interface it is advised to keep the pin-numbers for theJTAG - signals the same. This will reduce errors when connecting the programmer or a programmingmodule.

The dual row header and the platform Cable can be found here: XILINX - WEB page

The dimensions and pin-out are shown in Figure 4.1, “JTAG Footprint” [17]

Figure 4.1. JTAG Footprint



Non-standard Track

5. Power SupplyThe onboard power supply architecture will be different based on the FPGA and/or the application.The available chips on the market do not have the same pin-out and internal register sets etc. A lowlevel compatibility can be achieved via the control bus of the power controller and its on chip definedvendor ID and chip ID. This information can be retrieved via I2C from the power controller chip. Byknowing the vendor and the chip the control register-set can be derived. The VPD can be used topoint to vendor defined registers with associated addresses. After determining address and vendorIO, a SMBus compatible power supply will act as a sensor for voltage, current, and power consump-tion. Details shall be supplied in the card vendor's hardware reference manual.

5.1. I2C Control InterfaceThis sub section will describe how designers may handle the I2C interface. This I2C interface doesnot need to be accessible during standby. The power up and down sequence and other parametersthat are of importance to the FPGA is defined by XILINX. These details are described in the productfamily datasheet. The datasheet may be found at https://www.xilinx.com/products/silicon-devices/fpga/kintex-ultrascale-plus.html#documentation .

5.1.1. I2C Base AddressThe chip's base address is stored to the VPD and can be retrieved from a pre-defined location.Please find the details in Section A.3, “VPD Structure” [32].

5.2. Current MonitorThe accelerator card shall have the capability to report the current on the 12V input. A PCI Expressx16 Card is specified at 75W total. For a x8 PCIe card the integrator needs to verify the maximumslot power from the host system datasheet in case the CAPI card is over this limit. Otherwise the25W power limit will be the maximum based on PCI-SEG CEM. This includes the power on the 3.3Vlogic as well as the 3.3V standby. The maximum current is specified at 5.5A and the on-card capaci-tance should not exceed 2000uF. The system motherboard must be able to withstand the inrushcurrent which is much higher than the maximum continuous current for the FPGA card. The powercontroller or another circuit connected to I2C must provide the current monitor capability. If the I2Cbase address of the current monitor is different than the power controller the details can be retrievedfrom the fields described in Section A.3, “VPD Structure” [32].

5.3. Power ConsumptionXILINX provides a comprehensive Power Estimator UG to calculate the total power consumption ofthe FPGA when utilized. Please use the tool to provide the power numbers for your application anddescribe this in the card vendor hardware design manual.

5.3.1. Current per device of the accelerator cardThe Power Estimator for the FPGA can only calculate the FPGA power. There are other componentsand circuits that will draw substantial current; DRAM components for example. Table 5.1 could be

https://www.xilinx.com/products/silicon-devices/fpga/kintex-ultrascale-plus.html#documentationhttps://www.xilinx.com/products/silicon-devices/fpga/kintex-ultrascale-plus.html#documentation



Non-standard Track

used to summarize the contributors to the overall power consumption of the card. It is recommendedthat the card vendor hardware design manual include a summary table simlare to like Table 5.1.

Table 5.1. Card Power Estimate (Example)Function Total power Tolerance

DRAM

Network

FPGA



Non-standard Track

6. ThermalMany host systems use forced air for cooling. In order to support these applications the CAPI cardmust be designed with a passive 'heat sink' which does not require an active fan or blower attachedto the card. Air flow for cooling will come from the system fans only. Servers using Power9 and followon systems have different chassis and the air flow will likely vary between the different servers. Thedesigner must provide thermal characteristics vs. airflow rates and maximum junction temperatures.

This data will usually come from thermal simulations and tests and will show the corner cases. Itis good design practice to provide the thermal characteristics in the hardware design guide of theFPGA card to enable system integrators to estimate if the cooling system can support such cards.

6.1. Passive Heat-sink - No FanThe airflow within the system is from front to back. To provide optimum cooling the heatsink fins mustbe oriented in the direction the air is forced through the fins.

Mounting the heat-sink will add pressure to the BGA balls. These forces can be controlled but forcescaused by the environment should also take in into account. Mechanical support of the heat-sink isvery important to ensure no additional mechanical stress to the components underneath the heat-sink. Pressure from the top or shearing force caused by shock and vibration must not add X/Y/Zforces to the BGA balls or traces on the PCB. It may be necessary to add a stiffener underneath theFPGA to prevent cracking the traces or pads when vibrations occur.



Non-standard Track

7. Human FactorsWhen the card is installed in the system and the system cover is closed internal LEDs are notvisible and internal connectors are out of reach. As a result, there are vital functions that need tobe available from the bracket of the card. The following subsection will describe a minimum set ofindicators that shall be available from the tailstock of the system.

7.1. LED Color CodesThere is important information that needs to be visible when the system is installed in the rack. Thefollowing status LEDs need to be on the tailstock (bracket) so that the current status (card active/working or card down/defective) can be seen from a distance. The 'card active/working' state will beshown with a green LED. The 'card down / defective' state will be shown with a red LED. The use ofa single bi-color LED is also valid as long as the colors are red and green.

There may be additional LEDs on the card to show different states of the hardware or software andthose LEDs may be placed freely on the PCB.

All LEDs should be described in a table similar to Table 7.1.

Table 7.1. LED List - ExampleLED Function LED Color Location

Card active green Tailstock, Bracket

Card fail red Tailstock, Bracket

7.2. Programmer AccessIdeally, the programming interface should be reachable when the card is installed into a system. Theoptimal solution would be on the tailstock (bracket) but this might not be practical because the lowprofile card may not have enough available space. It is up to the designer where to place the JTAGDongle connector. Restrictions are based on the outline drawings from PCISIG CEM.



Non-standard Track

8. Pre-Defined I/OA minimum number of I/O pins are predefined to the FPGA to support CAPI 2.0. This is a subsetof what is available from the high speed I/Os to all the sideband signals. The reference image tosupport CAPI 2.0 will support specific pins for all three types of the FPGAs. The physical designof a CAPI card will also support these specific pins. This will reduce the effort to close placement,routing, and timing. Extended designs utilizing more I/O signals shall keep the predifined pins andadd additional signals to support the new application on top of the pins used in the reference image.

For further details please look at Section A.1, “FPGA Implementation Guidelines” [25]

8.1. Network or Other InterfacesOver time the function set may be extended by contributions from OpenPOWER members. Newfunctions using external hardware connected to the FPGA will have dedicated pins added to thisdocument in Section A.2, “Future Updates” [31].

8.1.1. I2C Base AddressBase Address of the network modules or other interfaces on the reference card can be found in theVPD. See Section A.3, “VPD Structure” [32] in the appendix.



Non-standard Track

9. Mechanism to unlock a 'Bricked'CardIn the case a problem occured during programming of the FPGA the whole card can "disappear"from the system and not be accessible anymore. If this occurs it is assumed that the flash content isincomplete or corrupted and preventing the accelerator from booting. There are ways to recover fromthis state. The developer will need to physically go to the system where the 'bricked' card is installedand use the programming tools to reconfigure the FPGA to a point that it is available to the systemagain.

A JTAG dongle or USB cable and a laptop with the XILINX tools are required to reconfigure theFPGA. This is no problem in a Lab environment, but if the card is installed in a datacenter it can bemuch more difficult to gain access to the system again. Ideally, the FPGA card should be recoveredremotely. This approach is often refered to as "anti-bricking."

It is important that the remote recovery procedure be initiated by the same commands independentof the internal mechanism or the implemented devices.

9.1. InterfaceThe only interface besides CAPI that is permanently connected to the device that needs recoveryis the SMBus. The SMBus also works under the standby condition so that reconfiguring of theFPGA can be accomplished before the OS boots the adapter (activate special circuits at a definedaddress). This reconfigurtion might only be a step to prepare the accelerators internal devices sothat the FPGA may be reprogrammed when the card is powered up.

Because the SMBus is very slow compare to CAPI / PCI Express the SMBus interface might not beused to recover the whole application image. A better way to recover would be to program a smallspecial image that will enable flashing of the application when the high speed interface is activeagain and will be available to the systems device tree.

The details of the recovery circuit are up to the designer or manufacturer but the initial procedureto initiate a recovery of a bricked card is always the same and described in Section 9.2, “CommonProcedure” [24].

The Reference Card Design provides two simple implementation approaches shown before. TheSMBus attached devices are available at standby



Non-standard Track

Figure 9.1. Bricked Card - Base

Figure 9.2. Bricked Card - Extended

Note

In Figure 9.1, “Bricked Card - Base” [24] and Figure 9.2, “Bricked Card - Extend-ed” [24] the path in blue shows the recovery path

9.2. Common ProcedureThe procedure to start recovery should be an IPMI command that can be initiated from the highlevel software or via a terminal session from the BMC or system controller at 'Standby'. This isthe preparation cycle and at this time the address bits for the recovery image will be active. Theprocedure will enable a PCIe/CAPI simple endpoint device when the OS boots again. Once theFPGA card is accessible via PCIe, the FPGA can be updated with the user application code.

The detailed mechanism is not part of this documentation and the sequence of events and steps areup to the card designer.



Non-standard Track

Appendix A. Design GuidelinesThis appendix will cover design guidelines for a successful card design.

A.1. FPGA Implementation GuidelinesThis section will provide more detailed information on how to implement the FPGA on the PCIe lowprofile card.

This paragraph is not complete and will be worked soon.

A.1.1. KU15P

A.1.1.1. Pin-out

Figure A.1. KU15P Pin-out



Non-standard Track

Figure A.2. From PCIe edge connector to FPGA

A.1.2. VU3P

Figure A.3. VU3P Pin-out



Non-standard Track




Non-standard Track

A.1.3. VU33P

Figure A.5. VU33P Pin-out



Non-standard Track




Non-standard Track

A.1.4. ZUG19EG

Figure A.7. ZU19EG Pin-out



Non-standard Track


A.2. Future Updates------------ Chapter will be updated by OpenPower Members to implement bigger FPGAs--------

This section is included in the reference document to provide a space for future implementationguidelines / details to be described. All Accelerator Workgroup members are encouraged to addideas and solutions. Please use the following structure for future updates:

1. Short description of what is added2. Name and Link to the section in this document (added to the end)3. For additional documents add a link to the Accelerator Workgroup space on OpenPOWER

Foundation members website (Causeway)



Non-standard Track

A.3. VPD StructureFigure A.9. CAPI Accelerator Reference Card VPD - Structure

The VPD Reference Spreadsheet may be found at https://openpowerfoundation.org/wg/aclwg/document/1576 .

A.4. Basic IPMI commandsTable A.1. IMPI Commands

IPMI Command Function

sdr list > Lists status of all sensors a

fru print Prints the FRU information from VPD

card recover Bricked card recovery

card status Checks the card status

aSensors

• Power subsystem:

– return voltages rails on card– return currents

• Thermal subsystem:

– return thermal sensor readings

• Clock subsystem:

– Return frequency settings

A.5. Reference Card - Hardware DetailsThis section provides an example of how to implement the function blocks in particular. There areseveral other approaches on how to implement such an application but I have chosen to describeone specific design implementation.

https://openpowerfoundation.org/wg/aclwg/document/1576https://openpowerfoundation.org/wg/aclwg/document/1576



Non-standard Track

The following Figure A.10, “Block Diagram” [33] depicts the function blocks of the CAPI -Reference Card. The color-code will break down the blocks into design topics that can be found onthe web using the associated table with links to the topic.

Figure A.10. Block Diagram

Design Link table provides links to the implementation details:

Function Block Document (link)

PCIe Connector> PCI-SIG Card Electro-Mechanical Spec .

Power Distribution PCI-SIG Card Electro-Mechanical Spec .

SMBus Card Management CAPI 2.0 Reference Card Document .

System Sensors (example) CAPI 2.0 Reference Card Document .

PCI Express FPGA pinout (GTY x8) CAPI 2.0 Reference Card Document .

Power controller (example) CAPI 2.0 Reference Card Document .

Card Status LED CAPI 2.0 Reference Card Document .

Connector Headers CAPI 2.0 Reference Card Document .

FPGA XILINX - Document Navigator .

User Flash XILINX - Document Navigator .

Network XILINX - Document Navigator .

SODIMM XILINX - Document Navigator .

GTH/GTY XILINX - Document Navigator .

Clock Distribution XILINX - Document Navigator .

Power Supplies XILINX - Document Navigator .

https://pcisig.com/specifications/pciexpress/https://pcisig.com/specifications/pciexpress/https://openpowerfoundation.org/?resource_lib=capi-0-reference-card-documenthttps://openpowerfoundation.org/?resource_lib=capi-0-reference-card-documenthttps://openpowerfoundation.org/?resource_lib=capi-0-reference-card-documenthttps://openpowerfoundation.org/?resource_lib=capi-0-reference-card-documenthttps://openpowerfoundation.org/?resource_lib=capi-0-reference-card-documenthttps://openpowerfoundation.org/?resource_lib=capi-0-reference-card-documenthttps://www.xilinx.com/support/documentation-navigation/overview.htmlhttps://www.xilinx.com/support/documentation-navigation/overview.htmlhttps://www.xilinx.com/support/documentation-navigation/overview.htmlhttps://www.xilinx.com/support/documentation-navigation/overview.htmlhttps://www.xilinx.com/support/documentation-navigation/overview.htmlhttps://www.xilinx.com/support/documentation-navigation/overview.htmlhttps://www.xilinx.com/support/documentation-navigation/overview.html



Non-standard Track

A.6. Schematic DetatilsThe following pages show one possible implementation solution to fulfil the defined requirements ofthe CAPI 2.0 Reference Card.

The documentation starts with the CAPI Card control and status infrastructure since this partis essential and one part of the Core Components. Further schematic pages will show corecomponents as part of an example implementation based on the defined FPGA and describedfunction. OpenPOWER Foundation Accelerator Workgroup members might provide future funcitonblocks and its associated schematics. This will make this chapter open for participation of activeworkgroup members on open issues and new ideas.

As long as the Appendix is not officially released by the AWG the information will be kept within thework-group members only. An official release will open it to the public..

Figure A.11. High Level Description of the SMBus Card Management:



Non-standard Track

Figure A.12. SMBus Card Management:



Non-standard Track

Appendix B. OpenPOWER FoundationoverviewThe OpenPOWER Foundation was founded in 2013 as an open technical membership organiza-tion that will enable data centers to rethink their approach to technology. Member companies areenabled to customize POWER CPU processors and system platforms for optimization and innova-tion for their business needs. These innovations include custom systems for large or warehousescale data centers, workload acceleration through GPU, FPGA or advanced I/O, platform optimiza-tion for SW appliances, or advanced hardware technology exploitation. OpenPOWER members areactively pursing all of these innovations and more and welcome all parties to join in moving the stateof the art of OpenPOWER systems design forward.

To learn more about the OpenPOWER Foundation, visit the organization website atopenpowerfoundation.org.

B.1. Foundation documentationKey foundation documents include:

• Bylaws of OpenPOWER Foundation

• OpenPOWER Foundation Intellectual Property Rights (IPR) Policy

• OpenPOWER Foundation Membership Agreement

• OpenPOWER Anti-Trust Guidelines

More information about the foundation governance can be found at openpowerfoundation.org/about-us/governance.

B.2. Technical resourcesDevelopment resouces fall into the following general categories:

• Foundation work groups

• Remote development environments (VMs)

• Development systems

• Technical specifications

• Software

• Developer tools

The complete list of technical resources are maintained on the foundation Technical Resources webpage.

http://openpowerfoundation.orghttps://members.openpowerfoundation.org/document/dl/635https://members.openpowerfoundation.org/document/dl/596https://members.openpowerfoundation.org/document/dl/595https://members.openpowerfoundation.org/document/dl/498http://openpowerfoundation.org/about-us/governance/http://openpowerfoundation.org/about-us/governance/http://openpowerfoundation.org/technical/working-groups/http://openpowerfoundation.org/technical/technical-resources/development-environmentvm/http://openpowerfoundation.org/technical/technical-resources/development-systems/http://openpowerfoundation.org/technical/technical-resources/technical-specifications/http://openpowerfoundation.org/technical/technical-resources/software/http://openpowerfoundation.org/technical/technical-resources/openpower-developer-tools/http://openpowerfoundation.org/technical/



Non-standard Track

B.3. Contact the foundationTo learn more about the OpenPOWER Foundation, please use the following contact points:

• General information --

• Membership --

• Technical Work Groups and projects --

• Events and other activities --

• Press/Analysts --

More contact information can be found at openpowerfoundation.org/get-involved/contact-us.

http://openpowerfoundation.org/get-involved/contact-us/

CAPI Reference CardTable of ContentsPreface1. Conventions2. Document change history

1. Introduction2. Reference Documents2.1. PCI-SIG® Card Electromechanical Specification (CEM)2.2. PCI-SIG Base Specification2.3. OpenPower Compliance2.4. OpenPower Coherent Accelerator Interface Architecture2.5. OpenPower Ready™2.6. Field Replaceable Unit (FRU) Service Interface (FSI)2.7. IPMI and MCTP

3. FPGA and Core Components3.1. XILINX® part definition3.1.1. Small FPGA3.1.2. Medium FPGA3.1.3. Large FPGA

3.2. PSL Physical Pins3.2.1. PCIE x8 basic pin out3.2.2. PCIE x16 basic pin out3.2.3. I2C / SMBus at PCIe edge Connector and on Card

3.3. Minimum of Target Devices (Base Version)3.4. Extended Implementation3.5. IPMI support3.6. Card Control Subsystem Blocks3.7. Core - Function Definition3.8. On Board additional I2C3.8.1. I/O Control and Status3.8.2. Environmental

4. Infrastructure/Pervasive4.1. VPD4.1.1. Minimal Content4.1.2. I2C Base Addresses in VPD

4.2. Configuration Flash4.2.1. Default User Image Location and Fallback location for card recovery

4.3. Card Clock Generator or Clock Buffer4.3.1. Status Register (read only)

4.4. DIP Switches4.5. Programming Interface4.5.1. On Board - Module (optional)4.5.2. JTAG Pin assignment / Footprint

5. Power Supply5.1. I2C Control Interface5.1.1. I2C Base Address

5.2. Current Monitor5.3. Power Consumption5.3.1. Current per device of the accelerator card

6. Thermal6.1. Passive Heat-sink - No Fan

7. Human Factors7.1. LED Color Codes7.2. Programmer Access

8. Pre-Defined I/O8.1. Network or Other Interfaces8.1.1. I2C Base Address

9. Mechanism to unlock a 'Bricked' Card9.1. Interface9.2. Common Procedure

Appendix A. Design GuidelinesA.1. FPGA Implementation GuidelinesA.1.1. KU15PA.1.1.1. Pin-out

A.1.2. VU3PA.1.3. VU33PA.1.4. ZUG19EG

A.2. Future UpdatesA.3. VPD StructureA.4. Basic IPMI commandsA.5. Reference Card - Hardware DetailsA.6. Schematic Detatils

Appendix B. OpenPOWER Foundation overviewB.1. Foundation documentationB.2. Technical resourcesB.3. Contact the foundation