infiniband fpga
DESCRIPTION
InfiniBand FPGA. Iowa State University Senior Design May 09-04. Matthew Wall Rachel Ayoroa Xiang Li Ryan Schwarzkopf Tim Prince Alex Burds Adviser: Dr. Morris Chang Client: Troy Benjegerdes. What is InfiniBand. Switched fabric communication link 2.5 Gbit /s signaling - PowerPoint PPT PresentationTRANSCRIPT
InfiniBand FPGAIowa State University Senior Design
May 09-04
Matthew WallRachel AyoroaXiang LiRyan SchwarzkopfTim PrinceAlex Burds
Adviser: Dr. Morris ChangClient: Troy Benjegerdes
What is InfiniBand
Protocol Stack› Physical Layer› Link Layer› Network Layer› Transport Layer
Switched fabric communication link
2.5 Gbit/s signaling Primarily used in
supercomputers Specification maintained
by the InfiniBand Trade Association
IP Cores
Reusable hardware cores accelerate development
E.g. DSP, communication controllers, cryptographers, memory, processors, Ethernet MAC
Opencores.org
Problem/Need
Currently have to use proprietary hardware interface (e.g. Mellanox)
Can’t perform research on internals Need a open-source hardware
implementation
Design Concept Hardware core in
VHDL Compatible with
InfiniBand specifications
Implementation on an FPGA or in simulator
Xilinx FPGA Chip
InfiniBand HDL Implementation
Physical Layer
Xilinx RocketIO
Link Layer
Fabric
Operating Environment
Proprietary Environments Xilinx ISE (Development Environment) ModelSim (Simulator) Virtex-5 Development Board
User Interface
No UI in conventional terms VHDL Synthesizer VHDL Simulator FPGA pinout
Functional Requirements
FR01: The system shall provide InfiniBand compliant input/output signals.
FR02: The system shall be able to transmit and receive packets according to InfiniBand specifications.
FR03: The system shall be able to handle errors according to InfiniBand specifications.
Nonfunctional Requirements
NFR01: The system shall be written in VHDL.
NFR02: The system shall be distributed under an open source license.
NFR03: The system should use only standard VHDL libraries.
NFR04: The system should be compatible with open source VHDL simulators.
Market Research
InfiniBand Trade Association (Cisco, IBM, Intel, etc.) maintains specification
No open-source IP core competitors Specification is freely available/open
Deliverables
CD containing:› Source Code (VHDL)› Compilation Instructions› User’s Manual
Submit to OpenCores.org
Project Plan
Work BreakdownInfiniBand Project
Planning Documentation Design Implementation Testing
Scheduling
Resources
Risks
Bound Report(s)
User’s Guide
Compilation Instructions
Switch Controller
Link Layer
Physical Layer
Device Integration
Requirements
Top-Level Design
Unit Testing
Layer Integration
Testing
Device Integration
Layer Design
Layer Component
Design
Resources
Team Members (Alex Burds, Tim Prince, Matt Wall, Ryan Schwarzkopf, Xiang Li)
Ryan Schwarzkopf (Communications Coordinator)
Matt Wall (Project Leader) Jason Boyd (Lab Coordinator) Dr. Morris Chang (Project Advisor) Troy Benjegerdes (Client) Lab with Virtex-5 board and compatible
synthesizer (Coover 2041)
Schedule Overview
Implementation begins – 1/12/09 Layers completed – 4/3/09 Layers tested – 4/22/09 System integration completed –
4/26/09 Integration testing completed – 4/26/09 System testing completed – 4/29/09 Documentation completed – 5/4/09 Bound Report completed – 5/4/09
Identified Risks
Complexity of system may prevent complete hardware implementation
Complete testing may require more time than is available
May need to obtain additional test equipment
Detailed Design
System Overview Two Layers
› Link› Physical
Top-down design approach
Xilinx FPGA Chip
InfiniBand HDL Implementation
Physical Layer
Xilinx RocketIO
Link Layer
Fabric
Link Layer Handles the sending and receiving
of data across the links at the packet level
Provides addressing, flow control, and error detection
Virtual Lanes provide buffering Link State Machine responds to
changes in the link status
Components Link State Machine Packet Receiver
Machine Data Packet Check
Machine Link Packet Check
Machine Flow Control
Generator Virtual Lane
Arbitration CRC Generators
Link Layer
To Physical Layer
Link State Machine
Packet Reciever State Machine
Data/Link Packet Check Machine
Virtual Lane 0 Virtual Lane 15 (Subnet Management)
Flow Control Packet Generator
Packet Priority Selector
Virtual Lane Selector
Virtual Lanes
Buffer packets to and from Link Layer Two VLs required, VL0 and VL15 VL15 reserved for subnet management
packets
Flow Control
Prevent buffer overflow Send flow control packets periodically Updates credit information from Virtual
Lane Stop transmitting if receiver doesn’t
have enough credits
Packet Check MachinesData Packet Check Machine Variable packet headers and payloads
mean more robust error checking Error checks performed by DPCM
› VCRC and ICRC check› Link version check› Destination Local Identifier check› VL and VL15 checks› Buffer availability check› Verifies the length of the packet
Reports errors in correct precedence and discards packets with errors
Error free packets passed to Virtual Lane
Packet Check Machines
Link Packet Check Machine Only one type of packet (flow control
packet) Checks for errors within a link packet
› Verifies Link Packet CRC› Flow control opcode is flagged› Verifies correct virtual lane is selected and supported› Verifies the length of the flow control packet (6 Bytes)
Reports errors in correct precedence and discards packets with errors
Error free packets passed to Virtual Lane
CRC Generators
Two CRC Types Variant
› Entire Packet (Including Payload)› Changes between transmissions
Invariant › Data payload› Static
Physical Layer Maintains
physical link Three main
components› Link/Phy Layer› RocketIO Adapter› RocketIO Module
Physical Layer
Encoded Lanes
Link/Phy Layer
RocketIO Adapter
Physical Interconnects
Physical RocketIO Module
Link Layer Byte Streams
Link/Physical Layer Manages link
training process Maintains link
state once completed
Inserts control sequences to allow clock synchronization
Link/Physical Layer
L.T.F.S.M.
TX Buffer
RX BufferIdle
SequenceSkip
SequenceTraining
SequenceError
Status
Control Sequence Detectors
To Link Layer
To RocketIO Adapter
Link Training FSM Maintains link
state Handles errors
Polling
Configuration
Link Up
Recovery
Config OK
Config Failed
Received TS1
Recovery OK
Major Error or Link initiated
recovery
Recovery Failed
RocketIO Adapter Manages Xilinx
RocketIO operation› Clock generation
and recovery› 8b/10b Encoding
Converts Link/Phy control signals into the equivalent RocketIO signals
Testing
Simulation Testing
Simulate individual components to determine if they behave as expected
Integrate components in a layer to determine if they work together
Use ModelSim VHDL simulation
Integration Testing
Once individual layers are simulated, we will progressively integrate them together and physically test them
Phy/Phy testing › Connect the Physical layer programmed on a
single board through loopback and ensure that a data stream can be transmitted
Integration Testing Contd.
Phy+Link/Phy+Link› Add the Link layer and determine if packets
can be transmitted.› Also check for packet integrity and flow control
behavior
Test Harness
PPC440CPU Core
PLB
Test Harness Instance
Test Harness Instance
Test Harness Instance
Virtex-5 XC5VFX70T
RocketIO RocketIO RocketIO
Connector
SMA Cable (loopback)
Connection to an InfiniBand device or another test harness
Xilinx ML507 Development Board
Connector (e.g., SMA) Connector
Test Harness
Physical Layer
TX
Link LayerTX
RocketIO
RX FIFO
Physical Layer
RX
Link LayerRX
TX FIFOControl Register
Control Signals
PLB Interface
Instance of the link and physical layers
PLB control interface
Programmable interconnects
Inject and extract data into and out of the data-path
Development
Accomplishments
Physical Layer compiles Physical Layer simulation Physical Layer physical testing Link Layer compiles Link Layer simulation Performed extended testing with a
5-10 Giga-sample oscilloscope
Results
Results
Challenges Debugging the serial loop back Switching from Vertex-II to Vertex-5
board Only having 1 Vertex-5 board VHDL intense project Lost a team member
Conclusion Have a springboard for further
development Further verification and development is
needed for a robust IP core
Lessons Learned Evaluate project complexity in greater
detail Divide tasks into smaller parts Set stricter guidelines for tasks Obtain clearer end goal from client Define specific skill set required
Future Work Verification of layers against
specifications in detail Potentially port to different transceivers Create an open hardware development
platform Make more robust for usability
Demo
Questions
?