Open Source HPC:An Open Source Supercomputing Platform
The idea of SoCs is starting to be applied to HPC, building chips from IP building blocks.
Motivation
Goal
Tiled ArchitectureEach tile will contain the open source RISC-V based Rocket core with a custom Personal Memory Engine (PME), connected to an OpenSoC Fabric network.
Hardware Architecture
Programming ModelThe Open Source HPC programming model will rely upon a MIMD approach where a single host (x86_64) application will launch multiple RISC-V threads. Each RISC-V thread may have additional, extended instruction support for application-specific computing workloads.
Software Infrastructure OpenHPC System Architect
FPGA Prototyping
To be able to test our proposed design, we’re using COTS FPGA systems with stacked memory to prototype the design.
Further Readingand Acknowledgements
For further reading, please visit our website:
http://www.opensuco.community
We’d like to acknowledge the Laboratory for Physical Science and the Department of Energy for helping fund this work.
Using pre-verified open source IP, we want to create an open source silicon chip complete with a full software toolchain, including specialized instructions.
§Lawrence Berkeley National Lab, ‡ Texas Tech University
Farzad Fatollahi-Fard§, David Donofrio§, John Shalf§{ffard, ddonofrio, jshalf}@lbl.gov
John Leidel ‡, Xi Wang ‡, Yong Chen ‡
{john.leidel, xi.wang, yong.chen}@ttu.edu
MemoryDRAM
DRAM
MemControlMem
Control
PCIe
FLASHControl
IB or GigE
IB or GigE
I/O
Verilog
FPGA ASIC
Hardware Compilation
Software Compilation
SystemC Simulation
C++ Simulation
Scala
Chisel
Tile
RISC-VRocket Core
L1 I$ L1 D$
Personal Memory Engine(PME)
ROCCIO
MemIO
Mem
IO
Mem
IO
TileLink to AXI
TileLink
TileLink
Router
AXI
Message Passing Interface (AXI)
North
East
South
West
Personal Memory Engine
Outgoing Messages
Incoming Messages
ScratchPad
Memory
Scatter-Gather Engine
To Rocket Core
Messages(AXI)
Messages(AXI)
MemIO
To Network
ToL1 D$
Instruction Set Extensions Black Box
Modules
RISC-V ISA & Extensions
Chisel Modules
LLVM Compiler Backend
Full-Chip Verilog
Chisel SoC
C++ Cycle-Based
Simulator
ModuleTestBenches
VLSI ToolsFPGA Tools
More and more projects are being made open-source, but they’re still based on closed-source IP.
New companies are being founded based on open source.
Open Source IP
Full Compiler andSilicon
Design
Personal Memory EngineThe PME sits on the co-processor interface to the main Rocket core, providing a scratch pad, a basic scatter-gather engine, and message passing queues.
RocketTile
Router
RocketTile
Router
RocketTile
Router
RocketTile
Router
Off-ChipSerDes
Router
RocketTile
Router
RocketTile
Router
RocketTile
Router
RocketTile
Router
Mesh NetworkThe tiles will be arranged in a mesh networking using the open source OpenSoC Fabric, with a node for a connection to main memory and off-chip communication.
Compiler & Tool ChainThe Open Source HPC compiler and tool chain will be based upon the LLVM compiler and tool infrastructure. The initial support will include C, C++ and ObjectiveC source compilation using the RISC-V LLVM target. In addition to basic RISC-V support, the OpenHPC System Architect environment will also provide auto-generated instruction, register and inline intrinsic extensions for application-specific architecture support.
Execution EnvironmentThe Open Source HPC execution environment will provide essential execution and debugging capabilities using an abstract set of interfaces. The OHPCRUN tool will provide users the ability to initiate application workloads on the device as well as provide a messaging interface for client debuggers to interact with an active workload. The LLVM LLDB client debugger will be utilized in order to provide a facility for breakpoints, single stepping, core file analysis, reading/modifying register state and reading/modifying memory state on an active OpenHPC device.
FPGA Top
HMC Controller
Crossbars provide connection to memory
and between chips
Processor Arrays
HMC
PLXPCIe to Host
RISC-V Array
FPGA 2
RISC-V Array
FPGA 1
RISC-V Array
FPGA 3
RISC-V Array
FPGA 4
RISC-V Array
FPGA 5
GlobalHMC
AC-510 Module AC-510 Module AC-510 Module AC-510 Module
LocalHMC
LocalHMC
LocalHMC
LocalHMC
AC-510 Module
We are targeting multiple FPGA boards to maximize prototyping capabilities.
PLXPCIe to Host
RISC-V Array
FPGA 2
RISC-V Array
FPGA 1
RISC-V Array
FPGA 3
RISC-V Array
FPGA 4
GlobalHMC
LocalDDR
LocalDDR
LocalDDR
Host
x86_64 x86_64
Board Support Package
OpenHPC Driver
OpenHPCLauncher
OpenHPC
RISC-V........
RISC-V RISC-VOpenHPCCustom
Intstructons
OpenHPCCustom
Intstructons
OpenHPCCustom
Intstructons
Clang Frontend
C, C++, ObjC
LLVM OptimizerOpenHPC Pragmas &
Library Calls
LLVM Codegen
OpenHPC/RISC-VAssembly
.text
OpenHPC/RISC-Vobject.o
LLD Linker
app.exe
OpenHPC Runtime Libraries
Auto-generated instruction and
intrinsic extensions
$> ohpcrun --debug /path/to/my.exe ARGS
ohpcrun
[ohpcrun]Hardware Interaction
Thread
Board Support Package
OpenHPC HardwareInterface
[ohpcrun]Debugging Interaction
Thread
Hardware Commands
PME Debug Module
Command Interface
Context Save/RestoreMemory
OpenSoC Fabric Interface
External Debugger[LLDB]
libohpcdebug.a
Users