open source hpc: an open source supercomputing platform

1
Open Source HPC: An Open Source Supercomputing Platform The idea of SoCs is starting to be applied to HPC, building chips from IP building blocks. Motivation Goal Tiled Architecture Each tile will contain the open source RISC-V based Rocket core with a custom Personal Memory Engine (PME), connected to an OpenSoC Fabric network. Hardware Architecture Programming Model The Open Source HPC programming model will rely upon a MIMD approach where a single host (x86_64) application will launch multiple RISC-V threads. Each RISC-V thread may have additional, extended instruction support for application-specific computing workloads. Software Infrastructure OpenHPC System Architect FPGA Prototyping To be able to test our proposed design, we’re using COTS FPGA systems with stacked memory to prototype the design. Further Reading and Acknowledgements For further reading, please visit our website: http://www.opensuco.community We’d like to acknowledge the Laboratory for Physical Science and the Department of Energy for helping fund this work. Using pre-verified open source IP, we want to create an open source silicon chip complete with a full software toolchain, including specialized instructions. § Lawrence Berkeley National Lab, Texas Tech University Farzad Fatollahi-Fard § , David Donofrio § , John Shalf § {ffard, ddonofrio, jshalf}@lbl.gov John Leidel , Xi Wang , Yong Chen {john.leidel, xi.wang, yong.chen}@ttu.edu Memory DRAM DRAM Mem Control Mem Control PCIe FLASH Control IB or GigE IB or GigE I/O Verilog FPGA ASIC Hardware Compilation Software Compilation SystemC Simulation C++ Simulation Scala Chisel Tile RISC-V Rocket Core L1 I$ L1 D$ Personal Memory Engine (PME) ROCCIO MemIO MemIO MemIO TileLink to AXI TileLink TileLink Router AXI Message Passing Interface (AXI) North East South West Personal Memory Engine Outgoing Messages Incoming Messages Scratch Pad Memory Scatter-Gather Engine To Rocket Core Messages (AXI) Messages (AXI) MemIO To Network To L1 D$ Instruction Set Extensions Black Box Modules RISC-V ISA & Extensions Chisel Modules LLVM Compiler Backend Full-Chip Verilog Chisel SoC C++ Cycle- Based Simulator Module TestBenches VLSI Tools FPGA Tools More and more projects are being made open- source, but they’re still based on closed-source IP. New companies are being founded based on open source. Open Source IP Full Compiler and Silicon Design Personal Memory Engine The PME sits on the co-processor interface to the main Rocket core, providing a scratch pad, a basic scatter-gather engine, and message passing queues. Rocket Tile Router Rocket Tile Router Rocket Tile Router Rocket Tile Router Off-Chip SerDes Router Rocket Tile Router Rocket Tile Router Rocket Tile Router Rocket Tile Router Mesh Network The tiles will be arranged in a mesh networking using the open source OpenSoC Fabric, with a node for a connection to main memory and off-chip communication. Compiler & Tool Chain The Open Source HPC compiler and tool chain will be based upon the LLVM compiler and tool infrastructure. The initial support will include C, C++ and ObjectiveC source compilation using the RISC- V LLVM target. In addition to basic RISC-V support, the OpenHPC System Architect environment will also provide auto-generated instruction, register and inline intrinsic extensions for application-specific architecture support. Execution Environment The Open Source HPC execution environment will provide essential execution and debugging capabilities using an abstract set of interfaces. The OHPCRUN tool will provide users the ability to initiate application workloads on the device as well as provide a messaging interface for client debuggers to interact with an active workload. The LLVM LLDB client debugger will be utilized in order to provide a facility for breakpoints, single stepping, core file analysis, reading/modifying register state and reading/modifying memory state on an active OpenHPC device. FPGA Top HMC Controller Crossbars provide connection to memory and between chips Processor Arrays HMC PLX PCIe to Host RISC-V Array FPGA 2 RISC-V Array FPGA 1 RISC-V Array FPGA 3 RISC-V Array FPGA 4 RISC-V Array FPGA 5 Global HMC AC-510 Module AC-510 Module AC-510 Module AC-510 Module Local HMC Local HMC Local HMC Local HMC AC-510 Module We are targeting multiple FPGA boards to maximize prototyping capabilities. PLX PCIe to Host RISC-V Array FPGA 2 RISC-V Array FPGA 1 RISC-V Array FPGA 3 RISC-V Array FPGA 4 Global HMC Local DDR Local DDR Local DDR Host x86_64 x86_64 Board Support Package OpenHPC Driver OpenHPC Launcher OpenHPC RISC-V ........ RISC-V RISC-V OpenHPC Custom Intstructons OpenHPC Custom Intstructons OpenHPC Custom Intstructons Clang Frontend C, C++, ObjC LLVM Optimizer OpenHPC Pragmas & Library Calls LLVM Codegen OpenHPC/RISC-V Assembly .text OpenHPC/RISC-V object.o LLD Linker app.exe OpenHPC Runtime Libraries Auto-generated instruction and intrinsic extensions $> ohpcrun --debug /path/to/my.exe ARGS ohpcrun [ohpcrun] Hardware Interaction Thread Board Support Package OpenHPC Hardware Interface [ohpcrun] Debugging Interaction Thread Hardware Commands PME Debug Module Command Interface Context Save/Restore Memory OpenSoC Fabric Interface External Debugger [LLDB] libohpcdebug.a Users

Upload: others

Post on 31-Jan-2022

17 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Open Source HPC: An Open Source Supercomputing Platform

Open Source HPC:An Open Source Supercomputing Platform

The idea of SoCs is starting to be applied to HPC, building chips from IP building blocks.

Motivation

Goal

Tiled ArchitectureEach tile will contain the open source RISC-V based Rocket core with a custom Personal Memory Engine (PME), connected to an OpenSoC Fabric network.

Hardware Architecture

Programming ModelThe Open Source HPC programming model will rely upon a MIMD approach where a single host (x86_64) application will launch multiple RISC-V threads. Each RISC-V thread may have additional, extended instruction support for application-specific computing workloads.

Software Infrastructure OpenHPC System Architect

FPGA Prototyping

To be able to test our proposed design, we’re using COTS FPGA systems with stacked memory to prototype the design.

Further Readingand Acknowledgements

For further reading, please visit our website:

http://www.opensuco.community

We’d like to acknowledge the Laboratory for Physical Science and the Department of Energy for helping fund this work.

Using pre-verified open source IP, we want to create an open source silicon chip complete with a full software toolchain, including specialized instructions.

§Lawrence Berkeley National Lab, ‡ Texas Tech University

Farzad Fatollahi-Fard§, David Donofrio§, John Shalf§{ffard, ddonofrio, jshalf}@lbl.gov

John Leidel ‡, Xi Wang ‡, Yong Chen ‡

{john.leidel, xi.wang, yong.chen}@ttu.edu

MemoryDRAM

DRAM

MemControlMem

Control

PCIe

FLASHControl

IB or GigE

IB or GigE

I/O

Verilog

FPGA ASIC

Hardware Compilation

Software Compilation

SystemC Simulation

C++ Simulation

Scala

Chisel

Tile

RISC-VRocket Core

L1 I$ L1 D$

Personal Memory Engine(PME)

ROCCIO

MemIO

Mem

IO

Mem

IO

TileLink to AXI

TileLink

TileLink

Router

AXI

Message Passing Interface (AXI)

North

East

South

West

Personal Memory Engine

Outgoing Messages

Incoming Messages

ScratchPad

Memory

Scatter-Gather Engine

To Rocket Core

Messages(AXI)

Messages(AXI)

MemIO

To Network

ToL1 D$

Instruction Set Extensions Black Box

Modules

RISC-V ISA & Extensions

Chisel Modules

LLVM Compiler Backend

Full-Chip Verilog

Chisel SoC

C++ Cycle-Based

Simulator

ModuleTestBenches

VLSI ToolsFPGA Tools

More and more projects are being made open-source, but they’re still based on closed-source IP.

New companies are being founded based on open source.

Open Source IP

Full Compiler andSilicon

Design

Personal Memory EngineThe PME sits on the co-processor interface to the main Rocket core, providing a scratch pad, a basic scatter-gather engine, and message passing queues.

RocketTile

Router

RocketTile

Router

RocketTile

Router

RocketTile

Router

Off-ChipSerDes

Router

RocketTile

Router

RocketTile

Router

RocketTile

Router

RocketTile

Router

Mesh NetworkThe tiles will be arranged in a mesh networking using the open source OpenSoC Fabric, with a node for a connection to main memory and off-chip communication.

Compiler & Tool ChainThe Open Source HPC compiler and tool chain will be based upon the LLVM compiler and tool infrastructure. The initial support will include C, C++ and ObjectiveC source compilation using the RISC-V LLVM target. In addition to basic RISC-V support, the OpenHPC System Architect environment will also provide auto-generated instruction, register and inline intrinsic extensions for application-specific architecture support.

Execution EnvironmentThe Open Source HPC execution environment will provide essential execution and debugging capabilities using an abstract set of interfaces. The OHPCRUN tool will provide users the ability to initiate application workloads on the device as well as provide a messaging interface for client debuggers to interact with an active workload. The LLVM LLDB client debugger will be utilized in order to provide a facility for breakpoints, single stepping, core file analysis, reading/modifying register state and reading/modifying memory state on an active OpenHPC device.

FPGA Top

HMC Controller

Crossbars provide connection to memory

and between chips

Processor Arrays

HMC

PLXPCIe to Host

RISC-V Array

FPGA 2

RISC-V Array

FPGA 1

RISC-V Array

FPGA 3

RISC-V Array

FPGA 4

RISC-V Array

FPGA 5

GlobalHMC

AC-510 Module AC-510 Module AC-510 Module AC-510 Module

LocalHMC

LocalHMC

LocalHMC

LocalHMC

AC-510 Module

We are targeting multiple FPGA boards to maximize prototyping capabilities.

PLXPCIe to Host

RISC-V Array

FPGA 2

RISC-V Array

FPGA 1

RISC-V Array

FPGA 3

RISC-V Array

FPGA 4

GlobalHMC

LocalDDR

LocalDDR

LocalDDR

Host

x86_64 x86_64

Board Support Package

OpenHPC Driver

OpenHPCLauncher

OpenHPC

RISC-V........

RISC-V RISC-VOpenHPCCustom

Intstructons

OpenHPCCustom

Intstructons

OpenHPCCustom

Intstructons

Clang Frontend

C, C++, ObjC

LLVM OptimizerOpenHPC Pragmas &

Library Calls

LLVM Codegen

OpenHPC/RISC-VAssembly

.text

OpenHPC/RISC-Vobject.o

LLD Linker

app.exe

OpenHPC Runtime Libraries

Auto-generated instruction and

intrinsic extensions

$> ohpcrun --debug /path/to/my.exe ARGS

ohpcrun

[ohpcrun]Hardware Interaction

Thread

Board Support Package

OpenHPC HardwareInterface

[ohpcrun]Debugging Interaction

Thread

Hardware Commands

PME Debug Module

Command Interface

Context Save/RestoreMemory

OpenSoC Fabric Interface

External Debugger[LLDB]

libohpcdebug.a

Users