Shiyu Liu, Balaji Prabhakar, and Mendel RosenblumStanford University

Overview of Smart NICs I:CPU cores and “Deliberate Control”

SoC based Smart NICs

• System on chip: ARM processor + NIC

Mellanox Bluefield Broadcom PS225

http://www.mellanox.com/related-docs/prod_adapter_cards/PB_BlueField_Smart_NIC.pdfhttps://www.broadcom.com/products/ethernet-connectivity/smartnic/ps225#overview

X86 Host

Smart NIC

Ethernet controller

ARM

Eth 0 Eth 1

PCIe

OVS

SoC based Smart NICs

• System on chip: ARM processor + NIC

• Full OS, any software– OVS, DL inference,

VM/container hypervisor, etc.

• Highest flexibility & easiest programmability

• 8-core 64-bit ARMv8 A72• 8 ~ 16 GB RAM• 16 GB Flash Memory

• Two 25Gbps Ethernet interfaces• Transport Offloads: LSO/LRO/TSO, RoCE, … • Hardware-based I/O Virtualization• Cryptography Acceleration• Security Support

Packet flow in SoC based smart NICs

Mellanox Bluefield as an example:• Delay in SW OVS = ~ 15us when load is

low (red line)• OVS forwarding rules can be offloaded

to HW: delay < 1us (yellow line)

Smart NIC board

Eth 0eSwitch

Eth 1eSwitch

ARM

PCIe RC

vNIC vNIC

PCIe EP

vNIC vNIC

PCIe EP

Host CPU

PCIe RC

OVS

X86 Host

Smart NIC

Ethernet controller

ARM

Eth 0 Eth 1

PCIe

OVS

Huygens on Smart NICs• Huygens: a software-based

clock sync system, achieving accuracy of 10s ns

• NIC sends probes, takes timestamps, processes data locally

• A coordinator collects local results to do final calculation and make the decision of clock correction

Huygens on Smart NICs: Evaluation• Offload both probers and the coordinator into Smart NICs.

– All-to-all probing. 250 probe pairs / sec.

• Network load: all-to-all iperf on hosts. 40% load.• Clock offsets after sync (1 hour exp):

• Overhead on ARM side:

• Huygens on Smart NICs helps controlling Tx time of pktsprecisely: TDMA is possible now!

50% 99% maxSW timestamps 382 ns 2488 ns 3979 ns

HW timestamps* 14 ns 57 ns 101 ns

CPU Memory Net bandwidthProber 5.4% of one core 13.1 MB 0.89 Mbps

Coordinator 3.6% of one core 150.5 MB 0.0061 Mbps

Mellanox SN2100 switch

Host 1Bluefield

Host 2

Bluefield

Host 3PS225

Host 4

PS225

*: We have some issues on HW timestamping on Broadcom PS225 for now. This result is for Mellanox Bluefield only.

Future work• Next step:– SmartNIC-based transaction tracer and analyzer based on SIMON