Shiyu Liu, Balaji Prabhakar, and Mendel RosenblumStanford University
Overview of Smart NICs I:CPU cores and “Deliberate Control”
SoC based Smart NICs
• System on chip: ARM processor + NIC
Mellanox Bluefield Broadcom PS225
http://www.mellanox.com/related-docs/prod_adapter_cards/PB_BlueField_Smart_NIC.pdfhttps://www.broadcom.com/products/ethernet-connectivity/smartnic/ps225#overview
X86 Host
Smart NIC
Ethernet controller
ARM
Eth 0 Eth 1
PCIe
OVS
SoC based Smart NICs
• System on chip: ARM processor + NIC
• Full OS, any software– OVS, DL inference,
VM/container hypervisor, etc.
• Highest flexibility & easiest programmability
• 8-core 64-bit ARMv8 A72• 8 ~ 16 GB RAM• 16 GB Flash Memory
• Two 25Gbps Ethernet interfaces• Transport Offloads: LSO/LRO/TSO, RoCE, … • Hardware-based I/O Virtualization• Cryptography Acceleration• Security Support
Packet flow in SoC based smart NICs
Mellanox Bluefield as an example:• Delay in SW OVS = ~ 15us when load is
low (red line)• OVS forwarding rules can be offloaded
to HW: delay < 1us (yellow line)
Smart NIC board
Eth 0eSwitch
Eth 1eSwitch
ARM
PCIe RC
vNIC vNIC
PCIe EP
vNIC vNIC
PCIe EP
Host CPU
PCIe RC
OVS
X86 Host
Smart NIC
Ethernet controller
ARM
Eth 0 Eth 1
PCIe
OVS
Huygens on Smart NICs• Huygens: a software-based
clock sync system, achieving accuracy of 10s ns
• NIC sends probes, takes timestamps, processes data locally
• A coordinator collects local results to do final calculation and make the decision of clock correction
Huygens on Smart NICs: Evaluation• Offload both probers and the coordinator into Smart NICs.
– All-to-all probing. 250 probe pairs / sec.
• Network load: all-to-all iperf on hosts. 40% load.• Clock offsets after sync (1 hour exp):
• Overhead on ARM side:
• Huygens on Smart NICs helps controlling Tx time of pktsprecisely: TDMA is possible now!
50% 99% maxSW timestamps 382 ns 2488 ns 3979 ns
HW timestamps* 14 ns 57 ns 101 ns
CPU Memory Net bandwidthProber 5.4% of one core 13.1 MB 0.89 Mbps
Coordinator 3.6% of one core 150.5 MB 0.0061 Mbps
Mellanox SN2100 switch
Host 1Bluefield
Host 2
Bluefield
Host 3PS225
Host 4
PS225
*: We have some issues on HW timestamping on Broadcom PS225 for now. This result is for Mellanox Bluefield only.
Future work• Next step:– SmartNIC-based transaction tracer and analyzer based on SIMON