Building a Data Infrastructure(A WAN Provider’s Perspective)

1

Chin GuokESnet Planning & Architecture Group LeadLawrence Berkeley National Laboratory

Data on the NetworkComputing and Software RoundtableVirtual MeetingApr 6, 2021

What Do You Need for a Data (Network) Infrastructure?

• Capacity - Planning and deploying of the right amount of capacity in the right place, and at the right time

• Capability - Building services that will be integral to a Data Infrastructure that is beyond just networking

• Coordination - Enabling the orchestration of resources to build end-to-end solutions whose whole is greater than the sum of its parts

2

ESnet Data Infrastructure Activities

• Capacity– ESnet6 CY2022 Planned Capacity

• Capability– In-Network Caching Pilot– DTN-as-a-Service efforts– Edge Computing

• Coordination– SENSE: SDN for End-to-end Networking Science at the Exascale– ASCR Integrated Research Infrastructure Task Force

3

An Exabyte Network Today

4

~55% growth year-on-years

1025 PB/year as of FY2020

Impact of pandemic

ESnet6 Planned Backbone Capacity for FY2022

5

ESnet Data Infrastructure Activities

• Capacity– ESnet6 CY2022 Planned Capacity

• Capability– In-Network Caching Pilot– DTN-as-a-Service efforts– Edge Computing

• Coordination– SENSE: SDN for End-to-end Networking Science at the Exascale– ASCR Integrated Research Infrastructure Task Force

6

In-Network Caching Pilot Experiment

• In-network temporary data cache for data sharing

• Collaboration with UCSD, US CMS, Open Science Grid (OSG)– ESnet cache node as a part of SoCal Petabyte scale cache for US CMS

– Petabyte scale cache is deployed/operated by UCSD and Caltech

– ESnet: Provide a storage host to US CMS

– UCSD: Operation support for the ESnet node

• Goals

– Study how network cache storage helps network traffic performance and the overall application performance

– Accumulate experience on how the US DOE scientific experiments and simulations share data among their users

7 Sim et al.

In-Network Caching Pilot Data Movements

8 Sim et al.

Data Transfers from Source to Xcache - Cache Misses

2020-05 2020-112020-06 2020-07 2020-08 2020-09 2020-100TB

2TB

4TB

2020-05 2020-112020-06 2020-07 2020-08 2020-09 2020-100TB

2TB

4TB

6TB

8TB

10TB

12TB

Shared Data Access from Xcache to Applications - Cache Hits

Total Data Transfer = 168.083TB

Total Shared Data Access= 322.748TB

Network demand is reduced by a factor of ~3

In-Network Caching Pilot File Reuse Rates

9 Sim et al.

• Data file reuse rate = (sum of reuses) / (total number of unique files)– Sum of reuses = all shared data access counts of the day (cache hits)– Total number of unique files = number of unique files for the day

In-Network Caching Pilot Summary

• 1,286,748 accesses from May 2020 to Oct 2020– Total 490.831 TB of client data access (first time reads and repeated reads)

– Transferred/cached 168.08 TB (from remote sites to cache)

– Saved 322.748 TB of network traffic volume (repeated reads only)

– Network demand reduced by a factor of ~3

• Future work

– Deployment of 2 additional caches in summer 2021

– Cache miss rate study

– Cache utilization study

10 Sim et al.

DTN-as-a-Service

• Exploring an ESnet-managed data movement service

– Support a pool of transfer software images that “just work” across the ESnet footprint

– Reduce reliance on varying levels of (site) network/system expertise for DTN deployments

• Investigate opportunities to expand data movement software availability and to support more flexible service deployments

• A potential candidate for supporting the ESnet6 smart services edge

11 Kissel et al.

1. Data mover software in containers

2. Network and storage performance optimization

3. Configuration and tuning flexibility

4. Lightweight service orchestration

12

DTN-as-a-Service Prototype Service

Kissel et al.

DTN-as-a-Service Use Cases and Ongoing Engagements

• Transitioning the ESnet 10G/40G testing DTNs to DTNaaS

• Deployed in the ESnet 100G Testbed to manage experimenter images

• Evaluation of data mover software stacks supported by DTNaaS– For example - Zettar/zx1

• Investigating a deployment model to support In-Network Data Caching effort

• Seeking partnerships for service trial– Using DTNaaS to accelerate an existing data workflow challenge– Outreach to JGI, FRIB, EMSL

13 Kissel et al.1 E. Kissel, C. Fang, Zettar zx Evaluation for ESnet DTNshttps://www.es.net/assets/Uploads/zettar-zx-dtn-report.pdf

Edge Computing for Real Time Workflows - Today

14 Yatish et al.

Instrument

Custom DAQ

A/D+

Trigger+

Timing

DAQ data reformatting

Pre-filtering

Stream assembly for

event/particle extraction

Compute cluster

GPUs

Fast Storage

Batch mode event/particle

extraction

N x100G

Up to 1 Tbps Today,100Tbps Next Generation

Servers + DDR3 + 100G NIC cards

The Molecular Foundry (TMF)Advanced Light Source (ALS)

Linac Coherent Light Source (LCLS)

Continuous Electron Bean Accelerator Facility (CEBAF)

Edge Computing for Real Time Workflows - Future

15 Yatish et al.

Instrument

Custom DAQ

A/D+

Trigger+

Timing

DAQ data reformatting

Pre-filtering

Stream assembly for

event/particle extraction

Compute cluster

GPUs

Fast Storage

Batch mode event/particle

extraction

Out of order reassembly

Local compute load balancing

N x100G

Up to 1 Tbps,100Tbps NxGen

100% link load balancing(Out of order UDP protocols)Real time data compression

FPGA HW laid balancing intocluster. Group DAQ data by

timestamp and frame forparallel data processing

Low cost scalable FPGApacket processors toreplace expensiveCPU+DRR solutions

The Molecular Foundry (TMF)Advanced Light Source (ALS)

Linac Coherent Light Source (LCLS)

Continuous Electron Bean Accelerator Facility (CEBAF)

ESnet Data Infrastructure Activities

• Capacity– ESnet6 CY2022 Planned Capacity

• Capability– In-Network Caching Pilot– DTN-as-a-Service efforts– Edge Computing

• Coordination– SENSE: SDN for End-to-end Networking Science at the Exascale– ASCR Integrated Research Infrastructure Task Force

16

SENSE: SDN for End-to-End Networked Science at the Exascale- Coordination of End-to-End Network Cyberinfrastructure

WAN

Application Workflow Agents

SDX

End Site

SDN Layer

ComputeDTNs Storage Instruments

End Site

Compute DTNsStorageInstruments

Regional

SDMZ

Regional

SDMZ

WAN

SENSE

Types of Interactions● What is possible?● What is recommended?● Requests with negotiation● Scheduling

17

● Intent Based APIs● Resource Discovery● Service Life Cycle

Monitoring/Troubleshooting● Deterministic Networking

Lehman et al.

SENSEArchitecture

18 Lehman et al.

SENSE-O SENSE-O

Application Workflow Agent (e.g., CMS)

Application Workflow Agent (e.g., ATLAS)

Application Workflow Agent(s)

Collaboration / Project Centric

Resource Administrative Domain Centric

SENSE-NRM

(NSI) Domain Controller (e.g.,

OpenNSA)

SENSE-DTN-RM

DTN

NSIDomain

SENSE-NRM

(NSI) Domain Controller (e.g.,

OpenNSA)

(NSI) Domain Controller (e.g.,

OpenNSA)

(NSI) Aggregator (e.g., Safnari)

SENSE-NRM

(NSI) Domain Controller (e.g.,

OESS)

Domain Controller (e.g.,

OSCARS)

SENSE-NRM

Domain with network using NSI under SENSE control (e.g., Internet2)

Domain with network under SENSE (native) control (e.g., ESnet)

Domain with DTN under SENSE control (e.g., Caltech, UCSD)

Federated domains using NSI under SENSE control (e.g., AutoGOLE)

Domain with network and DTN under SENSE control (e.g., CENIC, WIX, MANLAN)

SENSE-DTN-RM

DTN

Application Agent:SENSE-O = N:1

SENSE-O:SENSE-RM = N:N

O: OrchestratorRM: Resource ManagerDTN-RM: Data Transfer Node RMNRM: Network RM

SENSE ExaFEL Use Case - Superfacility Automation Prototype

19 Lehman et al.

DTNs

FFB Layer (nVRAM)

ESNET

DAQ (1 per instrument, 10 total)

Online System (1 instance per experimental hall, 2 total)

Compressor Nodes

PGP

PGP

PGP

PGP

Eth

IB

Beam Line Data

LCLSLCLS

ESNET

Exascale HPC

1 TB/s

DTNs

NERSCRouter

Burst Buffer nVRAM: Streaming Analysis

Software ETH Routers

NGF / Lustre: Offline from HDF/XTC files.

IB

Exascale HSN

HPC System with SDN & Burst Buffer Supporting

(1) file-based transfer path (2) stream-based data transfer path

Aggregate detector data, EPICS data, beamline data - selection and compression

10 GB/s - 1Tb/sEPICSData

Timing

Online Monitoring & FFB Nodes

Event builder nodes

ExaFEL Data Flow

LCLS Data Transfer Workflow SENSE-O

SENSE-NRM SENSE-DTN-RMSuperfacility

(SF) API*

*NB: Superfacility (SF) API under development by NERSC

SENSE Future Domain Science Integrations

20

• RUCIO/FTS expected to be used in multiple domain science workflows

• SENSE Service and Orchestration model designed for adaptation and customization for different infrastructures, service requirements, and workflows

Lehman et al.

ASCR Integrated Research Infrastructure Task Force

21

ALCFNERSCOLCFESnet

AL

DA

APSC

WF

AR

PB

AC

Experimental Facility

2

ALCFNERSCOLCFESnet

AL

DA

APSC

WF

AR

PB

AC

Experimental Facility

Experimental Facility

Experimental Facility

3

ALCF

AL

DA

APSC

WF

AR

PB

AC

ESnet

AL

DA

APSC

WF

AR

PB

AC

OLCF

AL

DA

APSC

WF

AR

PB

AC

NERSC

AL

DA

APSC

WF

AR

PB

AC

Experimental Facility

1

AL Allocations

AccountsAC

DA

AP

SC

WF

AR

PB

Data

Workflows

Applications

Scheduling

Archiving

Publication

Toward a Seamless Integration of Computing, Experimental, and

Observational Science Facilities: A Blueprint to Accelerate Discovery*

1. Today, an experimental facility must arrange separate bespoke interactions with individual HPC/HPN facilities.

2. A future paradigm with common interfaces could simplify integration of an experimental facility with multiple HPC/HPN facilities.

3. In turn, these common interfaces could support expansion and integration across multiple experimental facilities and HPC/HPN facilities

Brown et al.*NB: Whitepaper to be released soon.

Questions...

22