Next Generation Distributed Environments For Global

Science

Joe Mambretti, Director, (j-mambretti@northwestern.edu)

International Center for Advanced Internet Research (www.icair.org)

Northwestern University

Director, Metropolitan Research and Education Network (www.mren.org)

Director, StarLight, PI StarLight IRNC SDX,Co-PI Chameleon, PI-iGENI, PI-

OMNINet (www.startap.net/starlight)

Asia Pacific Advanced Network (APAN)

Co-Located With Supercomputing Asia

March 26-29, 2018

Singapore

Sloan Digital Sky

Survey

www.sdss.org

Globus Alliance

www.globus.org

LIGO

www.ligo.org TeraGrid

www.teragrid.org

ALMA: Atacama

Large Millimeter

Array

www.alma.nrao.edu

CAMERA

metagenomics

camera.calit2.net

Comprehensive

Large-Array

Stewardship System

www.class.noaa.gov

DØ (DZero)

www-d0.fnal.gov

ISS: International

Space Station

www.nasa.gov/statio

n

IVOA:

International

Virtual

Observatory

www.ivoa.net

BIRN: Biomedical

Informatics Research

Network

www.nbirn.net

GEON: Geosciences

Network

www.geongrid.org

ANDRILL:

Antarctic

Geological

Drilling

www.andrill.org

GLEON: Global Lake

Ecological

Observatory

Network

www.gleon.orgPacific Rim

Applications and

Grid Middleware

Assembly

www.pragma-

grid.net

CineGrid

www.cinegrid.orgCarbon Tracker

www.esrl.noaa.gov/

gmd/ccgg/carbontrack

er

XSEDE

www.xsede.org

LHCONE

www.lhcone.net

WLCG

lcg.web.cern.ch/LCG/publi

c/

OOI-CI

ci.oceanobservatories.org

OSG

www.opensciencegrid.org

SKA

www.skatelescope.o

rg

NG Digital

Sky Survey

ATLAS

Compilation By Maxine Brown

New Science Communities Using LHCONE

• Belle II Experiment, Particle Physics Experiment Designed To

Study Properties of B Mesons (Heavy Particles Containing a

Bottom Quark).

• Pierre Auger Observatory, Studying Ultra-High Energy Cosmic

Rays, the Most Energetic and Rarest of Particles In the Universe.

• In August 2017 the PAO, LIGO and Virgo Collaboration Measured a

Gravitational Wave Originating From a Binary Neutron Star Merger.

• The NOvA Experiment Is Designed To Answer Fundamental

questions in neutrino Physics.

• The XENON Dark Matter Project Is a Global Collaboration Investing

Fundamental Properties of Dark Matter, Largest Component Of The

Universe.

• ProtoNUMA/NUMA – Collaborative Research On Nutrinos

iCAIR Basic Research Topics

• Transition From Legacy Networks To Networks That Take Full Advantage

of IT Architecture and Technology

• Extremely Large Capacity (Multi-Tbps Streams)

• Specialized Network Services, Architecture and Technologies for Data

Intensive Science

• High Degrees of Communication Services Customization

• Highly Programmable Networks

• Network Facilities As Enabling Platforms for Any Type of Service

• Network Virtualization

• Tenet Networks

• Network Virtualization

• Network Programming Languages (e.g., P4) API (e.g., Jupyter)

• Disaggregation

• Orchestrators

• Highly Distributed Signaling Processes

• Network Operations Automation (Including Through AI/Machine Learning)

• SDN/SDX/SDI/OCX/SDC/SDE

Issues

• 21st Century Scientists Encounter Unprecedented Opportunities As Well

As Deeply Complex Challenges By Utilize New Knowledge Discovery

Techniques Based On Exploring Hidden Patterns In Exabytes of Globally

Distributed Data.

• Scientific Data is Growing Exponentially

• Growth Will Further Accelerate As New Major Instrumentation Is

Implemented.

• Also, New, Highly Sophisticated Analytic Techniques For Big Data Based

Investigations Are Being Created.

• The Challenges That Arise From This Global Scale Research Are Being

Addressed Through the Creation of Powerful, Innovative Globally

Distributed Computational Science Platforms, Such As the Global

Research Platform (GRP).

Global Research Platform (GRP).

• The Global Research Platform (GRP) Provide Next Generation Services.

Architecture, Techniques, and Technologies For World-Wide Data

Intensive Scientific Research.

• This Platform Provides Exceptional Support For Capturing, Managing,

Indexing, Analyzing, Storing, Sharing, Visualizing and Transporting

Exescale Data.

• Innovations Being Developed Include Those Related To:– Heterogeneity

– Virtualization,

– Segmentation

– Open Infrastructure Architecture and Components

– Orchestration

– Sliceability

– Multi-level Resource Abstraction and programmability,

– Granulated Customization,

– Fluid Data Distribution

– AI/machine Learning/Deep Learning

• Built-In Preconfigured Examples/Templates To Establish Infrastructure

Foundation Workflows

• Orchestration

• Zero-Touch “Playbooks” For Different Segments of Infrastructure

Foundation Workflows After Implementing Initial Suites (e.g., Using

Jupyter)

• Interactive Control Over Running Workflows

• Portability for Different Infrastructure Foundation Workflows

• Options/Capabilities for Specialized Customization

• Options For Real Time Visualization Of Individual Workflows, At Highly

Granulated Levels

Emerging Capabilities/Technologies

Network Research: Kernel Bypass

Networks

Drivers:

• AI Training/DL

• Distributed Storage Systems

• Virtual Networking

• Programmable Networks/SDN

• NFV

• In-Memory Caching

• Low Latency

• High Capacity (e.g., Big Data)

• Low Latency and High Capacity

Motivation For KBNets & Issues

• Need To Minimize Network CPU Overhead

• Sub Performance of Traditional OS Stack

Issues To Be Addressed

• Control Planes

• Data Planes (Transport)

• Management Plane

• Virtualization

• Programming Languages

• NIC/Backplane/Switch/Chip Architecture

Techniques

• RDMA (Remote Direct Memory Access)

• DPDK (Data Plane Development Kit)

• SmartNICs

• New Backplane/Switch Fabrics

• RoCE (RDMA Over Converged Ethernet)

• Et Al

• New High Performance OS Stacks

Programmable Network Techniques and

Devices

• SDN/SDX & Network Programming Languages

• Programmable Switch ASICs

• Programmable Network Processors

• FPGAs

• Programmable NICs

• Ref: Barefoot Tofino, Intel FlexPipe, Cavium XPliant,

Netronome Agilio.

• P4 Based In-Network Telemetry

• AI/ML/DL Integrated With Network Programming

• Jupyter

• DTNs

iCAIR: Founding Partner of the Global Lambda Integrated Facility

Available Advanced Network Resources

Visualization courtesy of Bob Patterson, NCSA; data compilation by Maxine Brown, UIC.

www.glif.is

AutoGOLE

International Multi-Domain Provisioning Using AutoGOLE

Based Network Service Interface

(NSI 2.0)

* Network Service Interface (NSI 2.0)

* An Architectural Standard Developed By the *Open Grid

Forum (OGF)

* OGF Pioneered Programmable Networking (Initially Termed

“Grid Networking”)

Techniques That Made Networks ‘First Class Citizens” in Grid

Environments – Programmable With Grid Middleware

* Currently Being Placed Into Production By R&E Networks

Around the World

IRNC: RXP: StarLight SDX A Software Defined

Networking Exchange for Global Science Research and

Education

Joe Mambretti, Director, (j-mambretti@northwestern.edu)

International Center for Advanced Internet Research (www.icair.org)

Northwestern University

Director, Metropolitan Research and Education Network (www.mren.org)

Co-Director, StarLight (www.startap.net/starlight)

PI IRNC: RXP: StarLight SDX

Co-PI Tom DeFanti, Research Scientist, (tdefanti@soe.ucsd.edu)

California Institute for Telecommunications and Information Technology (Calit2),

University of California, San Diego

Co-Director, StarLight

Co-PI Maxine Brown, Director, (maxine@uic.edu)

Electronic Visualization Laboratory, University of Illinois at Chicago

Co-Director, StarLight

Co-PI Jim Chen, Associate Director, International Center for Advanced Internet

Research, Northwestern University

National Science Foundation

International Research Network Connections Program

Workshop

Chicago, Illinois

May 15, 2015

IRNC SDX

IRNC SDX

IRNC SDX

IRNC SDX

GENI

SDX

UoM SDX

Emerging US SDX Interoperable Fabric

GENI

SDX IRNC SDX

Global LambdaGrid Workshop 2017

Demonstrations, Sydney Australia

International Multi-Domain Provisioning Using AutoGOLE Based

Network Service Interface (NSI 2.0)

Using RNP MEICAN Tools for NSI Provisioning

Large Scale Airline Data Transport Over SD-WANs Using NSI and

DTNs

Large Scale Science Data Transport Over SD-WANs Using NSI

and DTNs

SDX Interdomain Interoperability At L3

Transferring Large Files E2E Across WANs Enabled By SD-WANs

and SDXs

ESA Service Prototype

Source; John Graham UCSD

Source; Jim Chen, iCAIR

Implementing a SCinet DTN

SUPERMICRO 24X NVMe SUPER

SERVER

NVMe Type A: 8 X Intel P3700 800G

NVMe Type B: 8 X SamSung 950 512G

+ M.2 to U.2 Adopter

Dell 14G Solution Configuration

(Co-Research & Development in Collaboration

with Dell)

Intel® Xeon®

processorE5-2600 v4

Intel® Xeon®

processorE5-2600 v4

QPI2 Channels

DDR4

LANUp to

4x10GbE

PCIe* 3.0, 40 lanes

Intel® C610

series

chipset

WBG

DDR4

DDR4

DDR4

DDR4

DDR4

DDR4

DDR4

PowerEdge R740XD Server

2 X Intel® Xeon® Gold 6136 3.0G,12C/24T,10.4GT/s 2UPI,24.75M Cache,Turbo,HT (150W)

192G DDR4-2666

PCI-e Configuration Investigation:

2 X Mellanox ConnectX-5 100GE VPI

4 X Kingston/Liqid AIC NVMe PCI-e X8 SSD Drives

Optional SAS/SATA Drives

Recent WAN DTN Testing

• Preparation – Tests Conducted By Se-Young Yu (iCAIR)

• DTNs:

• @iCAIR : Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz, Mellanox

ConnectX-5 100G NIC

• @PACWAVE - LA : Intel(R) Xeon(R) CPU E5-2667 v4 @ 3.20GHz,

Mellanox ConnectX-5 100G NIC

• @UvA : Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz, Mellanox

ConnectX-3 40G NIC

• @CERN : Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, Intel 82599ES

10G NIC

• Tuning Parameters :

• BIOS, CPU, NIC, TCP Stack, O/S and MTU Tuning Applied

Memory-to-Memory Test Results

Disk To Memory Test Results

Wenji Wu, Phi Demar et al

Source: Wenji Wu

Source: Wenji Wu

Source: Wenji Wu

Source: Wenji Wu

Source: Wenji Wu

L2 SW Disaggregation: Falconwitch

Prototype ConfigurationNVMe Type A: Intel P3700 800G x 8

NVMe Type B: M.2 to U.2 Adopter with

SamSung 950 Pro 512G X 8

SamSung 960 Pro 1T X 8

SamSung 960 Pro 2T X 8

4 X Mellanox ConnectX-5 100GE

2 X PCI-e X16 host adapter to host

Host node: SuperWorkstation 7048GR-TR

2 X E5-2667 V4

2 X PCI-e X16 host adapter

Falconwitch PCI-e Switching Fabric

Selected ML Frameworks (Of Many):

• Apache Singa

• Caffe

• H2O

• MLlib (Apache Spark)

• Scikit-Learn (Python)

• Shogun (C++)

• TensorFlow

• Theano (Python)

• Torch (~ Scientific Computing)

• Veles (C++, w/ Some Python)

Summary

• Data Intensive Science Can Benefit From A GRP, Including

Enhanced Services/Techniques/Technologies For High

Performance WAN Data Transport

• One Approach: Relying On L2 WAN Transport Channels

• A Complementary Enabling Capability: Using DTNs Integrated With

Specialized WAN Paths To Optimize E2E Data Flows

• Core Components Can Be Supplemented By Enhancing Software

Stacks, e.g., Jupyter, NSI, MEICAN, P4 Programs, BDE, AI/ML/DL,

etc

• Today, Many Components Exist To Create An E2E Services For

Data Intensive Science

• Major Opportunity: Creating This Service and Placing It Into

Production

www.startap.net/starlight

Thanks to the NSF, DOE, DARPA,

NIH, USGS, NASA,

Universities, National Labs,

International Partners,

and Other Supporters