Pacific Wave and PRP UpdateBig News for Big Data

David ReeseDr. Larry Smarr

CENIC ANNUAL CONFERENCE 2016

Pacific WaveInternational Exchange

A project of CENIC and PNWGPDavid Reese

Managing Director

Pacific Wave• Began as first geographically distributed exchange in 2004• Pacific Wave is an open exchange supporting both

commercial and R&E peers• Currently serves 29 countries peering across the Pacific

and Western United States• With PNWGP and TransPac, announced the first 100Gbps

Trans-Pacific link from Tokyo to Seattle in 2015

Pacific Wave Exchange Map

R&E Exchanges within R&E• Pacific Wave (Western US)

– CENIC and PNWGP• StarLight (Chicago, IL)

– StarLight Consortium/MREN• MANLAN (New York, NY)

– NYSERnet• WIX (Washington, DC)

– University of Maryland/MAX GigaPOP• AmLight (Miami, Florida)

– Florida International University/Florida LambdaRail

National/Global Activities• NSF provides support of the R&E exchange points

through the competitive IRNC (International Research Network Connections) program with funding for backbone, infrastructure and innovation

• The Global Lambda Integrated Facility– The GLIF brings together some of the world’s premier

networking engineers who are working together to develop and international infrastructure

GLIF/GOLE

Pacific Wave and NSF/IRNC• Pacific Wave has been partially supported

through three separate five-year National Science Foundation grants supporting growth, connectivity and innovation

• Current award promotes 100G expansion and implementation of SDX capabilities within Pacific Wave (ACI-1451050)

SDX = SDN + IXP

9

AS A Router

AS C Router

AS B Router

BGP Session

SDN Switch

SDX ControllerSDX

Pacific Waveand WRN

• Pacific Wave and the Western Region Network provide for a 100Gbps network spanning the Western United States serving PNWGP, CENIC, FRGP, ABQGP and UH.

• Pacific Wave and NSF IRNC awardee PIREN (Univ of Hawaii) work together supporting AARNet links to California and Washington and expansion of high-speed service through the Pacific Islands Region

Pacific Wave and Collaboration• Pacific Wave and StarLight are working together

on the SDX implementation to provide seamless cross-domain support for participants of both exchanges.

• Have direct interconnectivity between the exchanges with 2 100G links between Seattle-Chicago and Denver-Chicago

Nx100G Across the Pacific• CURRENT:

– TransPac/Pacific Wave (Tokyo-Seattle)– SINGAREN/Internet2 (Singapore-Los Angeles)– SINET/SoftBank/Pacific Wave (Tokyo-Los Angeles)

• FUTURE:– AARNET/PIREN/Pacific Wave (Australia-LA)– AARNET/PIREN/Pacific Wave (Australia-SEA)– UH/PIREN/Pacific Wave (Guam-Hawaii-LA)

The Pacific Research Platform (PRP)• NSF CC-NIE and similar projects represent significant investments in campus

infrastructure including SDN, DMZ’s (~130 projects)

• But the scientists are still struggling with the complexity of using the network and interoperability between different implementations of DMZ’s

• PRP focuses on enabling the science communities to make effective use of the high performance infrastructure that is available.

• The idea was hatched in December 2014 – take advantages of the infrastructure, including a PERFSONAR grid for measurement.

• And DTN’s and common software suite to demonstrate a proof of concept for the PRP

• Demonstrated at the CENIC Spring meeting (March 2015)

CENIC/PRP Backbone Sets Stage for 2016 Wireless Expansion of HPWREN into Orange and Riverside Counties

• CENIC/PRP Will Connect UCSD and SDSU

– Data Redundancy – Disaster Recovery – High Availability

• CENIC Extension to UCI & UCR– Data Replication Sites

UCR

UCI

UCSD

SDSU

Source: Frank Vernon, Greg Hidley, UCSD

“Pacific Wave & Pacific Research Platform Update:

Big News for Big Data”

Invited Presentation with David ReeseAnnual CENIC Conference 2016

UC DavisMarch 21, 2016

Dr. Larry SmarrDirector, California Institute for Telecommunications and Information Technology

Harry E. Gruber Professor, Dept. of Computer Science and Engineering

Jacobs School of Engineering, UCSDhttp://lsmarr.calit2.net 16

For Big Data Science, One Needs Bandwidths Orders of Magnitude Higher Than the Shared Internet

Bandwidth from My Office in Calit2’s Qualcomm Institute

Bandwidth On the Pacific Research Platform:

500 Times the Bandwidth of the Shared Internet!

How Prism@UCSD Transforms Big Data Microbiome Science:Preparing for Knight/Smarr 1 Million Core-Hour Analysis

12 Cores/GPU128 GB RAM3.5 TB SSD48TB Disk

10Gbps NIC

Knight Lab

10Gbps

Gordon

Prism@UCSD

Data Oasis7.5PB,

200GB/s

Knight 1024 ClusterIn SDSC Co-Lo

CHERuB100Gbps

Emperor & Other Vis Tools

64Mpixel Data Analysis Wall

120Gbps

40Gbps

1.3Tbps

Next Step: The Pacific Research Platform Creates a Regional End-to-End Science-Driven “Big Data Freeway System”

NSF CC*DNI Grant$5M 10/2015-10/2020

PI: Larry Smarr, UC San Diego Calit2Co-Pis:• Camille Crittenden, UC Berkeley CITRIS, • Tom DeFanti, UC San Diego Calit2, • Philip Papadopoulos, UC San Diego SDSC, • Frank Wuerthwein, UC San Diego Physics and

SDSC

Dan Cayan USGS Water Resources Discipline

Scripps Institution of Oceanography, UC San Diego

much support from Mary Tyree, Mike Dettinger, Guido Franco and other colleagues

NCAR Upgrading to 10Gbps Link from Wyoming and Boulder to CENIC/PRP

Sponsors: California Energy Commission NOAA RISA program California DWR, DOE, NSF

Planning for climate change in California substantial shifts on top of already high climate variability

UCSD Campus Climate Researchers Need to Download Results from NCAR Remote Supercomputer Simulations

to Make Regional Climate Change Forecasts

average summer afternoon temperatureaverage summer afternoon temperature

Downscaling Supercomputer Climate SimulationsTo Provide High Res Predictions for California Over Next 50 Years

21

Source: Hugo Hidalgo, Tapash Das, Mike Dettinger

Cancer Genomics Hub (UCSC) is Housed in SDSC:Large Data Flows to End Users at UCSC, UCB, UCSF, …

1G

8G

Data Source: David Haussler, Brad Smith, UCSC

15GJan 2016

Cancer Genomics Hub Users are Downloading 30,000 TB per Year

GE’s Industrial Internet Generates 10,000 TB per Day! 30,000 TBPer Year

Two Automated Telescope SurveysCreating Huge Datasets Will Drive PRP

300 images per night. 100MB per raw image

30GB per night

120GB per night

250 images per night. 530MB per raw image

150 GB per night

800GB per nightWhen processed

at NERSC Increased by 4x

Source: Peter Nugent, Division Deputy for Scientific Engagement, LBLProfessor of Astronomy, UC Berkeley

Precursors to LSST and NCSA

PRP Allows Researchersto Bring Datasets from NERSC

to Their Local Clusters for In-Depth Science Analysis

OSG Federates Clusters in 40/50 States:Creating a Scientific Compute and Storage “Cloud”

Source: Miron Livny, Frank Wuerthwein, OSG

We are Experimenting with the PRP for Large Hadron Collider Data Analysis Using The West Coast Open Science Grid on 10-100Gbps Optical Networks

Crossed 100 Million

Core-Hours/MonthIn Dec 2015

Over 1 Billion Data Transfers

Moved200 Petabytes

In 2015

Supported Over200 Million Jobs

In 2015

Source: Miron Livny, Frank Wuerthwein, OSG

ATLAS

CMS

PRP Will Support the Computation and Data Analysisin the Search for Sources of Gravitational Radiation

Connecting Caltech at 10Gb/s to SDSC Comet PetaFLOP supercomputer,

enabling LIGO computations to enter via the same PRP “job cache” as for LHC.

800,000 core-hours LIGO data analysis7 million core-hours computing waveforms

Forty Years of Computing Gravitational Waves From Colliding Black Holes

1977

L. Smarr and K. EppleyGravitational Radiation Computed from

an Axisymmetric Black Hole Collision 40 Years

2016

LIGO ConsortiumSpiral Black Hole Collision

MegaFLOPSPetaFLOPS

40G FIONAs

20x40G PRP-connected

WAVE@UC San Diego

PRP LinksCreates Distributed Virtual Reality

PRP

CAVE@UC Merced

UCD

UCSF

Stanford

NASAAMES/NREN

UCSC

UCSB

Caltech

USC UCLA

UCIUCSD SDSU

UCR

EsnetDoE Labs

UW/PNWGPSeattle

Berkeley

UCM

Los Nettos

Internet2

Internet2Seattle

Note: This diagram represents a subset of sites and connections.

* Institutions withActive Archaeology

Programs

“In an ideal world –Extremely high bandwidth to

move large cultural heritage datasets around the PRP cloud for

processing & viewing in CAVEs around PRP with Unlimited

Storage for permanent archiving.”-Tom Levy, UCSD

PRP is NOT Just for Big Data Science and Engineering:Linking Cultural Heritage and Archaeology Datasets

Building on CENIC’s ExpansionTo Libraries, Museums,

and Cultural Sites

Next Step: Global Research PlatformBuilding on CENIC/Pacific Wave and GLIF

Current InternationalGRP Partners

Learn More About PRP on Wednesday

PRP Science Driver PI Workshop Description

[Flyers will be available at the CENIC registration desk]

Date and Time: Wed., March 23 from 9 - 11 am

Location: Executive Meeting Room, Hyatt Place UC Davis

Description: Panel discussion on capacity and promise of the Pacific Research Platform, a high-speed, expanded cyberinfrastructure that will move data 1,000 times faster than today's inter-campus shared Internet.