accelerating hpc with ethernet
DESCRIPTION
In this presentation, Derek Granath from Extreme Networks presents: Accelerating HPC with Ethernet. "High Performance Computing (HPC) requires high-speed, high bandwidth, low latency network infrastructures. Converged Ethernet networks are now displacing dedicated compute, storage and data networks of the past in today's HPC deployments. Top industry institutions; petroleum, human genome, finance and universities have partnered with Extreme Networks to solve their toughest HPC networking challenges." Learn more: http://extremenetworks.com Watch the video presentation: http://insidehpc.com/2013/12/13/slidecast-accelerating-hpc-ethernet/TRANSCRIPT
© 2013 Extreme Networks Inc. All rights reserved.
Senior Director, Product Line Management
December 10, 2013
Derek Granath
Accelerating High Performance
Computing with Ethernet
© 2013 Extreme Networks Inc. All rights reserved.
Agenda
High Speed, Low Latency Ethernet for HPC
Architecture Examples
HPC Interconnect Challenges and Alternatives
Transforming Data into Information
© 2013 Extreme Networks Inc. All rights reserved.
How Much Data?
3
The World Generates 2.5 Quintillion bytes each day…
or 57.5 Billion 32Gb iPad’s worth
each day
© 2013 Extreme Networks Inc. All rights reserved.
Image: NCSA’s Blue Waters sustained petascale supercomputing facility
Growth of Data
90% of the data in the world was created in the past
TWO YEARS Source: IBM, 2012
© 2013 Extreme Networks Inc. All rights reserved.
Data Analytics Challenge
5
User Generated Content
Social Interactions & Feeds
Spatial & GPS Coordinates
External Demographics
Business Data Feeds
HD Video, Audio, Images
Speech to Text
Product/Service Log
SMS/MMS
BIG DATA Sensors/RFID/Devices
Mobile Web
User Click Stream
Sentiment
Affiliate Networks
Search Marketing
Behavioral Targeting
Dynamic Funnels
A/B Testing
Dynamic Pricing
Web Logs
Offer History
WEB
Source: Contents of above graphic created in partnership with Teradata, Inc. http://tinyurl.com/mt4ltah
Big Data = Transactions + Interactions + Observations
CRM Segmentation
Offer Details
Customer Touches
Support Contacts
ERP
Purchase Details
Purchase Record
Payment Record
Increasing Data Variety and Complexity
Petabytes
Terabytes
Gigabytes
Megabytes
Interactions
Transactions
Observations
© 2013 Extreme Networks Inc. All rights reserved.
Structured and Unstructured Data
Example
6
Structured Data
• Fits neatly into traditional
database schemas
• Email metadata
• Call records
• Can be easily
• Stored
• Queried
• Analyzed
Unstructured Data
• Everything else…
• May contain patterns
• Also it might not!
• Video
• Audio
• Photos
• This Presentation!
• Doesn’t fit in fixed length fields
© 2013 Extreme Networks Inc. All rights reserved.
Utilities Weather impact analysis on
power generation Transmission monitoring Smart grid management
Retail 360° View of the Customer
Click-stream analysis
Real-time promotions
Law Enforcement Real-time multimodal surveillance
Situational awareness and threat detection
Cyber security detection
Transportation Weather and traffic
impact on logistics and
fuel consumption
Traffic congestion
Financial Services Fraud detection
Risk management
High Frequency Trading
360° View of the Customer
IT System log analysis Cybersecurity
Telecommunications CDR processing
Churn prediction
Geomapping / marketing
Network monitoring
7
Health & Life Sciences DNA sequencing
Epidemic early warning
ICU monitoring
Remote healthcare monitoring
HPC – What Can You Do with it?
7
© 2013 Extreme Networks Inc. All rights reserved.
HPC Architectural Evolution
– Clusters Dominate
8
© 2013 Extreme Networks Inc. All rights reserved.
Moore’s Law For HPC
Doubles Every 1.5 Years
Doubles Every 4 Years
Doubles Every 1.5 Years
Compute Data I/O
9
© 2013 Extreme Networks Inc. All rights reserved.
Interconnect Challenges
10
• Flexibility to grow big cheaply
• Deploy and re-deploy assets
Massive Scalability
• More servers, more storage, more applications
Manageability
• Proven, certified interoperability
• Standards-based technology
Eased Convergence
• Energy
• Operational
Efficiency
• Resilient architectures
Availability
© 2013 Extreme Networks Inc. All rights reserved.
Network Considerations for Big Data
11
Processing Time
Traffic Burstiness
Barrier Synchronized Computation
Buffering & Burst Handling
Das, Anupam et. al, "Transparent and Flexible Network Management for Big Data Processing in the Cloud."
Data Volume
Large long-lived flows
© 2013 Extreme Networks Inc. All rights reserved.
Next-Generation HPC clusters demand:
Latency
• Least time taken to complete a job o Lowest port-to-port latency in the I/O fabric
o Jumbo-frame support
o Short-reach optics and cables support
Throughput
• Non-blocking performance for any-to-any connectivity o Switch fabric performance
o Non-oversubscribed, wire-speed architecture
o Cut-through support
Bandwidth
• Ample bandwidth for multiple applications and jobs o Higher speeds and feeds: 10/40/100 Gigabit Ethernet
o Higher density per slot
o Link aggregation (LAG) support
Top 3 Requirements for HPC/SC
Interconnect Fabric
© 2013 Extreme Networks Inc. All rights reserved.
I/O Technology to Meet the Future
Demand
To bridge the gap, you need an I/O technology, that is:
• Minimal cost increase with speed migrations
• Reusable in terms of infrastructure and training Economical
• Short, medium, long range connectivity options
• I/O diversity and mix-n-match Flexible
• Bandwidth aggregation to multiply I/O
• Seamless migration to higher speeds and feeds Scalable
• Is resilient and time tested
• Provides required level of service up time Reliable
© 2013 Extreme Networks Inc. All rights reserved.
HPC I/O Technology Alternatives
Infiniband Fiber Channel Ethernet
Scalability
Flexibility
Reliability
Bandwidth
Latency
Economics
© 2013 Extreme Networks Inc. All rights reserved.
Ethernet and InfiniBand Dominate
Note: Gigabit Ethernet Category includes 10GbE
15
© 2013 Extreme Networks Inc. All rights reserved.
Ethernet Penetration in Top 500
212 (42.4%) of world’s top 500 fastest supercomputers use Ethernet
Source: Top500.org, November 2013
© 2013 Extreme Networks Inc. All rights reserved.
RoCE – RDMA over Converged Ethernet
• RoCE is a link layer protocol between two hosts in a broadcast domain
• Allows Remote Direct Memory Access (RDMA) similar to Infiniband to run over Ethernet
• RoCE replaces IB link layer with Ethernet
• Simpler than iWARP (Internet Wide Area RDMA Protocol)
• RoCE is marginally slower than IB which is sub μs, and latencies can approach 1-3 μs (micro-seconds), but less expensive and lower power to deploy than IB
• Ideal for High Performance Cluster Computing environments already familiar with Ethernet Technology, but need the speed and agility of IB
• No Support for IP (unlike iWARP): Need to use a head-end gateway to access closed- cluster environments
• Multicast RDMA is defined for RoCE (also unlike iWARP)
• Requires IEEE Data Center Bridging for PFC support on the network, a RoCE capable Ethernet adapter for hardware acceleration
17
© 2013 Extreme Networks Inc. All rights reserved.
Access
High Performance Interconnect Fabric
40G
10G
Back End
Sample Architecture
Clients
Front End
Compute Nodes
Cluster
Ma
ste
r N
od
es
Storage
Compute Nodes
Cluster
40G
© 2013 Extreme Networks Inc. All rights reserved.
HPC Design with Centralized Storage
Master
Nodes Master
Nodes
Compute Nodes Compute Nodes Storage Nodes
High-Performance Fabric
10GbE Data Path
40GbE Storage Path
Ac
ce
ss
Front End Front End
Back End
Ac
ce
ss
© 2013 Extreme Networks Inc. All rights reserved.
Cluster Interconnect Fabric Options
20
Fully Non-blocking Architecture • Six 32 x 40G switches
• 256 10G ports, fully non-blocking
• 64 10G ports to each server
• 16 40G uplinks per rack to spine
• Less than 1.8 microsecond latency
3:1 Over-subscribed • Ten 32 x 40G switches
• 768 10G ports to servers
• 8 40G uplinks per rack to spine
• Less than 1.8 microsecond
latency
© 2013 Extreme Networks Inc. All rights reserved.
Ethernet evolves –
That’s what it does!
21
Thank You