download - snia

EDUCATION

Unification of Data Center Fabrics with Ethernet“A High Speed Peek into the Future!”

Matthew Brisse, Dell Inc.

EDUCATION

Unification of Data Center Fabrics with Ethernet: “A High Speed Peek into the Future!”© 2007 Storage Networking Industry Association. All Rights Reserved. 2

SNIA Legal Notice

• The material contained in this tutorial is copyrighted by the SNIA.

• Member companies and individuals may use this material in presentations and literature under the following conditions:– Any slide or slides used must be reproduced without

modification– The SNIA must be acknowledged as source of any

material used in the body of any document containing material from these presentations.

• This presentation is a project of the SNIA Education Committee.

EDUCATION


Abstract

Unification of Data Center Fabrics with Ethernet: A High Speed Peek into the Future!

– A key challenge that the network and storage industries are now facing is the abundance of new high speed interconnect protocols proposed for future data center applications. In this presentation we take a peek into what the future may hold for high speed fabrics and investigate the potential for their unification. We will provide a market and technical overview of the competitive landscape for next generation 10Gb technologies with particular focus on the operational characteristics and implementation aspects of Ethernet.

EDUCATION


10GE Technology

• 10GE is 10 Gbps Ethernet– It has four different types of physical layer interfaces

• 1m: Backplane (802.3ap), XAUI (KX4) – 2.65GB/lane, 4 lane = 10GBKR (Serial) – 10 GB 1 lane

• 100m to 10km: Optical (SR, LR, LRM, ER)• 15m: CX-4 (Infiniband Type Cable-XAUI)• 55-100m 10GBase-T Copper (802.3an), over CAT6, CAT6a

or CAT7– Offload MUST be a part of any 10GE solution

• A “dumb-only” 10GE NIC is of limited value– 10GE readily consumes compute and memory resources (unless

offloaded)– Less latency and greater bandwidth (both are important)

• 10GE is more than just another, interchangeable physical interface– Offload and system characteristics are key to plumbing 10GE properly

EDUCATION


Copper Cable Options

CAT52 twisted pairs

CAT5e4 twisted pairs

CAT6Longitudinal spacer

CAT6aLongitudinal spiral spacer

CAT7Longitudinal spacer

Wrapped pairs, Shielded

Alien Crosstalk

Category Type Spectral B/W

Length Applications

Notes

Cat5 UTP 100MHz 100m 100Base-T Common

Cat5e UTP 100MHz 100m 1000Base-T Common

Cat6 UTP 250MHz 55m 10GBase-T Emerging

Cat6a UTP 400MHz 100m 10GBase-T Emerging

Cat7 ScTP 600MHz 100m 10GBase-T Emerging

(55m)(100m)(100m)

(100m)

$0.10/ft$0.15/ft$0.65/ft $0.05/ft$1.25/ft

? RJ45 RJ45 RJ45 RJ45

1GE

10GE

SHIELDED UNSHIELDED

RJ45 Challenge!

EDUCATION


40 Gbps, bi-directional datarate

PCIe and 10GE

16 Gbps, effective bi-directional throughput

10GE

20Gbps bi-directional

PCIe, Gen1 x8

Server

80 Gbps, bi-directional datarate

32 Gbps, effective bi-directional throughput

10GE

20Gbps bi-directional

PCIe, Gen2 x8

Server

PCI Express Generation 2 will be required to support full 10GE with x8(Current 10GE NICs can’t realize full throughput due to PCIe Gen 1 bus speed limitations but PCIe Gen 2

will realize the maximum throughput)

Datarate: Bits on the wireThroughput: Information on the wire

EDUCATION


10 GE Requires Multi-Function Offload Capabilities

• Simple NICs offload nothing• Stateless offload accelerates packet processing (RSS, LSO, TSO, Checksum)• Stateful offload provides protocol and packet processing (TOE, Chimney, RDMA, iWARP)• RDMA abstracts the underlying transport and achieves direct memory to memory data placement • Virtualization mediates access to multiple instantiations of limited hardware resources

NIC

TOE

RDMA (iWARP)RDMA over TCP/IP

VIRTUALIZATION

ACCELERATION

Lower CPU usageGreater I/O efficienciesHigher network throughput

10 GE Viability

1 GE Viability

Now 2007

ProtocolAcceleration

Check SumOff Load

No

Off

Load

EDUCATION


• 10GE is an acceptable technology for fabric convergence• IB and 10GE bandwidth both have visibility to 100GE• FC will continue to have a place for a considerable time• 10GE latency will settle at < 10 microseconds end-to-end• Cost will come down – Economy of scale

10GE Overall Technology Comparison

EN

FC

IBSPEED LATENCY(Gbps)

(µsec)

108

15

10

6

400

300

400

COST

($)

16

POWER

5

4 4UNIFIED FABRIC

SWEET SPOT

(End-to-End)

Fastest, less latency, cluster + storage “was” the fabric convergence in the past

(W)(NIC/HBA) (SR MAC/PHY)

Fast, some latency good enough

Faster, less latency, storage

EDUCATION


10GE non-Technology Comparison• FC and IB will both perform “better” than 10GE in cluster and storage fabrics,

respectively• 10GE will, however, be “good enough” to potentially gain market share from IB

and FC• Non technical factors will drive 10 GE adoption and continued evolution

– Ethernet trust, ubiquity and robust investment climate are important driving factors• Ethernet always incorporates the best from competing technologies

– EtherNOT becomes EtherNET

TRUST

LO

HIHI

UBIQUITY

LO

MED

HI

INVESTMENT

LO

MED

HI

Well-known protocolsWell-known management

“Staying power”

Ethernet everywhere More $ to EthernetMore Ethernet R&D

EN

FC

IB

EDUCATION


Unified Fabric “A peek in the future”

• Reduce the number of disparate networks and Data Center complexity

• Provide a seamless model for networking, clustering and storage applications– Not to mention thin client, KVM and other usages

• Consolidate Management protocols over a common IP framework• Leverage economies of scale in terms of silicon integration,

interoperability and product development• Provide support for legacy network architectures in the transition to a

unified fabric architecture

EDUCATION


Unified Fabric VisionMaking Many Fabrics……………

IBEN

FC

COMPATIBLE SCALABLE

HIGH TRANSACTION RATE

HIGH

THROUGHPUTLOW

LATENCY

BLOCK

TRANSFERS

LOW CPU

UTILIZATION

HPCLAN

SAN

NAS

EDUCATION


10GE

Making Many Fabrics……………

Simplifying

COMPATIBLE SCALABLE

HIGH TRANSACTION RATE

HIGH

THROUGHPUT

LOW

LATENCY

BLOCK

TRANSFERS

LOW CPU

UTILIZATION

HPC

LAN

SAN

NAS

OneUnified Fabric Vision

EDUCATION


Emerging Trends• Clustering

– Improves cost-performance over large scale cache coherent systems– Brings flexibility to the data center– Provides better fault tolerance

• Grid– Academic processing Grids tried and tested– Commercial deployment in niche areas only– Storage Grids now in discussion

• Object Storage– Completing standardisation– New paradigm of intelligent storage– Requires converged compute & storage platform

• Systems Management– Keyboard Video & Mouse (KVM) already available and other tools are

coming• System Level Virtualization• Ethernet is rapidly becoming the key to enterprise systems convergence

EDUCATION


Industry Initiatives

• Blurring of compute and storage is now happening– Traditional view of totally discrete functions is getting old– New applications require dynamic configurations– Lightweight blades clustered with shared access to I/O resources are

enabling convergence

• Systems Virtualization – Protected virtual memory architectures return to give scale up

virtualisation– Scale out virtualization still requires major developments in the industry

but everyone expects it to happen

• Server re-purposing paradigm maturing – The 3 tier Data center model is now passing its sell by date

• Evolving Ethernet enables system wide convergence

EDUCATION


Target 10GE Applications

• Servers– HPC

• server-to-server interconnect– Virtualization

• increased bandwidth– High-Performance Transactions

• financial applications• IPTV streaming media

applications• Digital Cinema• web farms

• Switches– Edge Layer

• server-to-switch uplink• iSCSI storage array to switch

– Aggregation & Core Layer• switch-to-switch interconnect

• iSCSI Storage Arrays– General Purpose

• array-to-switch connectivity

• Single Adapter• Multiple Offloads• Common

Management• Simplicity

10GE provides the basis for a unified datacenter fabric

10GE UNIFIED FABRIC

LAN

HPC

SAN

EDUCATION


Enterprise Network Today

DatabaseClients Application ServersWeb Servers

SwitchesLoad

Balancers

IP/Ethernet Storage Cluster Management

StorageArray Units

SAN

Cluster

DAS

DAS

RAID Storage

RAID Storage

Enterprise Network Today

EDUCATION


Converged NetworkDatabaseClients Application ServersWeb Servers

SwitchesLoad

Balancers

IP/Ethernet

StorageArray Units

Storage Cluster Management

Cluster

SANDAS

DAS

RAID Storage

RAID Storage

SAN

Converged Network

EDUCATION


Backup Networking SwitchEthernetEthernet

Server Blade

NetworkSwitch

StorageSwitch

ClusterSwitch

ManagementSwitch

CPU

Compute Blades

Mid-plane

Backup Storage Switch

Backup Clustering Switch

Management Switch

Clustering Switch

Storage Switch

Local Hard Disk

Unified Fabric Example –Server Blades

E-net IB

FCMGT

Networking SwitchEthernetEthernet

EDUCATION


Ethernet Ethernet

CPU

Compute Blades

Server Blade

NetworkSwitch

StorageSwitch

ClusterSwitch

ManagementSwitch

Mid-plane

iSCSI

New Ethernet Technologies

Remote Direct Memory Access

(RDMA)

Integrated Management

Accelerated Networking (TOE)

Local Hard Disk

Unified Fabric Example –Server Blades

MGT

E-net

EthernetEthernet

IB

FC

EDUCATION


10GE Switch Market Growth

• 10GE growth (2008-09) driven by adoption of low-cost 10GBASE-T copper• 1GE will be pushed to the edge of LANs and must be aggregated with 10GE• 10GE will be initially positioned for inter-switch connections and clusters• Vendors drive low-cost 10GE, accelerating adoption resulting from customer demand

CY03 CY04 CY05 CY06 CY07 CY08 CY090

5

10

15

20

Rev

enue

($B

)

100 Mbps

1 Gbps

10 Gbps

Compound Annual Growth RateCY03 – CY09

-30%

10%

55%

Source: IDC 2005

EDUCATION


10GE Market

Available market data represents growth in the 10GE switch port market, not the NIC or other 10GE markets. The growth in 10GE infrastructure, however, implies growth in other 10GE markets, including server port shipments.

Source: Dell’Oro Group January 2006

000’s switch ports 2004 2005 2006 2007 2008 2009 2010

Copper Fiber Total % Copper

2556961

57 785

58

163

169

1% 4%

854

8%

1,616

1,871

14%

1,013

2,511

3,524

29%

2,509

4,008

6,517

39%

4,209

5,748

9,958

42%

5x port growth from 2005 to 2006; 10m ports, $4B market in 2010

0

1000

2000

3000

4000

5000

6000

2004 2005 2006 2007 2008 2009 2010

Fiber

Copper

2008 – Year of copper10W today, 5W power ’082-3W down on the Motherboard

EDUCATION


Cluster Fabric Trends

INFINIBAND

MYRINET

OTHER

ETHERNET1 GE 1 GE

I0GE

Top500.org, Nov 2005 Projection, 2008

EDUCATION


NIC 10GE Cost Trends

2005 2006 2007 20080

2x

4x

6x

8x

10x

Rel

ativ

e C

ost (

1GE

Cy0

4 =

1.0)

1GE vs. 10GE RELATIVE PRICE TARGET

10 GE 3-4x 1 GE for wide adoption

2004

10GE

1 GE

~$100/port 10GE NICs

EDUCATION


10GbE Standards

Media Access Controller (MAC) Full-Duplex

10 Gigabit Media Independent Interface (XGMII) or10 Gigabit Attachment Unit Interface (XAUI)

Serial LAN PHY(64B/66B)

Serial WAN PHY(64B/66B + WIS)

LAN PHY(8B/10B)

Various Optical Technologies

802.3ae

10GBaseT

10GBackplane

Cat6/7 Cable

802.3an 802.3ap

Layer 2

802.3an and 802.3ap make 10GbE even more compelling for data center applications

EDUCATION


10GE Copper/Backplane Standards and Opportunity Timeline

CY05 CY06 CY07 CY08 CY09 CY10

802.3an ratified

(10GBASE-T)

MATURE PRODUCTS

EMERGING SOLUTIONS

CONCEPT and DESIGN

BARRIERS TO OVERCOME

SWITCH

NIC

BLADEBLADE

LOM

802.3ap ratified

(Backplane)

EDUCATION


Changes with 10GbE

• 802.3ae defines 10GbE– 10Gbps Data Rates– Full-duplex only; no need for carrier-sensing multi-

access / collision detection (CSMA/CD)– Optical Physical Layer

• LAN PHY and WAN PHY options– WAN PHY compatible with SONET

• 802.3an adds twisted pair cabling– Cat 6 and Cat 7

• 802.3ap adds backplane specifications– Blade servers and communications equipment

EDUCATION


IEEE 10GE Standards

• IEEE P802.3ae, 10GE Working Group– Definition of the 10GE MAC architecture and components

• IEEE P802.3an, 10GBASE-T Task Force.– Definition of 10GE encoding over CAT 6/7 (Copper) cable

• IEEE P802.3ap, Backplane Ethernet Task Force.– Definition of 1/10GE encoding over backplanes (KX, KX4, KR)

• IEEE P802.3aq, 10GBASE-LRM Task Force.– Definition of short haul MMF/1310nm LR encoding

• IEEE P802.3ar, Congestion Management Task Force.– Definition of L2 congestion discovery and avoidance protocols

• IEEE P802.3as, Frame Expansion Task Force.– Not “Jumbo” frames; expansion of frame envelope, not data size

• IEEE P802.3at, DTE Power Enhancements Task Force.– Definition of how to deliver power at 40-60W over 1 and 10GE

http://www.ieee802.org/3/ae

http://www.ieee802.org/3/an/index.html

http://www.ieee802.org/3/ap/index.html

http://www.ieee802.org/3/aq/index.html

http://www.ieee802.org/3/ar/index.html

http://www.ieee802.org/3/as/index.html

http://www.ieee802.org/3/at/index.html

EDUCATION


Unified Fabric End Goal

• STANDARDS• SIMPLICITY• PARTNERS• COORDINATION• CONSOLIDATION• SNIA Standards

LEADERSHIP

UNIFIED FABRIC10-Gigabit Ethernet

HardwareNIC/HBA

PHY

MACSWITCH

VIO

StandardsIEEE

IETF

DMTF

SoftwareOS

APPVOS

MGMT

SECURITY

InnovationSIMPLE

EFFICIENT

SECURE SNIA

EDUCATION


Call to Action

• Get involved!! – Participate in Standards– Drive your vendors– Leverage the power of the user

• Ethernet vendors are developing new protocol acceleration features and the speed is increasing from 1G to 10G, Understand the future impact

• 10Gb Ethernet with protocol acceleration could be used a 'Unified fabric' to handle SAN, HPC, management and client network needs in addition to traditional data networking requirements.

• Be prepared for the transition! Plan today!

EDUCATION


Q&A / Feedback• Please send any questions or comments on this presentation to

SNIA: [email protected]

Many thanks to the following individuals for their contributions to this tutorial.

SNIA Education Committee

Allen Light Robert WinterMatt Brisse Robert Peglar

mailto:[email protected]

EDUCATION

Questions?

download - snia

Documents