new switch architectures and the impact to 40/100g - bicsi
TRANSCRIPT
New Switch Architectures and the Impact to 40/100G Migrationthe Impact to 40/100G Migration
in the Data CenterGary Bernstein, RCDD, CDCDLeviton Network SolutionsLeviton Network Solutions
Agendag• Bandwidth Drivers & Expected Growth • Existing 40 & 100G Equipment Available Today• TIA-942-A-1 Standard Network Architectures• New Switch 40/100G Architectures Available
TodayToday• Impact of the New Architectures on Cabling
Topologies & DesignsTopologies & Designs
The Equation Remains the Sameq
2015 Global Users and Network Connections
• 15 Billion Total Network Connections by 2015y
Computing & Networking Growthp g g• 2012 study confirms data from 2007
» Source: IEEE 802 3 Bandwidth Assessment» Source: IEEE 802.3 Bandwidth Assessment » Report, July 2012
Server Bandwidth Growth
100G Equipment Available Todayq p y• Brocade MLX Series• Juniper T1600 Series• Cisco CRS-3 Series
40G Equipment Available Todayq p y• Extreme Networks Summit ToR & Black
Di d C S i hDiamond Core Switch• Cisco Nexus 3064 ToR Switch• Extreme X670 ToR with 10GBaseT
40G Equipment Available Todayq p y• Force 10 S4810 ToR & E-Series Core Switch• Arista 7050S ToR Switch• Arista 7050Q High Density Switchg y
Switch Density Continues to Increase
• Mellanox SX1024 ToR switch with 12x40G
• Extreme BlackDiamondX8 with 192x40G ports in 15Uin 15U
New 40G Extended Reach Transceivers
• QSFP+ 40G Short Reach Transceiver Available Soon• Exceeds IEEE802 3ba 40G requirements• Exceeds IEEE802.3ba 40G requirements• Capable of extended reach for 40G over MMF
– 300 meters over OM3 fiber
Source: Avago
– 400 meters over OM4 fiber
• Backwards compatible to 10Gl l di i h f ill b ff i i i• Several leading switch manufacturers will be offering option in
2013• Bottom Line…Bottom Line…
– MMF can support 40G beyond 150 meters– Lower cost option than current Single-mode links
Will lik l i f k d i– Will likely impact future network designs
Equipment Architecture Trendsq p• Leading active equipment manufacturers like Cisco,
Brocade Juniper Extreme Networks and others haveBrocade, Juniper, Extreme Networks and others have developed new fabric architectures
• There is a new TIA addendum which addresses some• There is a new TIA addendum which addresses some of these new architectures
• Data center trends driving change:Data center trends driving change:– Virtualization– Cloud Computingp g– Lower latency requirements– Mobile device apps driving east-west flow in DC
TIA-942-A-1 Proposed Standardp• New addendum which specifies recommendations for
telecommunications cabling to support data center switchtelecommunications cabling to support data center switch fabrics
• Switch Fabric architectures are intended to provide the following between any 2 points in data center:– Low-latency– High-bandwidth– Non-blocking ports
• Primary Switch Fabric Architectures:– Fat-Tree– Full-Mesh– Interconnected Mesh– Centralized Switch– Virtual Switch
TIA-942-A-1: Traditional Data Center ArchitectureArchitecture
This content, from the draft ANSI/TIA-942-A-1 Standard, Telecommunications Infrastructure Standard for Data Centers, is d d d itt i i f T l i ti I d t A i tireproduced under written permission from Telecommunications Industry Association.
Example: Traditional 3-Tier Architecture
» Source: Cisco Systems
Example: Traditional 3-Tier Architecture
» Source: Brocade
Cabling to Support 3-Tier Architecture
TIA-942-A-1: Fat-Tree (or Leaf-Spine)
This content, from the draft ANSI/TIA-942-A-1 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Associationreproduced under written permission from Telecommunications Industry Association.
Fat-Tree Architecture• All Access Switches are connected to every
Interconnection SwitchInterconnection Switch• Flattens and scales out Layer 2 network at the edge
C t bl ki l l t f b i• Creates a non-blocking, low latency fabric• Size and Performance of Interconnection Switches
will limit size of fabricwill limit size of fabric• Very scalable fabric
T i ll i l t d i l d t t ith• Typically implemented in large data center with Access Switches in HDAs
Example: Fat-Tree Architecturep
» Source: Extreme Networks
Cabling to Support Fat-Tree Architecture 40G Uplinks40G Uplinks
TIA-942-A-1: Full Mesh
This content, from the draft ANSI/TIA-942-A-1 Standard, Telecommunications Infrastructure Standard for Data Centers, is d d d itt i i f T l i ti I d t A i tireproduced under written permission from Telecommunications Industry Association.
Full Mesh Architecture• All switches are connected to every other
i hswitch• Does not scale very well• Primarily used in small data centers• Switches are not typically in EDASwitches are not typically in EDA• Fabric is not used for ToR topology
TIA-942-A-1: Interconnected Mesh
This content, from the draft ANSI/TIA-942-A-1 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association.
Interconnected Mesh Architecture• Typically has 1 – 3 Interconnection switches• Typically non-blocking within each POD• Fabric within POD is a full-mesh• Access switches may be in HDAs or EDAs• Very scalable fabric architecture• Very scalable fabric architecture
Example: Interconnected Full Mesh
40/100G-A & B
40G-A 40G-B
» Source: Extreme Networks
10G-B 10G-A
» Source: Extreme Networks
TIA-942-A-1: Centralized
This content, from the draft ANSI/TIA-942-A-1 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association.
Centralized Architecture• All servers are connected to all fabric switches• Simple and low latency• Does not scale well because of port limitationsp• Primarily used in small data centers
TIA-942-A-1: Virtual switch
This content, from the draft ANSI/TIA-942-A-1 Standard, Telecommunications Infrastructure Standard for Data Centers, is reproduced under written permission from Telecommunications Industry Association.
Virtual Switch Architecture• Similar to Centralized, except switches are
i d f i l i l i hinterconnected to form a single, virtual switch• Each server may be connected to multiple
switches for redundancy• Latency may be highery y g• Does not scale well, unless a fat-tree or full-
mesh implemented between virtual switchesmesh implemented between virtual switches
Example: Virtual Switch Architecture
» Source: Extreme Networks» Source: Extreme Networks
Impact to Cabling Design & Topologies
• Critical to understand client switch architecture• Flexibility in design is critical• Flexibility in design is critical• Design needs to be scalable• Pre-terminated 12 or 24fiber MPO/MTP cabling isPre terminated 12 or 24fiber MPO/MTP cabling is
best option in order to support higher bandwidth requirements in data center
• Make sure cabling design maximizes fiber utilization throughout infrastructure migration
10/40/100G…10/40/100G• Make sure there is an easy migration path to support
current or future IEEE 40 & 100G bandwidthcurrent or future IEEE 40 & 100G bandwidth
Impact to Future Needsp• In order to support these newer architectures, IEEE is
working on new physical layer requirementsworking on new physical layer requirements• Next generation 100GbE – 802.3bm Standard
Lower cost– Lower cost – Reduced power– Increased densityIncreased density
• 400G Call for interest scheduled in March 2013– Higher uplink speeds will be needed to support g e up speeds be eeded to suppo t
architectures– Needed for scalability
Summaryy• Bandwidth demand continues to grow
d i lldramatically• Surge in 40G & 100G active equipment
available today and in development• Several new Fabric Architectures• Understand impact of new fabrics to cabling
designsdesigns• End-users…ask for help from experts that
understand the data center environmentunderstand the data center environment
