Position PaperThe new data center edge — Horizontal Stacking and Switch Clustering

Data center requirements

The enterprise data center is one of the

most critical areas of the network. This

is especially true when consolidating

applications and services into a single

or geo-redundant data center construct.

The challenge to reduce overall costs

is realized as servers, applications and

network services are moved into central-

ized data centers. This consolidation

of critical resources places a greater

burden on network availability and

performance.

24 x 7 access to applications and data

stores is a common requirement in most

enterprises. Any disruption in network

service within the data center has reper-

cussions throughout the business. These

effects are different in nature, scope and

severity depending on the type of enter-

prise, but the effects are all negative and

likely impact the enterprise’s revenue.

The availability of applications begins

with the availability of the network.

The ability to nearly eliminate both

planned and unplanned network

downtime is an important factor when

designing the next-generation data center.

The need for continuous availability

makes it nearly impossible to schedule

any outages for network upgrades or

routine maintenance. A data center

architecture that provides the capability

to schedule maintenance without inter-

rupting the availability of the data center

network gives the network administrator

increased flexibility to maintain the

network.

The other major factor for design

consideration is Total Cost of Ownership

(TCO). A data center solution must be

cost-effective without sacrificing perfor-

mance or resiliency while still paying

attention to the need to be as eco-friendly

(Green IT) as possible. TCO is often

measured in two areas — capital expendi-

tures (CAPEX) and operational expendi-

tures (OPEX).

The initial cost to purchase the equip-

ment and services is considered CAPEX,

while all other costs to maintain the

network fall into OPEX. Over the

lifetime of the network, OPEX costs

can largely outweigh CAPEX. This can

be especially true when looking at the

requirements for maintenance contracts

and the energy costs associated with

operating the equipment. The ongoing

operation and management of the

network are OPEX costs that aren’t as

easily visible. If the network is simple to

operate and manage, then the amount of

time network administrators spend on

these tasks is reduced, thus freeing them

up to do other work and increasing their

overall productivity.

The Server Access Layer evolution

The data center Server Access Layer

continues to evolve. This is a direct

result of the types and numbers of servers

being deployed in the enterprise data

centers. When there were fewer physical

servers, it made sense to consolidate the

Server Access Layer into a few strategic

locations within the data center and

run cabling out to the servers. This

allowed for the best usage of ports on

the network equipment and therefore

the most economical solution.

As the trend moved away from large

servers to single purpose 1U rack-

mounted devices, the amount of network

connections increased dramatically.

Many of these servers came equipped

with multiple Network Interface

Cards (NICs), and server administra-

tors were taking advantage of multiple

network connections. As an example, a

common practice separates the produc-

tion traffic from the data backup traffic

across different network connections.

This increased number of connections

required many more cable runs from

the servers to the network equipment.

This initiated the trend to distribute the

network equipment out to the server

racks as opposed to running all those

cables to a centralized network rack.

Today, as the Ethernet switches are

distributed to the server racks, it has

become common practice to dual

home each switch back to the data

center Distribution Layer or Core

Layer. The availability requirements

mandate a solution that supports the

best utilization of bandwidth with

sub-second failover yet without adding

more complexity. This is the ultimate

resiliency model of Nortel’s Switch

Clustering technology discussed later.

Data centers continue to expand due

to consolidation efforts and increases

in the sheer number of applications

deployed within the enterprise. This

increases the number of server racks

and the number of Ethernet switches

deployed within the Server Access Layer.

A significantly increased number of

ports are required in the Distribution/

Core to collect all the Server Access

Switch connections. In an effort to curb

this sometimes uncontrolled growth

and to better utilize compute resources,

application virtualization and Blade

Center technology have become increas-

ingly popular. This is definitely a step in

the right direction; however, it is not the

total answer.

2

At the same time, as this increase in

the number of servers is happening,

so is the need for high-speed, low-

latency connectivity between servers.

Many applications and services require

extensive communications between the

web, application and data base tiers

in order to service end-user requests.

These applications may be latency-

sensitive and intolerable to any network

delay. High-speed computing and grid

computing environments also require

ultra-fast server-to-server connectivity.

A better solution for inter-server

connectivity is required as millisecond

latency between servers is no longer

sufficient; it must be reduced to micro-

seconds. This can be accomplished with

Nortel’s hardware-based forwarding and

stacking technology as described next.

Figure 1. Each server rack with two uplinks to the core (Gigabit or 10 Gigabit)

No inter-rack stacking

2Gbps or2 x 10Gbps

uplinksbandwidthper rack

($$$)

To core To core To core To core To core To core To core To core

3

Stacking technology

Nortel introduced one of the industry’s

first true resilient stacking technolo-

gies in 1998 with the BayStack 450

Ethernet Switch. This resilient stacking

architecture allowed a stack to be

managed as a single entity and continue

to operate as normal even in the event

of a failure of one of the units. Since

that time, Nortel’s Flexible Advanced

Stack (FAST) architecture technology

has matured considerably. This matu-

rity is seen in functionality with many

additional features such as:

•Abilitytoaddorremoveswitchesinthe stack on the fly without any resets or reboots required

•AutoUnitReplacementperformsconfiguration of the new unit and image upgrade/downgrade to match the existing stack

•StackMonitorprovidesawarningifthe size of the stack changes

•StackLoopbacktestallowsthenetwork administrator to remotely test stack port and stack cable integrity

•StackCountersprovideinformationon traffic flowing across the stack ports

Along with feature enhancements,

there has also been an appreciable

increase in stacking bandwidth. A 2.5

Gigabit stack was sufficient to handle

10/100 ports (BayStack 450), but with

the addition of 10/100/1000 and 10

Gigabit ports, there came the need to

increase the stacking bandwidth, which

now stands at 144 Gigabits per switch

and up to 1.152 Terabits per stack of

eight(EthernetRoutingSwitch5600).

This increase in bandwidth becomes

critical for server-to-server communica-

tions within a single stack as well as

supporting shared access to common

uplinks.

The common deployment of stackable

switches is vertically stacked within a

single rack. These switches are normally

installed in wiring closets as edge access

ports for users. The stack is created by

cabling one switch into another and then

completing the stack with a return cable

connecting the last switch in the stack to

the first switch in the stack. This stacking

technology not only provides high band-

width between switches, but also provides

a single management interface, so all are

managed together as a single logical unit.

Horizontal Stacking

By leveraging Nortel’s resilient stacking

architecture with its FAST technology,

Horizontal Stacking provides a unique

solution to solve today’s challenges in

the data center Server Access Layer.

The concept simply takes the “normal”

verticalstackingoftheEthernetRouting

Switches and cables the switches in a

horizontal orientation. This now allows

switches across multiple racks to be

connected as a stack. No special soft-

ware or configurations are required to

take advantage of this unique physical

solution. Nortel offers various lengths of

stacking cables — 42cm, 1m, 3m and

5m — providing the needed flexibility

to connect the switches across multiple

server racks.

The physical configuration of the hori-

zontal stack is extremely flexible. Figure

3 depicts one permutation in which

there is a single switch installed in each

rack. However, if needed, multiple

switches can be installed in the same

rack and horizontally stacked with other

switches in adjacent racks. This can be

thought of as combining both vertical

and horizontal stacking simultaneously.

Figure 2. Ethernet Routing

Switch 5600 Stack

A Horizontal Stack can range in

size from two switches up to eight

switches. Switches can be added to the

stacks without affecting existing stack

operations. This allows for pay-as-you-

grow functionality and no need for

network downtime to add capacity.

The Horizontal Stack can handle up to

38410/100/1000and1610Gigabit

connectionsusingtheEthernetRouting

Switch5650TD.AnyoftheEthernet

RoutingSwitch5000seriescanbe

stacked together as required to meet the

port count and interface requirements

in the individual server racks.

Once the Horizontal Stack has been

cabled up correctly, with every switch

having two stacking connections, it is

recommended to renumber the units

in the stack. The base unit should be

the leftmost or rightmost unit in the

stack and identified as unit #1. Moving

sequentially across the racks either

left to right or right to left, number

the units in the stack as #2, #3, #4,

etc. Having the Horizontal Stack in

sequential order makes it much easier

for ongoing network management of

the stack. This ensures that configura-

tion changes are made to the appro-

priate switch in the appropriate rack.

In the event there is uncertainty about

the unit numbers in the stack, the

EthernetRoutingSwitchesprovidea

feature to identify their unit number by

illuminating port LEDs on the front of

the switch that correlate to their unit

number(i.e.,unitnumber6willillumi-

nateports1through6LEDs).

By stacking the Ethernet switches, there

is a reduction in the number of uplinks

required to the Distribution/Core Layer.

The number of uplinks and the speed

of those uplinks (Gigabit or 10 Gigabit)

are dependent on the amount of traffic

expected to traverse the uplinks. There

is no absolute formula for number of

servers to uplink bandwidth as this is

dependent on types of applications and

amount of data flow expected.

In order to help characterize data flow

and to assist in troubleshooting, Nortel

supports IPFIX (IP Flow Information

eXport) which is the IETF standard for

flow management based on NetFlow v9.

Offering these capabilities at the Server

Access Layer allows the network admin-

istrator to understand flows to and from

users as well as between servers. IPFIX

can easily be used for real-time traffic

monitoring, troubleshooting, security

and trending network traffic for capacity

analysis. Nortel’s IP Flow Manager is

the network management software that

provides the collection and analysis of

the IP flow data.

IPFIX can help identify when there is a

need to add uplink capacity. This proac-

tive approach alleviates the uncertainty

of uplink utilization and also mitigates

the need for any outage to simply add

uplink bandwidth when required.

Nortel supports up to eight physical

links within a single trunk group.

Providing high-bandwidth and low-

latency connectivity between servers

highlights another major advantage in

stacking the Ethernet switches across

theracks.TheEthernetRoutingSwitch

5600supports144Gbpsstacking

bandwidth per switch and up to

1.152Tbps of bandwidth in a stack of

eight units. Using these capabilities

allows a high-bandwidth pipe between

servers on the same stack. The Ethernet

RoutingSwitch5600alsosupportsthe

Nortel FAST technology which uses a

shortest path algorithm across the stack

for the most efficient use of that stack

bandwidth.

The latency for server-to-server

communication can be reduced from

milliseconds to microseconds when

utilizing Nortel’s stacking technology.

The hardware-based forwarding and

unique stacking algorithm support a 9µs

average port-to-port latency within a

stack. This order of magnitude improve-

ment can be critical to time-sensitive

applications and is near mandatory

when working in high-speed computing

and grid computing environments.

Another performance benefit realized by

stacking is better resource utilization of

the Ethernet switches. The more traffic

that stays local to the stack suggests

there is less traffic that must traverse

the uplinks. In many cases, traffic from

a server must traverse the uplinks and

4

Figure 3. Extend a stack up to eight racks wide with scalable capacity

Extend a stack up to 8 racks — total stack distance up to 35 meters

160 GbEuplinks

bandwidth

Multilink Trunking into Switch Cluster Core

Figure 4. Resiliency all the way to the server interface

Square or Full Mesh SMLT into Switch Cluster Core

Extend a stack up to 8 racks wide

Up to 320Gbuplink

bandwidth

IST

core just to have it come right back to

the server in the next cabinet. This is a

waste of bandwidth on the uplinks and

degrades overall server performance. It

is common with many applications that

there is significant data flow between

servers, and therefore advantageous

to engineer the server connectivity to

remain within a Horizontal Stack.

Horizontal Stacking with Switch Clustering

Nortel’s Switch Clustering technology

was introduced in 2001 and offers

a simplified alternative to Spanning

Tree. Simply aggregate two like Nortel

EthernetRoutingSwitchesorstacks

(EthernetRoutingSwitches8600,

8300,1600,5000)withanInter-Switch

Trunk (IST). This virtualizes the two

Switch Cluster Cores and allows them

to act as a single logical entity for all

devices that are dual attached. No ports

or links are blocked and there are no

proprietary protocols required from

the Switch Cluster Core to the edge

devices. Therefore, Switch Clustering is

totally interoperable with all third-party

devices that support any form of link

aggregation.

The technology supports an active/

active model for uplinks and sub-second

failover, and eliminates the need for

Spanning Tree, which in turn signifi-

cantly reduces the complexity of the

network. Likewise, Switch Clustering

supports the same active/active connec-

tivity for servers. This now extends the

network resiliency from the data center

Core all the way to the server NIC. Any

server that supports link aggregation can

take full advantage of Nortel’s Switch

Clustering technology. This active/active

configuration provides added band-

width to servers and added resiliency

with sub-second failover, making it the

perfect architecture to support virtual-

ized applications with VmWare and

Microsoft’s Hyper-V technology.

TheNortelEthernetRoutingSwitch

5600supportsSwitchClustering

with an Advanced Software License.

By combining the resilient stacking

architecture with the resiliency of

Switch Clustering using Split Multi-

Link Trunking (SMLT) or Single Link

Trunking (SLT), the overall data center

design provides the highest level of

resiliency for all dual connected devices.

The ideal design includes installing a

switch from each Horizontal Stack in a

server cabinet/rack. This would allow

dual connections from the server to

the Ethernet switches inside a rack —

making cable installation, maintenance

and troubleshooting much easier as all

network connections are contained in

the server rack.

Combining these two technologies

provides several key advantages in the

data center — the most important

being the ability to dual connect the

servers into network infrastructure in

such a way as to allow for any planned

or unplanned network outages without

interrupting server access. Switch

Clustering provides sub-second failover

and therefore any outage of a switch or

stack will not affect traffic to and from

5

Horizontal Stacking Horizontal Stacking with Switch Clustering

Scalability from 640Gbps to 1.152Tbps Scalability from 640Gbps to 1.152Tbps

Potential single point of failure (stack) No single point of failure

Requires RPSU (ERS 5500) No RPSU (redundant power from rack)

Limited MLT scaling SLT scales to all ports in the stack

Very cost-effective and energy-efficient Very cost-effective and energy-efficient

6

the server. Outages due to hardware

failure happen, but are very rare. What

is more common is the need to perform

network maintenance on a regular basis.

Switch Clustering provides the answer

to the needed resiliency in either case

as there will be no outages to applica-

tions, since the network connectivity

will always be maintained.

Switch Clustering allows the servers to

be dual connected to the network in

an active/active model, which increases

availability and increases available

network bandwidth to the server. There

is no proprietary software required on

the servers to take advantage of Nortel’s

Switch Clustering technology. Simply

configure the servers for link aggregation.

Terminology used by the various server

vendors varies for this functionality, but

all support some form whether called

NIC teaming, static 802.3ad, Multi-

Link Trunking, Sun Trunking, etc.). For

network bandwidth-intensive applica-

tions, Switch Clustering can support up

to eight physical links and speeds of 10,

100, 1000 and 10 Gigabit. It is required

that all physical links be of the same

speed and duplex setting. For details on

NIC teaming and Switch Clustering

configurations, please refer to the Data

Center Server Access Solution Guide

(NN48500-577).

The connectivity between the Horizontal

Stack Switch Cluster and the data center

Switch Cluster Core is comprised of a

Square or Full Mesh Split Multi-Link

Trunk topology. These topologies ensure

resiliency between the Server Access

Layer and the data center Core Layer

along with being scalable for the number

of uplinks and amount of bandwidth

required. Figure 5 shows the logical view

of the Horizontal Stack Switch Cluster

with server connectivity and Square

SMLT to the Distribution/Core.

Price, performance and the importance of Green IT

TheNortelEthernetRoutingSwitch

5600offerstop-of-the-lineperformance,

at a reasonable price. When comparing

the switching capacity, stacking band-

width and feature set, the Ethernet

RoutingSwitch5000familystandsout

as a prime offering. Its unique ability

to combine switching performance

with features such as Switch Clustering

putstheEthernetRoutingSwitch5000

family in a class by itself.

TheadditionoftheERS5600adds

significant features and capabilities,

especially when deployed in the Data

Center. By supporting built-in N+1

power redundancy that is hot swap-

pable,theERS5600eliminatesthe

need for any external redundant power

supply devices. This saves valuable

space in the rack and more importantly

reduces the overall cost of the solu-

tion while still providing full power

redundancy. The advanced chipsets of

theERS5600alsoprovideadditional

features such as many-to-many port

mirroring, hardware that is capable of

IPv6routingandintegratedIPS-Lite

functionality with its Denial of Service

(DoS) Attack Prevention Package

(DAPP).

The advantages don’t stop there. In

an effort to help reduce OPEX costs

specifically around network operations,

theEthernetRoutingSwitch5000

delivers a common industry de facto

CLI (command line interface) along

with TACACS+, making it easy to learn

and integrate into any network. Nortel

now offers industry-leading warranty

services for the portfolio of Stackable

EthernetRoutingSwitches—theERS

2500,ERS4500andtheERS5000

Series product lines. The complimentary

next-business-day shipment of failed

hardware is now extended to run for

the full life of the product. Nortel also

offers complimentary basic technical

support (Level 1) for the supported

lifecycle of the product. This includes

the software version shipped with or

updatedthroughoptionalSRSBasic,

or a separate Service Contract over and

above our normal Warranty.

TheEthernetRoutingSwitch5000

series is also making a critical impact

with regard to Green IT. When

compared to other industry leaders, the

EthernetRoutingSwitch5000family

provides significantly lower power

consumption (whether sitting at idle

or loaded up to 100 percent). This

lower power consumption translates

into energy cost savings to power the

device. Another benefit is lower BTU

output, which in turn reduces the

amount of cooling required in the data

center, which also translates into energy

cost savings. Last, but not least, is the

carbon footprint and once again, the

EthernetRoutingSwitch5000family

leads the way with a much lower value

than comparable switches from other

Distribution/Core Switch

Cluster

HorizontalStackSwitchCluster

Server withNIC team

SMLT

SMLT/SLT

Figure 5. Logical view

7

vendors1. For more information on

Nortel Energy Efficiency, please visit

www.nortel.com/startsaving and test out

the Nortel Energy Efficiency Calculator

tool to see how much savings are avail-

able with Nortel solutions.

Conclusion

The Nortel data center architecture

provides significant value to enter-

prise customers. A key piece of this

strategy is the Server Access Layer and

the use of Horizontal Stacking and

Switch Clustering. Horizontal Stacking

provides:

•Highbandwidthandlowlatencyfor server-to-server communication within the stack

•Reducesthenumberofportsrequiredin the data center Core — no need for an uplink per switch

•SimplifiescablingbetweenserverandEthernet switch — keeps all cabling within the server rack

•Easilyaddcapacity(switches,uplinks)on the fly without any network downtime

And when combined with Switch Clustering provides:

•Dual-homed,active/activeserverconnectivity — eliminates the network as a potential point of failure

•Increasednetworkbandwidthtoservers — multiple Gigabit or 10 Gigabit connections

•Simplenetworkdesign—noneedfor spanning tree or Layer 3 load balancing of NICs

•Abilitytoperformnetworkmainte-nance (i.e., switch software upgrades) with zero impact to server availability

1 TheTollyGroupReportNumber208298,July2008,reportedtheresultsofapowerconsumptionevaluationofsixNortelconvergeddatanetworkproductsconsistingoflargeandmediumcoreandwiringclosetEthernetRoutingSwitches, enterprise branch office routers and IP phones compared to similar Cisco products on the basis of power consumption, heat dissipation and five-year operational cost. See also The Tolly Group report Number 208275, January2008,andInfoTechResearchGroupReport,January30th2008.

Whether used separately or combined

together, these features provide added

benefits of performance, resiliency,

Green IT and lower overall TCO as

compared to any other vendor’s

offering. Please refer to the Cambridge

University Case Study posted on

the Nortel website at:

http://www.nortel.com/corporate/

global/emea/cstudies/collateral/

nn123206emea.pdf.

BUSINESS MADE SIMPLE

In the United States:

Nortel

35 Davis Drive

Research Triangle Park, NC 27709 USA

In Canada:

Nortel

195 The West Mall

Toronto, Ontario M9C 5K1 Canada

In Caribbean and Latin America:

Nortel

1500 Concorde Terrace

Sunrise, FL 33323 USA

In Europe:

Nortel

Maidenhead Office Park, Westacott Way

Maidenhead Berkshire SL6 3QH, UK

Email: euroinfo@nortel.com

In Asia:

Nortel

United Square

101 Thomson Road

Singapore 307591

Phone: (65) 6287 2877

Nortel is a recognized leader in delivering communications capabilities that make the promise of Business Made Simple a reality for our customers. Our next-generation technologies, for both service provider and enterprise networks, support multimedia and business-critical appli-cations. Nortel’s technologies are designed to help eliminate today’s barriers to efficiency, speed and performance by simplifying networks and connecting people to the information they need, when they need it. Nortel does business in more than 150 countries around the world. For more information, visit Nortel on the Web at www.nortel.com. For the latest Nortel news, visit www.nortel.com/news.

Formoreinformation,contactyourNortelrepresentative,orcall1-800-4NORTELor 1-800-466-7835fromanywhereinNorthAmerica.

Nortel, the Nortel logo, Nortel Business Made Simple and the Globemark are trademarks of Nortel Networks. All other trademarks are the property of their owners.

Copyright © 2009 Nortel Networks. All rights reserved. Information in this document is subject to change without notice. Nortel assumes no responsibility for any errors that may appear in this document.

NN123961-071509