www.brocade.com

Data Center ethernet Fabrics and Brocade VCS technology: networking for the Virtualized Data Center

WHITE PAPER

As data centers evolve to a world where information and applications can move anywhere in the cloud, classic Ethernet with its hierarchical architecture also has to evolve to a new data center “Ethernet fabric.”

2

Data center networks rely on ethernet. Over the decades, ethernet has evolved as new application architectures emerged. today, data center networks carry traffic for a diverse set of applications including client/server, Web services, and unified communications—each with different traffic patterns and network service requirements. applications are increasingly deployed within virtual machines hosted on server clusters. and ethernet is used to build shared storage pools, which places stringent demands on the network, demands for lossless packet delivery, deterministic latency, and high bandwidth. all together, these changes are the forces behind the next evolutionary step in ethernet: the ethernet fabric.

The ClassiC eTherneT neTwork in order to better understand the Ethernet fabric, consider first a classic Ethernet network. Data centers need more ports than are available in a single Ethernet switch, so multiple switches are connected to form a network with increased connectivity. For example, server racks often include a switch at the Top of the Rack (ToR) or servers in several racks connect to a Middle of Row (MoR) or End of Row (EoR) switch. All these Ethernet switches are connected, forming a hierarchical, or “Ethernet tree” topology, as shown in Figure 1.

limitations of Classic ethernetIn a classic Ethernet network, the connections between switches, or Inter-Switch Links (ISLs, shown as solid blue lines in Figure 1), are not allowed to form a loop or frames aren’t delivered. Spanning Tree Protocol (STP) prevents loops by creating a tree topology with only one active path between any two switches. (In Figure 1, inactive paths are shown as dotted lines.) This means that ISL bandwidth is limited to a single connection, since multiple paths between switches are prohibited. The tree topology requires traffic to move up and down the tree, or “north-south,” to get to an adjacent rack. When most of the access traffic is between servers in a rack, this is not a problem. But server clusters, such as those required for server virtualization, have traffic between servers in multiple racks, travelling “east-west,” so the tree topology increases latency with multiple hops and restricts bandwidth with single links between switches.

3

STP automatically recovers when a link is lost. However, it halts all traffic through the network and must reconfigure the single path between all switches in the network before allowing traffic to flow again. Halting all traffic for tens of seconds up to minutes on all links limits scalability and constrains traffic to applications that can tolerate data path blocking to achieve link resiliency. In the past, traffic relied on TCP to handle this interruption in service, but today, with almost all data center applications running in a 24 x 7 high availability mode and storage traffic growing on the Ethernet network, loss of connectivity in the data path for even a few seconds is unacceptable.

Finally, the classic Ethernet switch architecture presents other limitations. Each switch has its own control and management planes. Each switch has to discover and process the protocol of each frame as it arrives on an ingress port. As more switches are added, protocol processing time increases adding latency. Each switch and each port in the switch has to be configured individually, since there is no sharing of common configuration and policy information between switches. Complexity increases, configuration mistakes increase, and the total cost of operations and management does not scale well.

enhancements to ethernetAn additional Ethernet standard, Link Aggregation Groups (LAGs), was defined so that multiple links between switches were treated as a single connection without forming loops. But a LAG must be manually configured on each port in the LAG and is not very flexible. Flatter networks with self-aggregating ISL connections eliminate manual configuration.

Spanning tree has been modified to support multiple logical trees in a single physical tree (MSTP) and to reduce convergence time when links are added or removed (RSTP). But the topology is still a hierarchical tree in which convergence is measured in seconds.

Switch stacking was introduced so that multiple switches could be managed as one logical switch. But topologies are restricted, master switches and special ISL ports are required, and ISL oversubscription increases as the stack grows. Stacking does provide a single point of management for multiple switches, but adds complexity, restricts topology choices, and limits scalability.

Core

Server rack

ISLs

Inactivelinks

Acce

ssAg

greg

atio

n

Traf

fic fl

ow

Figure 1. Classic Ethernet network.

Properties of Classic ethernet networks

Control plane on the switch, policy • configuration and protocol processing required for every port.

Data path interconnect relies on STP • to prevent loops.

Link aggregation on interconnects • is manually configured on multiple switches.

Management is at the switch level, • limiting scalability.

Topology is restricted to a hierarchical • tree.

Scalability is limited by single-path • interconnects

Link resiliency is automatic but • disruptive to all traffic flows in the network.

Adding network services requires • careful placement to avoid Layer 2 bottlenecks.

4

The eTherneT FabriC arChiTeCTureFigure 2 shows the architecture of a classic Ethernet switch. The control, data, and management planes are logically connected to every port via a back plane. Control and management planes operate at the switch level not a network level.

Ethernet fabrics can be thought of as extending the control and management planes beyond the physical switch into the fabric. As shown in Figure 3, they now operate at a fabric level rather than at a switch level.

Control and management become scalable, that is, distributed services integrated into the network rather than restricted by the switch chassis, which provides a number of benefits. For example, a fabric scales automatically when another fabric-enabled switch is added. The new switch logically “plugs” into the distributed control plane, allowing control plane state and configuration parameters to be shared across all switches and ports in the fabric. Since the control plane is distributed, protocol packet processing is done once at the edge port, reducing overall packet latency; and policy and security settings are created once, ensuring consistent policy and security configuration across all switches in the fabric.

When a switch is added, it automatically joins a logical chassis, similar to adding a port card to a chassis switch. This simplifies management, monitoring, and operations since policy and security configuration parameters are automatically inherited by the new switch. Configuration and management are performed once for the fabric, not multiple times for each switch and switch port. In addition, information about device connections to servers and storage is known to all switches in the fabric, enabling Automated Migration of Port Profiles (AMPP) within the fabric. When virtual machines or servers are moved to another fabric port, AMPP ensures that all network policies and security settings continue to be applied to its traffic without reconfiguring the entire network.

The control path replaces STP with link state routing, while the data path provides equal-cost multipath forwarding at Layer 2 so data always takes the shortest path using multiple ISL connections without loops. Combined with the distributed control plane, scaling bandwidth is made simple. For example, when a new switch connects to any other switch in the fabric, ISLs come up automatically. When multiple ISLs connect two switches, trunks automatically

Control plane

Management plane

Data plane

Ethernetfabric

Self-

aggr

egat

ing,

sca

labl

e, fl

at

Figure 3. Ethernet fabric

architecture.

Control plane

Management plane

Data plane

Ethernetswitches

Figure 2. Ethernet switch

architecture.

switch Primary Functions: PlanesSwitches are designed using three primary functions, or planes, called the control, management, and data planes, each providing specific services. The control plane is responsible for setting up the data plane forwarding tables that record the optimal path over which to send a frame. The control plane also decides if an incoming frame should be dropped based on security settings and what QoS level to assign to the frame based on network policies. The data plane is responsible for moving frames between the ingress and egress ports as quickly as possible. The management plane is responsible for managing the switch including switch and port configuration, monitoring and sending alerts to the management console.

5

form with frame-based load balancing and automatic link failover. If a trunk link fails or is removed, traffic is rebalanced on the existing links non-disruptively. Finally, if an ISL is added or removed anywhere in the fabric, traffic on other ISLs continues to flow instead of halting as with STP.

In summary, the Ethernet fabric architecture requires:

A control plane distributed across all switches so that packet processing is performed once • and common attributes (policies and security settings) are configured once

ISLs that support any topology (ring, mesh, or core/edge)•

Equal-cost multipath routing at Layer 2 replacing STP on ISLs, removing loops, • automatically scaling bandwidth, and ensuring shortest path is always used in the fabric

Flatter network so that traffic flows scale out, have low latency, and are not congested•

Automatic link resiliency that is non-disruptive to traffic flowing on any other link•

Automatic link aggregation, with trunks delivering line-rate load balancing across all links • and automatic link failover within a trunk

A logical management architecture that is at a fabric level so that adding a new switch • does not require configuration of parameters as they are inherited and switches are managed like port cards in a chassis

Fabric support for traffic redirection to simplify deployment of network services such as • security, Layer 4–7 application control, and native Fibre Channel services

broCade VCs arChiTeCTureBrocade is the first vendor to deliver products that meet the all the requirements of a true Ethernet fabric. Brocade© VCS™ architecture redesigns traditional Ethernet networks, removing the limitations of classic Ethernet. The Brocade VCS architecture is ideally suited for both private and public cloud computing deployments.

Figure 4 shows the key capabilities of the Brocade VCS technology—capabilities that are responsible for implementing an Ethernet fabric:

VCS Ethernet fabric in the data plane•

VCS Distributed Intelligence in the control plane •

VCS Logical Chassis in the management plane•

Dynamic Services

EthernetFabric

LogicalChassis

DistributedIntelligence

Brocade VCS Technology

No STP

Multi-path, deterministic

Auto-healing, non-disruptive

Lossless, low latency

Convergence ready

Logically flattens and collapses network layers

Scale edge and manage as if single switch

Auto-configuration

Centralized or distributed management, end-to-end

Self-forming

Arbitrary topology

Network aware of allmembers, devices, VMs

Masterless control, noreconfiguration

VAL interaction

Connectivity over distance, Native Fibre Channel, Security Services, Layer 4 - 7, and so on

Figure 4. Brocade VCS architecture.

ethernet FabricsCompared to classic hierarchical Ethernet architectures, Ethernet fabrics provide higher levels of performance, utilization, availability and simplicity. They have the following characteristics:

Flatter. • Ethernet fabrics are self-aggregating, enabling a flatter network.

Intelligent. • Switches in the fabric know about each other and all connected devices.

Scalable. • All paths are available for high performance and high reliability.

Efficient. • Traffic automatically travels along the shortest path.

Simple. • The fabric is managed as a single logical entity.

6

In addition, Brocade VCS Dynamic Services support simple scaling of network services such as security, Layer 4–7 application delivery control, extended Layer 2 networks between data centers, and native Fibre Channel storage services for converged networks. Combined, these VCS capabilities transform classic Ethernet networks into Ethernet fabrics.

broCade VCs imPlemenTaTion oF an eTherneT FabriCFigure 5 shows a logical model of how VCS technology implements an Ethernet fabric.

VCs distributed intelligenceInstead of independent control planes for each switch, Brocade VCS Distributed Intelligence service uses a distributed control plane so that protocol processing occurs once at an edge port and common configuration parameters are set once and available to every switch in the fabric. For example, when a new switch is added to the fabric, it logically plugs into the distributed control plane, adding additional control path processing for scalability. At the fabric edge, packet processing occurs on any port of any switch, but is not required on every ISL port, greatly reducing latency and making configuration simple and scalable. Policies can be configured once and are available to all switch ports. With Automated Migration of Port Profiles (AMPP), virtual servers can migrate between switch ports without risking conflicts with network policies and security settings at edge ports; these settings automatically follow the MAC address of the virtual machine.

VCs logical ChassisInstead of managing each switch and its ports independently, the Brocade VCS Logical Chassis service creates a virtual management plane. A Logical Chassis has common configuration parameters for policies and can bind MAC address to policies with AMPP and apply them to all switches in the fabric. The Logical Chassis service creates a single point of management for monitoring all switches, ports and traffic in the fabric. Adding a new switch is as simple as adding a new port blade to a chassis switch. The switch and its ports are uniquely identified, share common configuration parameters, and are monitored from a single management connection. With the Logical Chassis service, management can scale with the network, since as adding switches and ports does not add complexity.

VCs ethernet FabricThe Brocade VCS Ethernet fabric provides equal-cost multipath forwarding, simplifies interconnect configuration, provides automatic link failover with no interruption to traffic on unaffected links, and provides plug-and-play fabric scalability. Instead of manually configuring LAG on individual ports on multiple switches, Brocade VCS Ethernet fabric lets the data

Core

Distributed Intelligence

Logical ChassisFigure 5.

Brocade VCS control and management

plane architecture.

7

plane automatically form trunks when multiple ISL connections are added between switches. Simply adding another cable increases Layer 2 bandwidth, providing linear scalability of server to server and server to storage traffic. ISL connections between switches form automatically, as long as the fabric identifier of the switch is set to the identifier of the fabric to which it’s connected. The data plane topology is user configurable; it is not dictated by the underlying technology nor the limitations inherent in STP and hierarchical Ethernet tree topologies. The flexibility of this topology allows easy specification of oversubscription ratios. For example, High Performance Computing (HPC) workloads may require 1:1 subscription, virtual servers 4:1, and client/server 10:1 or higher.

The broCade VdX 6720 Family oF daTa CenTer swiTChesThe introduction of the Brocade VDX™ 6720 family of 10 GbE-capable fabric switches delivers a new category of Ethernet switching, the Ethernet fabric switch. The Brocade VDX 6720 includes Brocade VCS architecture, offering revolutionary Ethernet fabrics, Distributed Intelligence, and Logical Chassis technology for the data center.

The Brocade VDX 6720 is available in 24- or 60-port rack-mount form factors and can be deployed in 16-, 24-, 40-, 50-, or 60-port configurations with Brocade exclusive Ports-on-Demand (POD) licensing. At launch, the Brocade VDX family interoperates with existing Ethernet switching products, providing an evolutionary path from legacy Ethernet networks to Ethernet fabrics. Existing legacy switches do not have to be replaced until the end of their useful life; Brocade VDX 6720 switches can be added at the top of rack or middle or end of row forming a scalable, simple-to-manage Ethernet fabric.

For example, several Brocade VDX 6720 switches deployed in a ToR configuration create a single Logical Chassis with a single distributed control plane across multiple racks of servers, delivering compelling reductions in capital and operating costs, while simplifying virtual machine migration. Automated Migration of Port Profiles (AMPP) is hypervisor agnostic, which is an important feature, as most data centers deploy different hypervisor stacks based on application and server requirements.

And, since the Brocade VDX 6720 family supports the emerging Ethernet Data Center Bridging (DCB), TRILL, and Fiber Channel over Ethernet (FCoE) standards, the Ethernet fabric is lossless, low latency, and convergence ready.

learn moreBrocade networking solutions help the world’s leading organizations transition smoothly to a virtualized world where applications and information reside anywhere. This approach is based on the Brocade One™ unified network strategy, which enables a wide range of consolidation, convergence, virtualization, and cloud computing initiatives

Learn more at www.brocade.com.

Figure 6. Brocade VDX 6720-24 Data Center Switch (top) and Brocade VDX 6720-60 Data Center Switch (bottom).

www.brocade.com

© 2011 Brocade Communications Systems, Inc. All Rights Reserved. 01/10 GA-WP-1551-01

Brocade, the B-wing symbol, BigIron, DCFM, DCX, Fabric OS, FastIron, IronView, NetIron, SAN Health, ServerIron, TurboIron, and Wingspan are registered trademarks, and Brocade Assurance, Brocade NET Health, Brocade One, Extraordinary Networks, MyBrocade, VCS, and VDX are trademarks of Brocade Communications Systems, Inc., in the United States and/or in other countries. Other brands, products, or service names mentioned are or may be trademarks or service marks of their respective owners.

Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment, equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, without notice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocade sales office for information on feature and product availability. Export of technical data contained in this document may require an export license from the United States government.

Corporate headquarters San Jose, CA USAT: +1-408-333-8000info@brocade.com

european headquarters Geneva, SwitzerlandT: +41-22-799-56-40 emea-info@brocade.com

Asia Pacific Headquarters SingaporeT: +65-6538-4700 apac-info@brocade.com

WHITE PAPER