data center systems’ structured connectivity solutions for ... · recommended cabling solutions...

Recommended cabling solutions for the Brocade DCX 8510 directorspopulated with Brocade FC16-64 high-density port blades

PRODUCT DESIGN GUIDE

Data Center Systems’ Structured Connectivity Solutions for Brocade High-Density Port Blades

53-1004327-0119 July 2016

© 2016, Brocade Communications Systems, Inc. All Rights Reserved.

Brocade, Brocade Assurance, the B-wing symbol, ClearLink, DCX, Fabric OS, HyperEdge, ICX, MLX, MyBrocade, OpenScript, VCS, VDX, Vplane, andVyatta are registered trademarks, and Fabric Vision is a trademark of Brocade Communications Systems, Inc., in the United States and/or in othercountries. Other brands, products, or service names mentioned may be trademarks of others.

Notice: This document is for informational purposes only and does not set forth any warranty, expressed or implied, concerning any equipment,equipment feature, or service offered or to be offered by Brocade. Brocade reserves the right to make changes to this document at any time, withoutnotice, and assumes no responsibility for its use. This informational document describes features that may not be currently available. Contact a Brocadesales office for information on feature and product availability. Export of technical data contained in this document may require an export license from theUnited States government.

The authors and Brocade Communications Systems, Inc. assume no liability or responsibility to any person or entity with respect to the accuracy of thisdocument or any loss, cost, liability, or damages arising from the information contained herein or the computer programs that accompany it.

The product described by this document may contain open source software covered by the GNU General Public License or other open source licenseagreements. To find out which open source software is included in Brocade products, view the licensing terms applicable to the open source software, andobtain a copy of the programming source code, please visit http://www.brocade.com/support/oscd.

Data Center Systems’ Structured Connectivity Solutions for Brocade High-Density Port Blades2 53-1004327-01

http://www.brocade.com/support/oscd

ContentsPreface.......................................................................................................................................................................................................................................................................................... 5

Overview............................................................................................................................................................................................................................................................................5Purpose of This Document.....................................................................................................................................................................................................................................6Audience............................................................................................................................................................................................................................................................................ 6Objectives..........................................................................................................................................................................................................................................................................6Terminology......................................................................................................................................................................................................................................................................7About Brocade................................................................................................................................................................................................................................................................7About DCS........................................................................................................................................................................................................................................................................ 7

Planning........................................................................................................................................................................................................................................................................................9Challenges with Unstructured High-Density Solutions.......................................................................................................................................................................... 9Using a Structured Approach................................................................................................................................................................................................................................ 9

Section 6.5.1 Introduction............................................................................................................................................................................................................................. 10Section 6.5.2 Guidelines...............................................................................................................................................................................................................................10Cabling High-Density, High-Port-Count Fiber Equipment..................................................................................................................................................... 10

Cabling Standards.........................................................................................................................................................................................................................................................11Establishing a Naming Scheme............................................................................................................................................................................................................................11Data Center Systems—Structured Connectivity Solution...................................................................................................................................................................... 11DCS 64-Channel FC16-64 Mimic Adapter Panels.................................................................................................................................................................................14

Installing DCX 8510 Cables............................................................................................................................................................................................................................................. 15

Summary.................................................................................................................................................................................................................................................................................... 17

Appendix A: Equipment List............................................................................................................................................................................................................................................19Director Chassis .......................................................................................................................................................................................................................................................... 19Patch Panels ................................................................................................................................................................................................................................................................. 19Customer Director Trunks...................................................................................................................................................................................................................................... 19Velcro Cable Straps.................................................................................................................................................................................................................................................... 19Labelers.............................................................................................................................................................................................................................................................................19

Appendix B: Supported Distance by Connection Type ..................................................................................................................................................................................21

Appendix C: Cable to Port Mapping and Diagrams........................................................................................................................................................................................ 23MTP/MPO to LC Port Mapping Diagram.................................................................................................................................................................................................. 25MTP/MPO to MTP/MPO Port Mapping Diagram............................................................................................................................................................................... 26

Appendix D: Reference Materials................................................................................................................................................................................................................................27

Data Center Systems’ Structured Connectivity Solutions for Brocade High-Density Port Blades53-1004327-01 3

Preface∙ Overview....................................................................................................................................................................................................................................5∙ Purpose of This Document............................................................................................................................................................................................ 6∙ Audience....................................................................................................................................................................................................................................6∙ Objectives.................................................................................................................................................................................................................................6∙ Terminology..............................................................................................................................................................................................................................7∙ About Brocade....................................................................................................................................................................................................................... 7∙ About DCS................................................................................................................................................................................................................................7

Brocade's FC16-64 high-density port blade for the DCX 8510 Backbone family supports a large number of device ports with asimplified cable plant. The FC16-64's QSFP optics condense connectivity from 64 cables on each blade down to 16 cables, significantlyreducing high-density port blade cable management. This document provides customers who deploy FC16-64 high-density port bladeswith guidelines, industry standards, and best-in-class structured connectivity solutions available from Data Center Systems (DCS).

OverviewThe Brocade

® FC16-64 high-density 64-port Fibre Channel blade combines port density, performance, scalability, and reliability to

maximize the benefits of SAN and server consolidation. The FC16-64 enables medium to large enterprise customers to deploy high-density modular chassis-based solutions that minimize the physical footprint without compromising performance.

High-density Fibre Channel port blades increase chassis density by 33 percent over a chassis populated with 48-port blades, enablingthe DCX 8510-4 to scale up to 256 ports and the DCX 8510-8 to scale up to 512 ports with 16-Gbps performance.

FIGURE 1 Brocade DCX 8510-4 with Four FC 48-Port Blades with an FC16-64 Blade to the Right


FIGURE 2 Brocade DCX 8510-8 with Eight FC 48-Port Blades

To reduce investment cost, energy consumption, and cabling requirements, Brocade uses a space-efficient, 4-channel QSFP (QuadSmall Form-factor Pluggable) optic on the FC16-64 blade, enabling high-density port configurations as well as improved serviceabilityand simplicity of use. These QSFPs retain all the performance and functionality of the standard SFP+ and still support individual, per-port LED indicators for easy troubleshooting and diagnostics.

Purpose of This DocumentThis document provides customers who are deploying FC16-64 high-density port blades with general guidelines for proper optical fibercable management using a low-loss, centralized structured cabling solution.

AudienceThis guide is for IT architects and SAN administrators who are directly responsible for SAN design and/or infrastructure managementbased on Brocade Gen 5 Fibre Channel SAN FC16-64 blades.

ObjectivesThis document aims to provide the best cable-plant deployment practices and enhanced port-connection management, leading tofacilitated MACs (moves, adds, and changes) and avoidance of QSPF implementation challenges. While this document is not intendedto be a definitive connectivity design document, it introduces concepts and guidelines to help you avoid potential issues that can resultfrom poor QSFP cable implementation practices.

This guide describes the following:

∙ Overview of the FC16-64 high-density FC port blade and QSFP optics.

∙ High-density cable management with true structured connectivity solutions designed for low overall link loss and connectorloss.

Preface


∙ Best-practice guidelines and recommendations for optical fiber cabling.

∙ Part numbers for optical cables, patch panels, and enclosures.

TerminologyBelow are some commonly used terms that you will find throughout this guide.

Term Description

LC Lucent Coupler supporting single-channel SFP, SFP+, and XFP transceivers.

MPO/MTP Industry term for Multi-fiber Push On connector; MTP is a trademarked name of an MPO connector with designenhancements to improve mechanical and optical performance. MTP is often used synonymously with MPO.

Note: The terms MPO and MTP may be used in this document in combination or interchangeably, but they are understood tobe compatible designs.

Patch cord Single or multiple strands of fiber cables used for connectivity.

QSFP cable A fiber cable that supports up to four independent serial channels.

QSFP transceiver A Quad Small Form-factor Pluggable (QSFP) transceiver supports up to four channels or ports from a single optic.

RU Rack Unit (4.4 centimeters/1.75 inches).

About BrocadeBrocade® (NASDAQ: BRCD) networking solutions help the world's leading organizations transition smoothly to a world whereapplications and information reside anywhere. This vision is designed to deliver key business benefits such as unmatched simplicity,non-stop networking, application optimization, and investment protection.

Innovative Ethernet and storage networking solutions for data center, campus, and service provider networks help reduce complexity andcost while enabling virtualization and cloud computing to increase business agility.

To help ensure a complete solution, Brocade partners with world-class IT companies and provides comprehensive education, support,and professional services offerings (www.brocade.com).

About DCSData Center Systems (DCS) designs, manufactures, and installs unique products and solutions specifically developed for the physicallayer of connectivity for enterprise data center environments. DCS solutions address our customers' specific needs incorporating expertunderstanding of enterprise protocols, equipment, and connectivity. Attention to current and future industry standards ensure next-generation technology preparedness paired with outstanding documentation and management necessary to future-proof theseincreasingly complex environments.

Many of the world's largest energy, financial services, telecommunications, and retail organizations partner with DCS to design, manage,and evolve mission-critical data center connectivity infrastructure.

Preface


http://www.brocade.com

Preface


Planning∙ Challenges with Unstructured High-Density Solutions..................................................................................................................................9∙ Using a Structured Approach........................................................................................................................................................................................9∙ Cabling Standards................................................................................................................................................................................................................ 11∙ Establishing a Naming Scheme................................................................................................................................................................................... 11∙ Data Center Systems—Structured Connectivity Solution............................................................................................................................. 11∙ DCS 64-Channel FC16-64 Mimic Adapter Panels........................................................................................................................................ 14

As port density per director and per rack increases, an appropriate cable management plan is key during servicing or scaling storagefabrics and it eases troubleshooting. The cable management plan should include current and future SAN design requirements to ensurecompliance with INCITS specifications for link loss. Cables can be managed in a variety of ways, such as routing cables below thechassis, through cable channels on the sides of the cabinet, or using patch panels. When planning a cable management solution and thecable routing path, consider the location of the airflow, the rack's power strip, and the DCX 8510 power supplies to eliminate cableinterference when servicing the power supplies and cords.

The cable management plan may involve installation in a new data center or upgrading the cabling in an existing data center:

∙ If an existing data center is being upgraded, evaluate, capture, and understand the present cabling infrastructure thoroughly.

∙ Document the current (if any) and projected network topologies using an application such as Microsoft Visio or Excel. Focus onthe physical aspects, especially equipment interfaces. Document the various cable types and counts (present, proposed, andprojected) and the approximate routed distances to distribution areas and equipment (present and anticipated equipment portcounts). Additionally, document any areas of concern and any established internal cabling standards.

∙ Plan to accommodate for current and future growth. Build in flexibility, so the central patching location (CPL) will allow a deviceto connect to any other device in the data center. This will permit devices to be located anywhere within the data center andallows for adding scale on a greatly accelerated basis.

Challenges with Unstructured High-Density SolutionsAll three growth factors—volume, performance, and distance—place enormous strain on IT organizations, requiring miles of cableinfrastructure to interconnect servers, storage, and Fibre Channel fabrics for fast, reliable data and application delivery. Unfortunately,many organizations still rely on traditional point-to-point cable solutions, reactively deploying cables one at a time to suit immediateneeds.

The resulting cable clutter inhibits intelligent, pragmatic growth, contributing to an inefficient growth strategy that will only worsen overtime. The tasks of verifying proper connectivity, troubleshooting, and managing device change also become more complex and time-consuming, and they can lead to planned or unplanned downtime of critical business applications.

This inefficient approach also contributes to overheating of data centers—particularly within raised flooring and around the racks wherecable clutter primarily occurs—requiring additional resources to cool the systems.

Using a Structured ApproachCable management solutions designed specifically for Brocade SAN infrastructures utilizing the Brocade DCX 8510 Backbone familyenable a reliable, flexible, and highly efficient cable infrastructure throughout the data center.

As far back as the mid 1990s, shortly after IBM introduced the first fiber-attached mainframe, IBM also introduced the first structuredcabling system for data centers known as the Fiber Transport System (FTS).


In 2005, this concept was adopted into TIA 942 Data Center Standards. Specifically, TIA 942 Section 6.5 addressed centralized opticalfiber cabling, mandating the administration of moves, adds, and changes be performed at the centralized cross-connect. The standardalso established design criteria for central patching and zone distribution. The following sections introduce this standard and associatedguidelines.

FIGURE 3 Cable Clutter

Section 6.5.1 IntroductionCentralized cabling provides connections from equipment distribution areas to a central cross-connect by allowing the use of pull-through cables, interconnects, or splicing in the horizontal distribution area.

Section 6.5.2 GuidelinesThe specifications of ANSI/TIA/EIA-568-B.1 shall be followed except the pull-through cable length shall be less than or equal to 300 m(984 ft), and, thus, the maximum horizontal cabling distance shall not exceed 300 m (984 ft) when a pull-through cable is used.Centralized cabling implementations shall be located within the same building as the equipment distribution areas served. The executionof moves, adds, and changes shall be performed at the centralized cross-connect.

While this concept is not new, it has become increasingly important as data center density and complexity grow at unprecedented rates.

Depending on specific requirements, organizations can choose from various structured fiber-optic cable management solutions. Bymoving from traditional low-density, duplex patch cord cable solutions to high-density, structured cable solutions, organizations canimplement the physical layer in a much more manageable and flexible manner while streamlining data center re-configurations.Structured cabling methodologies are also more energy efficient, promoting improved air flow; they are more cost efficient, allowing forlower operational expense associated with MAC activity; and they help organizations consolidate IT infrastructures without sacrificingmanageability.

Cabling High-Density, High-Port-Count Fiber EquipmentAs networking equipment becomes denser and port counts increase into the hundreds and thousands, managing cables connected tothese devices becomes a difficult challenge. Traditionally, connecting cables directly to individual ports on low-port-count equipment

Planning


was considered manageable. Applying the same principles to high-port-count equipment made the task more tedious, eventuallybecoming nearly impossible to add or remove cables connected directly to the equipment ports.

Structured cabling uses optical fiber connector housings connected through permanent links of optical cabling, typically configured in aphysical star topology from the various areas within the data center (Storage, Servers, SAN, and Network). Utilizing pre-terminatedcabling from each of these areas to a central patching location provides infrastructure where any port from any device can be connectedto any other port by simply running a jumper from one patch panel to another. This is demonstrated in Figure 4 in this document.

Cabling StandardsIndustry cabling standards are designed to protect the end user and provide a firm foundation for establishing a coherent infrastructure;and they set guidelines for maintaining high levels of cable performance. Cabling standards define cabling specifications that look out tothe next several years, thus supporting future needs for higher-speed transmissions. Standards enable vendors to use common media,connectors, test methodologies, and topologies, and they allow planners to design a cabling layout in the data center without worryingabout compatibility issues.

There are a number of standards organizations and standards. The best-known cabling standards are listed below.

Data Centers Specific Standards

∙ United States: ANSI/TIA-942 Telecommunications Infrastructure Standard for Data Centers

∙ Europe: CENELEC EN 50173-5 Information Technology - Generic Cabling Systems - Part 5: Data Centres

∙ International: ISO/IEC 24764 Information Technology - Generic Cabling Systems for Data Centres

General Commercial Building Cabling Standards

∙ United States: ANSI/TIA-568 Generic Telecommunications Cabling for Customer Premises

∙ Europe: EN 50173-1 Performance Requirements of Generic Cabling Schemes

∙ International: CSA ISO/IEC 11801:2009 Information Technology: Generic Cabling for Customer Premises

Cabling Administration Standards

∙ United States: ANSI/TIA-606 Administration Standard for the Commercial Telecommunications Infrastructure

Note: Cabling standards are reviewed and changed every five to ten years, which allows them to keep pace with technology advancesand future requirements. Standards may be purchased online from IHS at http://global.ihs.com/.

Establishing a Naming SchemeAfter the logical and physical layouts for the cabling are defined, apply logical naming that will uniquely and easily identify each cablingcomponent. Effective labeling promotes better communications and eliminates confusion when someone is trying to locate acomponent. Labeling is a key part of the process and should not be skipped.

Once the naming scheme is approved, start labeling the components. Be sure to create a reference document that will become part ofthe training for new data center administrators.

Data Center Systems—Structured Connectivity SolutionAdherence to industry guidelines related to link loss is crucial to avoid bit-rate errors or potential system failure. Per Appendix B,"Support Connection Type Distance," a total of 1.5 dB of connector loss is the maximum allowed per industry specifications. This appliesto a maximum distance of 100 meters running over OM3 MM fiber and 125 meters over OM4 MM fiber.

Planning


HTTP://GLOBAL.IHS.COM/

Cabling infrastructure, in particular director connectivity and patching, can be one of the most confusing tasks for a data center manager.Uptime on a director is imperative; therefore, you should not have to waste any time trying to discern the point of connectivity from onepiece of equipment to the other during MACs. By implementing a recommended industry-standard TIA-942 structured cablinginfrastructure, all patching is facilitated at the central patching location (CPL). The following figure illustrates how to create a moremanageable cabling solution, which simplifies the port identification on the patch panels within the CPL.

Planning


FIGURE 4 Data Center Systems—Centralized Structured Connectivity Solution—Core Edge Topology

Planning


DCS 64-Channel FC16-64 Mimic Adapter PanelsThe 64-Channel FC16-64 Mimic Adapter Panel available through Data Center Systems (DCS) is a 10U, modular adapter panel thatsupports 64 LC connections distributed in 16 groups, each containing 4 channels. Segments are numbered with an overlay to mapprecisely with QSFP ports and FC channels on the face of the Brocade FC16-64 blades. Located at the CPL, 16-64 utilizes a DCS 10U,8-Slot Modular Patch Panel Enclosure populated with eight DCS 64-Channel FC16-64 Mimic Adapter Panels; this solution provides a"mimic" of a fully populated DCX 8510-8 director chassis. Introducing the mimic at the CPL improves manageability and mitigates riskassociated with all MACs by taking management of up to 512 ports away from the active director. Converting from the 16 QSFP ports onthe face of each FC16-64 blade to LC connectors required at the CPL can be accomplished in the following ways:

∙ Implementing the structured connectivity solution shown in Figure 4 above provides the least amount of mated pair insertionloss.

∙ Quantity (2) DCS 32-channel, 96 fiber OM4 Plenum trunks terminated with (16) MTP/MPO connectors at the FC16-64 bladeand (32) LC-Duplex connectors at the back of the DCS 64-Channel FC16-64 Mimic Adapter Panel at the central patchinglocation.

∙ Quantity (1) DCS 64-Channel FC16-64 Mimic Adapter Panel.

See the cable-port mapping diagram within Appendix C.

To further enhance management, DCS offers Mimic Adapter Panels with color schemes to distinguish between A and B fabrics as wellas backup.

FIGURE 5 Data Center Systems 64-Channel FC16-64 Mimic Panel for DCX 8510

Planning


Installing DCX 8510 Cables1. For a DCX 8510-8 chassis with vertically mounted blades, start cabling from the bottom QSFP port group (e.g., ports 0-3) and

work up the blade to the top QSFP port (60-63). In this way, cables can be installed on top of the waterfall as the cables dropdown, rather than trying to work below the cascading cables. Similarly, for a DCX 8510-4 with horizontally mounted blades, startat the right and work to the left for cabling that will be routed on the right side.

2. Bundle the cables using Velcro cable wraps in groups of eight to match the ASIC or trunk boundaries (0-31, 32-63). This willfacilitate servicing of the system through easy identification of the cable path.

3. Work up to the top port.

4. Connect each cable to an MPO/MTP patch panel port using the numbering schema defined in Appendix C.

5. Route the cables down to the bottom of the DCX 8510 chassis (8-slot) and then below toward the back of the cabinet and tothe cable management area.


Installing DCX 8510 Cables


SummaryAlthough cabling represents less than 10 percent of the overall data center network investment, its relevance in terms of systemperformance, availability, and longevity should be considered at the same level of active equipment such as switch and storagecomponents. When designing cabling infrastructure, consider its ability to enhance manageability, to optimize system performance, andto comply with industry cabling standards for link loss. Evaluation criteria should not only include the initial implementation costs, butsubsequent costs as well.

Choose the strongest foundation to support your present and future network technology needs. Implement a structured connectivitysolution that centralizes management, allows for a flexible footprint, and accommodates your unique requirements. Build in additionalcapacity, since it is much easier to install now than later. Use higher bandwidth grades of cabling to postpone having to re-cable astechnologies advance. Cabling itself calls for the right knowledge, tools, patience, and discipline. Without discipline, it is common to seecomplex cabling "masterpieces" quickly get out of control, leading to increased support costs, risk of downtime, and poor, inaccuratedocumentation.

Because each environment is different, there is no explicit solution that will meet all your cable management needs. Following theguidelines and best practices highlighted in this document provides you with the knowledge required to successfully deploy cablinginfrastructure in your data center.


Summary


Appendix A: Equipment List

Director ChassisChassis Description

Brocade DCX 8510-8 Built for large enterprise networks, the 14U Brocade DCX 8510-8 has 8 vertical blade slots to support up to512 16-Gb Fibre Channel ports.

Brocade DCX 8510-4 Built for mid-size networks or edge connectivity in larger networks, the 8U Brocade DCX 8510-4 has4 horizontal blade slots to support up to 256 16-Gb Fibre Channel ports.

Patch PanelsVendor Type Rack Unit Number of

PortsPart Number

Data CenterSystems

Enclosure: 10U x 8 Slot x 15.5" D ECO Modular Enclosure 1-4,9-12 Overlay

10U Up to 512 7510-0101-010

MTP/MPO-LC Panel: 10U-8 64-Channel Black LC-QuadECO Modular Adapter Panel FC16-64 QSFP 0-15/FC 00-63

10U 64 7110-0118-000

Modular Cassette: 10U-8 64-Channel Black LC-Quad OM4ECO Modular Cassette FC16-64 QSFP 0-15/FC 00-63

10U 64 7310-0105-000

Customer Director TrunksVendor Type Rack Unit Number of

PortsPart Number

Data CenterSystems

Trunk: 64 Fiber DCX FC16-64 0-31 OM4 Plenum Trunk LC-MTPf xxx Feet

N/A 32 TB1964F5-3236-XXXF

Trunk: 64 Fiber DCX FC16-64 32-63 OM4 Plenum Trunk LC-MTPf xxx Feet

N/A 32 TB2064F5-3236-XXXF

Velcro Cable StrapsUse Velcro instead of plastic zip ties or metal tie wraps, which cause sheathing and overstress on patch cables leading to signal loss andlow performance. Velcro cable ties come in a roll or in predetermined lengths. Bundle groups of relevant cables with Velcro cable ties asyou install the cables, which will help you identify cables later and facilitate better overall cable management.

LabelersLabelers are used to print sticky labels for devices and cables. Here are some considerations when you choose a hand-held labeler:

∙ Should be capable of operating using batteries.

∙ Able to print labels on smooth, textured, flat, and curved surfaces.

∙ Label material should resist solvents, chemicals, and moisture.

∙ Labels are durable and resist fading.


∙ Adhesive should be long-lasting.

If you choose a labeler with bundled software, install it on a client workstation. You can then customize labels, print labels in batches, andstore the formats for future printing.

Appendix A: Equipment List


Appendix B: Supported Distance byConnection Type

Connection Type Speed Multi-Mode Maximum Distance Comments

OM3 OM4

QSFP<>QSFP

(1.5 dB of connector loss)

16 Gb 100 m 125 m Assumes 1.5 dB for all mated pair loss.

QSFP<>SFP

(1.5 dB of connector loss)

8 Gb 150 m 190 m Assumes 1.5 dB for all mated pair loss.


Appendix B: Supported Distance by Connection Type


Appendix C: Cable to Port Mapping andDiagrams


Appendix C: Cable to Port Mapping and Diagrams


MTP/MPO to LC Port Mapping Diagram



MTP/MPO to MTP/MPO Port Mapping Diagram



Appendix D: Reference Materials∙ TIA-568-C.0—Generic Telecommunications Cabling for Customer Premises

∙ TIA-568-C.3—Optical Fiber Cabling Components Standard

Data Center Systems

Phone: 1-972-620-4997

Email: [email protected]


data center systems’ structured connectivity solutions for ... · recommended cabling solutions...

Documents