plugin-user%27s guide - qlogic fc hba in an emc environment

B e s t P r a c t i c e s

QLogic FC HBA in an EMC Environment

Best Practices

About QLogic

QLogic is a leading supplier of high-performance storage networking solutions, producing the controller chips, host bus adapters (HBas) and

fabric switches that are the backbone of storage networks for most Global 2000 corporations. the company delivers a broad and diverse portfolio

of products that includes Fibre channel HBas, blade server embedded Fibre channel switches, Fibre channel stackable switches, iscsi HBas,

iscsi routers and storage services platforms for enabling advanced storage management applications. the company is also a leading supplier of

infiniBand switches and infiniBand host channel adapters for the emerging High Performance computing cluster (HPcc) market. QLogic products

are delivered to small-to-medium businesses and large enterprises around the world via its channel partner community. QLogic products are also

powering solutions from leading companies like cisco, Dell, eMc, Hitachi Data systems, HP, iBM, Nec, Network appliance and sun Microsystems.

QLogic is a member of the s&P 500 index. For more information go to http://www.qlogic.com.

recent accolades include:

- s&P 500 index

- Forbes’ Best 200 Best small companies

- Fortune’s 100 Fastest Growing companies

- Business Week Hot Growth company

- Network computing Well connected

- 2005 storage Magazine “saN Product of the Year”

- 2005 saNbox 5602 infostor and asNP “Most Valuable Product Finalist”

QLogic FC HBA in an EMC Environment JULY 2007

corporate Headquarters QLogic corporation 26650 aliso Viejo Parkway aliso Viejo, ca 92656 949.389.6000

europe Headquarters QLogic (UK) LtD. surrey technology centre 40 Occam road Guildford surrey GU2 7YG UK +44 (0)1483 295825

Fc0054602-00 rev a 7/07

QLogic FC H

BA in an EMC Envi ron m

ent

JULY 2007

JULY 2007QLogic Press V1

Best Practices GuideQLogic FC HBA in an EMC Environment

Version 1.0

AbstractThis guide discusses the best practices for planning, deploying,

maintaining, and performance tuning a QLogic 4Gb Fibre Channel HBA in a SAN connected to EMC storage arrays.

Copyright © 2003-2007 QLogic Corporation. All rights reserved.

THE INFORMATION PROVIDED IN THIS DOCUMENT IS PROVIDED “AS IS” WITHOUT WARRANTY OF ANY KIND, INCLUDING ANY WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, INTEROPERABILITY, OR COMPATIBILITY. All of the Partners' products are warranted in accordance with the agreements under which the warranty for the products are provided. Unless otherwise specified, the product manufacturer, supplier, or publisher of non-Partner products provides warranty, service, and support directly to you. THE PARTNERS MAKE NO REPRESENTATIONS OR WARRANTIES REGARDING THE PARTNERS’ PRODUCTS OR NON-PARTNER PRODUCTS, AND NO WARRANTY IS PROVIDED FOR EITHER THE FUNCTIONALITY OR PROBLEM RESOLUTION OF ANY PRODUCTS.

The inclusion of the Partners' products on an interoperability list is not a guarantee that they will work with the other designated storage products. In addition, not all software and hardware combinations created from compatible components will necessarily function properly together. The following list includes products developed or distributed by companies other than the Partners. The Partners do not provide service or support for the non-Partner products listed, but do not prohibit these products from being used together with Partner storage products. During problem debug and resolution, the Partners may require that hardware or software additions be removed from products to provide problem determination and resolution on the supplied hardware/software. For support issues regarding non-Partner products, please contact the manufacturer of the product directly.

This information could include technical inaccuracies or typographical errors. The Partners do not assume any liability for damages caused by such errors as this information is provided “AS IS” for convenience only; readers use this information at their own risk and should confirm any information contained herein with the associated vendor. Changes are periodically made to the content of this document. These changes will be incorporated in new editions of the document. The Partners may make improvements and/or changes in the product(s) and/or the program(s) described in this document at any time without notice.

Any references in this information to non-Partner websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this interoperability guide, and the use of those websites is at your own risk. Information concerning non-Partner products was obtained from the suppliers of those products, their published announcements, or other publicly available sources. The Partners have not tested those products and cannot confirm the accuracy of performance, compatibility, or any other claims related to those products. Questions about the capabilities of non-Partner products should be addressed to the suppliers of those products.

All statements regarding the Partners' future direction or intent are subject to change or withdrawal without notice and represent goals and objectives only. This information is for planning purposes only; any use of the information contained herein is at the user's sole risk. The information herein is subject to change before the products described become available.

QLogic, the QLogic logo, QLogic Press, the QLogic Press logo, Powered by QLogic, QLA, SANblade, SAN Pro, SANsurfer, SANsurfer Management Suite, and SANtrack are trademarks or registered trademarks of QLogic Corporation in the United States, other countries, or both. EMC and PowerPath are trademarks or registered trademarks of EMC Corporation in the United States, other countries, or both. Microsoft and Windows are trademarks or registered trademarks of Microsoft Corporation in the United States, other countries, or both. Other company, product, and service names may be trademarks or service marks of others.

The QLogic home page on the Internet can be found at http://www.qlogic.com.

QLogic Corporation 26650 Aliso Viejo Parkway Aliso Viejo, California 92656 Tel: 949.389.6000 Fax: 949.389.6114

http://www.qlogic.com

Table of ContentsIntroduction............................................................................................................... 9

Audience........................................................................................................................................ 9What are Host Bus Adapter Best Practices?.............................................................................. 9Supported HBAs ......................................................................................................................... 10How this Guide is Organized ..................................................................................................... 10

Statement of Services and Support ...................................................................... 11

SAN Planning and Deployment Best Practices ................................................... 13

Using a 4Gb HBA in a 2Gb FC SAN........................................................................................... 13High Availability Best Practices ................................................................................................ 14

Choosing One or More Single/Dual/Quad Port HBAs ............................................................ 14

High Availability Features of the QLogic HBA ........................................................................ 14

Connecting the SAN for High Availability ............................................................................... 14

Validating Failover Configurations.......................................................................................... 15

Choosing the Right Port on the Switch for Host and Storage Ports ....................................... 16

Considerations for Tape Access via an HBA ........................................................................... 16Check Interoperability of SAN Components ............................................................................ 16

HBA Installation Best Practices ............................................................................ 19

HBA Handling Best Practices .................................................................................................... 19Verifying the HBA Installation.................................................................................................... 19

Use Fast!UTIL to Verify HBA BIOS and Connectivity............................................................. 20

Verify that the Host Can See the HBA.................................................................................... 20

Verify that the Host Can See the LUNs .................................................................................. 21

Latest HBA Driver Versions ....................................................................................................... 21Understand the LED Scheme for QLogic HBAs....................................................................... 22

HBA Configuration Best Practices........................................................................ 23

QLOGIC FC HBA IN AN EMC ENVIRONMENT PAGE 5 VERSION 1.0, JULY 2007

Table of Contents

HBA Performance Tuning Best Practices ............................................................ 25

Introduction ................................................................................................................................. 25Understanding Application Workloads..................................................................................... 25

Transaction Based Processes (IOPs) .................................................................................... 26

Throughput Based Processes (MBps).................................................................................... 26

Differences Between Transaction Based and Throughput Based Workloads........................ 27

Characterization of Common Application Workloads ............................................................. 27

HBA Parameters that Impact HBA Performance...................................................................... 28HBA Performance Tuning in Microsoft Windows .................................................................... 28

Execution Throttle................................................................................................................... 29

HBA Performance Tuning in VMware ESX Server ................................................................... 31HBA Queue Depth.................................................................................................................. 31

Maximum Scatter/Gather List ................................................................................................. 38

Maximum Sectors................................................................................................................... 40

Monitoring Performance............................................................................................................. 40Gathering Host Server Data ................................................................................................... 41

Gathering Fabric Network Data .............................................................................................. 42

Choosing the Optimal PCI Slot for Your HBA .......................................................................... 43General SAN Considerations for Performance ........................................................................ 46

Fencing High Performance Applications ................................................................................ 46

Minimize ISL ........................................................................................................................... 46

Upgrade to 4Gb ...................................................................................................................... 46

Set Data Rate to Auto-negotiate............................................................................................. 46

Fan-In Considerations ............................................................................................................ 46

Avoiding RSCNs..................................................................................................................... 46

Linear Scaling of HBAs........................................................................................................... 47

I/O Load Balancing...................................................................................................................... 47

Fibre Channel Security Best Practices ................................................................. 49

Setting a Password for QLogic SANsurfer FC HBA Manager................................................. 49Generic Fibre Channel SAN Security ........................................................................................ 50

PAGE 6 QLOGIC FC HBA IN AN EMC ENVIRONMENT VERSION 1.0, JULY 2007

Table of Contents

Appendix A – QLogic HBA Tasks and Tools Sheet............................................. 51

Appendix B – QLogic HBA LED Scheme.............................................................. 55

References............................................................................................................... 57

QLogic Press Review ............................................................................................. 59

QLogic Company Information ............................................................................... 61


Table of Contents


Introduction

QLogic 4Gb Fibre Channel (FC) host bus adapters (HBAs) are designed and developed to provide industry leading performance, simplify management, and offer fully interoperable solutions in the most demanding enterprise storage area network (SAN) environments. This guide — QLogic FC HBA in an EMC Environment — presents a compilation of best practices to apply when using QLogic FC HBAs in an EMC® SAN.

AudienceThis guide is intended for SAN architects, IT administrators, and storage system professionals that currently utilize, or are considering utilizing, a QLogic HBA in a Fibre Channel SAN.

What are Host Bus Adapter Best Practices?Best practices are process-oriented steps that are planned and prioritized to facilitate a certain activity. Using best practices helps ensure the highest quality of service (QoS). Best practices also provide guidelines on how to best utilize your HBA in a SAN.

Please note that these practices are only recommendations, not requirements. Not following these recommendations does not affect whether your solution is supported by QLogic or EMC. Not all recommendations apply to every scenario. QLogic believes that their customers will benefit from reviewing these recommendations before making any implementation decision.

As with virtually any best practices recommendation, implementing security best practices inevitably involves some tradeoffs. The best practice recommendations provided in this guide include discussions regarding potential tradeoffs.

Every organization’s requirements for a Fibre Channel SAN are different. The requirements are frequently driven by factors such as the type of business in which the organization engages, the type of data stored in the SAN, the organization’s customer base, etc. All these factors must be carefully evaluated in conjunction with these best practices to develop an appropriate plan for deploying and using a SAN. This evaluation should include specific considerations as to how the data is stored and managed in a Fibre Channel SAN.

This set of best practices enables administrators to quickly and easily enhance the value of existing storage resources with little or no investment.


IntroductionSupported HBAs

Supported HBAsThis guide supports the following QLogic 4Gb FC HBAs used in EMC SAN environments. These QLogic SANblade® HBAs are collectively referred as QLogic HBAs or adapters throughout this guide.

How this Guide is OrganizedThe contents of this guide are described in the following paragraphs.

SAN Planning and Deployment Best Practices (page 13). This section discusses what to consider before implementing a SAN in your organization, including helping you choose the right equipment and making the right interconnections.

HBA Installation Best Practices (page 19). This section discusses best practices that you can implement to verify HBA installation, verify that there are no conflicts, choose the right driver, etc.

HBA Configuration Best Practices (page 23). This section discusses EMC recommended HBA parameter configurations for optimal operation in an EMC environment.

HBA Performance Tuning Best Practices (page 25). This section discusses various HBA driver and firmware parameters that you can modify to enhance the performance of the HBA. This section also discusses how server and slot types can affect HBA performance.

Fibre Channel Security Best Practices (page 49). This section discusses how to implement a secure SAN using the available features of the QLogic HBA.

Appendix A – QLogic HBA Tasks and Tools Sheet (page 51). This appendix lists the various common tasks a SAN Administrator performs on an HBA and the tools that can help complete the tasks.

Appendix B – QLogic HBA LED Scheme (page 55). This appendix defines the blink patterns of the various LEDs on the back of a QLogic HBA.

QLogic SANblade HBA FC Rate Host Bus Number of Ports

QLA®2460-E 4Gb PCI-X 2.0 1

QLA2462-E 4Gb PCI-X 2.0 2

QLE2460-E 4Gb PCI Express 1

QLE2462-E 4Gb PCI Express 2


Statement of Services and Support

The QLogic SAN Pro™ Service and Support Program allows customers to choose from a variety of service plans designed to meet specific business requirements. SAN Pro covers a diverse range of services, including remote technical assistance, on-site support and advanced hardware replacement. Best of all, no matter which plan is selected, customer satisfaction is guaranteed by QLogic and its network of authorized service partners.

QLogic switch products allow a wide range of organizations to exploit the power of a SAN. Whether it's a fast growing small firm implementing a network with 10–20 devices or a Fortune 100 Corporation creating a large infrastructure with thousands of devices, QLogic SANtrack Service and Support Program effectively addresses either set of business requirements.

The SAN Pro Service and Support Program offers a diverse range of services including SAN Pro Prime, SAN Pro Select, and SAN Pro Exchange. Customers may choose among the services that best meet the demands of their business. Most importantly, customers are assured complete satisfaction since QLogic and its qualified partners fully guarantee all products and services.links below.

For additional information on support, please see the QLogic website at: http://support.qlogic.com/support/service_programs.asp.

For warranty information, please visit QLogic at: http://support.qlogic.com/support/warranty.asp.


http://support.qlogic.com/support/service_programs.asp

http://support.qlogic.com/support/warranty.asp

Statement of Services and Support


SAN Planning and Deployment Best Practices

Using a 4Gb HBA in a 2Gb FC SANAs a best practice, use a 4Gb HBA even if the EMC SAN has 2Gb components (switch or storage) for the following reasons:

QLogic 4Gb FC HBA technology is backward compatible with 2Gb and 1Gb infrastructures.

QLogic 4Gb FC HBAs are offered at very competitive pricing compared to 2Gb FC HBAs.

User needs and data volumes continue to grow. Consequently, choosing and deploying a 4Gb FC HBA enables better investment protection to meet future needs.

QLogic 4Gb FC HBAs offer significantly higher performance, enhanced product and data reliability, and allow superior scalability (refer to the white paper Benefits of 4Gb HBA in 2Gb SANs, posted on http://www.qlogic.com/knowledgecenter/briefs_papers.asp).

A summary of the benefits of 4Gb FC HBAs versus 2Gb FC HBAs is described in the table below.

2Gb 4Gb QLogic 4Gb Benefits

Higher Performance

Maximum IOPs/port 40,000 150,000 Higher performance

Intelligent Interleaved DMA No Yes Enhanced Return on Investment (ROI) in mixed speed SANs

Dual Read DMA No Yes Superior performance through faster parallel processing

Out of Order Frame Reassembly No Yes Reduces congestion and improves network performance

Superior Scalability

Virtualization (N_Port Virtual ID) No Yes Higher availability and security

VSAN No Yes Improved QoS and lower TCO

Enhanced Reliability

T10 CRC Support No Yes Provides end-to-end data integrity

Overlapping Protection Domains No Yes Higher data reliability


http://www.qlogic.com/knowledgecenter/briefs_papers.asp

SAN Planning and Deployment Best PracticesHigh Availability Best Practices

High Availability Best PracticesThis section discusses the various High Availability features of the QLogic HBAs and discusses how to choose between single, dual, and quad port HBAs. In addition, the section provides best practices on how to connect the HBA port to a switch to attain High Availability and how to avoid a single point of failure (of an HBA).

Choosing One or More Single/Dual/Quad Port HBAs

Use the following guidelines when deciding how many ports per HBA you need for a server:

To achieve High Availability, there must be more than one physical HBA in a server.

The second best option to multiple HBAs is to have a single HBA with multiple ports. Use this option when server PCI slot space can accommodate only one HBA.

Two single-port HBAs provide better reliability and performance compared to one dual-port HBA.

When using a SAN to back up your array, it is best to have a dedicated third HBA. The third HBA allows the storage HBAs to perform their tasks while providing unprecedented bandwidth to the backup application.

High Availability Features of the QLogic HBA

The QLogic HBAs provide several High Availability features. To best utilize these features, use EMC PowerPath® software. EMC PowerPath is a server-resident solution that works closely with the QLogic HBA to enhance performance and application availability. It combines multiple path I/O capabilities, automatic load balancing, and path Failover functions into one integrated package.

Connecting the SAN for High Availability

It is recommended that all SANs be setup such that a single point of failure (failure of a single component of the SAN) does not disrupt application I/O. Redundant HBAs, switches, storage processors, and fabric should be used with EMC PowerPath to achieve a highly available SAN. The following EMC PowerPath table summarizes the various connectivity options and their benefits. Choose which option you want to deploy in your enterprise.


SAN Planning and Deployment Best PracticesHigh Availability Best Practices

Validating Failover Configurations

Once the host environment has been successfully configured for Failover, including setting up redundant paths and Path Management software, QLogic recommends that the host be tested to determine that Failover and Failback work correctly. This testing should be done in the deployment stage.

The following scenarios should be individually tested while there is active I/O level at the host. In each case, verify that the LUNs fail over to the alternate path. For each of these cases, following a successful Failover, revive/reconnect the failed component and see if Failback works as expected.

Disconnect the fiber cable connecting the HBA to the switch.

Disconnect the fiber cable connecting the switch to the storage system.

Disable or physically remove the HBA from the host.

Disable one of the storage controllers.


SAN Planning and Deployment Best PracticesConsiderations for Tape Access via an HBA

Choosing the Right Port on the Switch for Host and Storage Ports

In addition to validating failover configuration, if the switch being used contains multiple ASICs, QLogic recommends trying to connect host and storage pairs to the same ASIC. This connection prevents using the shared internal data transfer bus and reduces switch latency. In addition to fault tolerance concerns, performance is also a consideration. For example, if a host has two HBAs, each one accessing its own storage port, do not attach both HBAs, both storage ports, or all of the HBA and storage ports to the same switch ASIC.

Considerations for Tape Access via an HBAQLogic recommends that a separate HBA and switch zone be used in the fabric for tape access. The reasons for this recommendation are as follows:

Separate zoning keeps streaming tape traffic segregated from standard disk traffic in the SAN.

Zoning out the tape HBA and the tape port on the switch decreases the impact of tape reset/rewind issues on disk traffic.

Check Interoperability of SAN ComponentsToday’s IT environments incorporate products from many different vendors and sources with the possibility of millions of combinations. To take advantage of this breadth of products and to use the best product for the task at hand — whether servers, storage, operating system, or other components — it is critical to know:

Which products have been rigorously tested with all of the others and are known to work properly.

What configuration information and best practices are available to ensure that these products actually work together.

Where you can go for support of a mixed-vendor installation.

SPA SPB

Array

SPA SPB

Array

HBA HBA

Host

HBA HBA

Host

ASIC 1 ASIC 1FC Switch


SAN Planning and Deployment Best PracticesCheck Interoperability of SAN Components

EMC recommends using the E-Lab Interoperability Navigator to obtain this information. Follow the links from https://elabnavigator.emc.com/.

QLogic also provides a comprehensive SANtrack interoperability guide. This guide is available in electronic or print form from http://www.qlogic.com/interopguide/.


https://elabnavigator.emc.com/

http://www.qlogic.com/interopguide/

SAN Planning and Deployment Best PracticesCheck Interoperability of SAN Components


HBA Installation Best Practices

This section provides a list of best practices that QLogic recommends to ensure that your QLogic HBA is installed correctly, the right drivers are loaded, and that each of these steps are completed without error.

HBA Handling Best PracticesTo minimize the possibility of ESD-related damage, EMC strongly recommends using both a workstation antistatic mat and an ESD wrist strap.

Observe the following precautions to avoid ESD-related problems:

Leave the HBA in its antistatic bag until you are ready to install it in the system.

Always use a properly fitted and grounded wrist strap or other suitable ESD protection when handling the HBA. Also observe proper ESD grounding techniques.

Hold the HBA by the edge of the PCB or mounting bracket, not the connectors.

Place the HBA on a properly grounded antistatic work surface pad when the HBA is out of its protective antistatic bag.

Verifying the HBA InstallationQLogic recommends that every HBA installation be verified to guarantee that the HBA and drivers have been installed successfully and are working properly. After you have installed the QLogic HBA as per EMC guidelines and connected the HBA port to a storage array, follow these instructions to determine if your QLogic FC HBA and its driver have been correctly installed.

EMC recommends using the Customer Procedure Generator software to get step-by-step information about setting up a supported configuration. QLogic provides EMC Installation and Configuration Guides for various operating systems at http://support.qlogic.com/support/oem_emc.asp.

QLogic recommends using these manuals to complement the step-by-step instructions generated by the EMC Customer Procedure Generator.


http://support.qlogic.com/support/oem_emc.asp

HBA Installation Best PracticesVerifying the HBA Installation

Use Fast!UTIL to Verify HBA BIOS and Connectivity

While the host server is booting, the QLogic BIOS banner displays. At this time, press CRTL+Q to enter Fast!UTIL. Use the Fast!UTIL menu options to scan for fibre devices and view serial number and WWPN information. Verify that the HBA can see the EMC Storage Array LUNs.

Detailed instructions on how to use Fast!UTIL can be found at http://www.qlogic.com/ in the EMC section.

Verify that the Host Can See the HBA

Verify that the host can see the QLogic HBA. Do one of the following, depending in your operating system (OS):

In Windows, check that the QLogic HBA is visible in the Windows Device Manager under the SCSI and RAID Controllers section.

In Linux, run /sbin/lspci or /usr/bin/lsdev to verify that the QLogic HBA is among the list of devices.

In Solaris, use the configuration administration command cfgadm to check the status of the fc-fabric occupant.

In VMware, log into the Management User Interface (Virtual Infrastructure Client or Virtual Center), navigate to the Configuration Tab for the host, and then proceed to Storage Adapters. Verify that the QLogic HBA appears in the list of adapters.


http://www.qlogic.com/

HBA Installation Best PracticesLatest HBA Driver Versions

Verify that the Host Can See the LUNs

Verify that the host can see all the LUNs that were presented to the QLogic HBA ports (the ones that were visible at the HBA BIOS level in “Use Fast!UTIL to Verify HBA BIOS and Connectivity” on page 20). Do one of the following, depending in your OS:

In Windows, do one of the following:

Navigate to Computer Management and then Disk Management; verify that the LUNs can be seen as disks.

Use the QLogic SANsurfer Management application to view the LUNs.

In Linux, execute the cat /proc/scsi/scsi command to see a list of LUNs presented to the Linux host via the QLogic HBA. The information displayed by the cat /proc/scsi/scsi command is not dynamic and does not reflect state changes caused by fabric changes. Use the QLogic Linux Tool ql-scan-lun.sh to rescan the Fibre Channel for new LUNs. No reboot is required.

In Solaris, use the command format to see a list of LUNs.

In VMware ESX, log into the Management User Interface (Virtual Infrastructure Client or Virtual Center), navigate to the Configuration tab for the host. From there, go to Storage Adapters and verify that the LUNs can be seen. Try doing a rescan if you think new LUNs may have been added since the host was booted.

Latest HBA Driver VersionsEnsure that the latest HBA driver is installed on all the HBAs in your environment. The latest driver versions can be downloaded from http://support.qlogic.com/support/oem_emc.asp. QLogic drivers follow the unified driver model, where the firmware is bundled along with the driver package; this prevents any possible mismatch between the driver and firmware versions, which in turn reduces the number of software components a SAN administrator has to manage.



HBA Installation Best PracticesUnderstand the LED Scheme for QLogic HBAs

Understand the LED Scheme for QLogic HBAsSee “Appendix B – QLogic HBA LED Scheme” (page 55) to identify the HBA LEDs, and how the LED color and blink patterns indicate HBA status and connected components. Use this information as a starting point in all of your SAN link troubleshooting exercises and after you have successfully installed the QLogic HBA.


HBA Configuration Best Practices

EMC recommends that you use the EMC E-Lab Interop Navigator to obtain EMC approved HBA driver and NVRAM settings for optimal operation. See the following EMC – QLogic published documents to obtain EMC approved HBA settings. The documents can be found at http://powerlink.emc.com/.

NOTE: For most cases, the default HBA parameter settings are also the recommended values. For a detailed explanation of each parameter, refer to the appropriate guide listed below, depending on your operating system.

.


http://powerlink.emc.com/

HBA Configuration Best Practices


HBA Performance Tuning Best Practices

IntroductionSAN performance depends on many factors, including workload, hardware, vendor, HBA types, RAID levels, cache sizes, and stripe sizes.

This section describes performance topics as they apply to the QLogic Fibre Channel HBAs. The HBA is more than a piece of the SAN performance puzzle; it is the internal plumbing of the SAN. SAN performance tuning is not dependent on just the HBA level performance tuning; QLogic recommends that the SAN administrator look into all factors that influence SAN performance. This section also describes the different workloads generated by enterprise applications, their impact on performance, and how performance can be addressed by configuration and parameter modifications.

Managing SAN performance means meeting the demands of current workloads as well as ensuring the ability of applications to scale and handle larger workloads and peak demands. To ensure adequate performance, administrators must be able to monitor and measure system performance. This section describes the tools available to an administrator to measure and analyze performance, and shows how to use the tools effectively.

When tuning a SAN for performance, it is important to understand the system's operation and architectural design from end-to-end, so that you can match components such as I/Os per second (IOPS) and queuing. End-to-end refers to all operations and components of the entire system, from handling an application request to the underlying support software, the physical HBA, the SAN fabric components, the disk array controllers, and the disk drive. Without an understanding of the supporting components, you might make blind changes, or changes to one area of the system without considering all others. These changes not only yield no overall system performance improvement, but may make things worse. This section provides insight into many of these architectural considerations, and offers advice on how to avoid common mistakes.

Recommendations made in this section should enable optimal balance between performance, management, and functionality.

NOTE: QLogic recommends that performance tuning be conducted in a test SAN environment, and that I/O performance is monitored before implementing the changes in a live SAN environment. Performance monitoring and tuning is a continual process.

Understanding Application WorkloadsIn general, there are two types of data workload (processing):

Transaction based

Throughput based

These two workloads are very different, and must be planned for differently. Knowing and understanding how your host servers and applications handle their workload is an important part of being successful with your storage configuration efforts and the resulting performance of the QLogic HBA.


HBA Performance Tuning Best PracticesUnderstanding Application Workloads

Transaction based and throughput based processing are types of workload. The workload is the total amount of work that is performed at the storage server, and is measured through the following formula:

Workload = [transactions (number of host IOPS)] * [throughput (amount of data sent in one I/O)]

Since a storage server can sustain a given maximum workload, the above formula shows that when the number of host transactions increases, the throughput decreases. Conversely, if the host is sending large volumes of data with each I/O, the number of transactions decreases.

A workload characterized by a high number of transactions (IOPS) is called a transaction based workload.

A workload characterized by large I/Os is called throughput based workload.

These two workload types are conflicting in nature, and consequently require different configuration settings across all parts of the storage solution. Generally, I/O (and therefore application) performance is best when the I/O activity is evenly spread across the entire I/O subsystem. On the Windows platform, the I/O transfer size of an application can be determined using Perfmon.

The following sections further describe each workload type.

Transaction Based Processes (IOPs)

High performance in transaction based environments cannot be created with a low cost model (with a small number of physical drives) storage server. Transaction process rates are heavily dependent on the number of back-end drives that are available for the controller to use for parallel processing of the initiator/host generated I/Os.

Generally, transaction intense applications also use a small random data block pattern to transfer data. With this type of data pattern, having more back-end drives enables more host I/Os to be processed simultaneously, as read cache is far less effective, and the misses need to be retrieved from disk.

In many cases, slow transaction performance problems can be traced directly to hot files that cause a bottleneck on a critical component (such as a single physical disk). This situation can occur even when the overall storage server has a fairly light workload.

When bottlenecks occur, they can present a very difficult and frustrating task to resolve. As workload content can continually change throughout the day, these bottlenecks can be very mysterious in nature and appear and disappear, or move from one location to another over time. Generally, I/O (and therefore application) performance is best when the I/O activity is evenly spread across the entire I/O subsystem.

While transaction based workloads reduce block contention, they have a relatively large host CPU overhead.

Throughput Based Processes (MBps)

Throughput based workloads are seen with applications or processes that require large amounts of data to be sent, and generally use large sequential blocks to reduce disk latency. Generally, only a small number of drives (20–28) are needed to reach maximum throughput rates with the EMC storage servers. In this environment, read operations use the cache to stage greater chunks of data at a time, which improves overall performance.


HBA Performance Tuning Best PracticesUnderstanding Application Workloads

Throughput rates are heavily dependent on the storage server’s internal bandwidth. Throughput based workloads have a relatively low host CPU overhead.

Differences Between Transaction Based and Throughput Based Workloads

While no single enterprise application can be classified as having a solely transaction or throughput based workload, a large percentage of an enterprise application workload will be one or the other. Use this large percentage tilt to decide how you want to tune your QLogic HBA for optimum performance.

With the QLogic HBA, these two workload types have different parameter settings to optimize their specific workload environments. These settings are not limited to the storage server; they span the entire solution. With care and consideration, you can create an environment of very good performance with both workload types and share the same QLogic HBA. However, portions of the storage server configuration will be tuned to better serve one workload or the other.

For maximum performance of both workloads, QLogic recommends considering two separate QLogic HBAs, each tuned for their specific workload type, instead of sharing a single HBA to serve both workloads at the same time. However, datacenter budgets may dictate which model you choose.

Characterization of Common Application Workloads

The following table characterizes the most common workloads in an enterprise. This is a generalized characterization; use these definitions as a starting point and refine them based on your environment.

Oracle I/O Characteristics

Table 1. Workload Characterization

Block Size Workload Type Enterprise Application

4, 8, and 16KB Transaction oriented Oracle, Microsoft Exchange & SQL, Lotus Domino, Web Server, File Server

16–32KB Throughput oriented Log files

32–64KB Throughput oriented Backup and recovery; video, and audio streaming

Table 2. Oracle I/O Characteristics

SubsystemI/O Block

Size Access Type I/O Type Workload Type

OLTP – Log Up to 60KB (Sector Aligned)

Sequential Read or Write Primarily throughput

OLTP – Data 8KB Random Read Transactional

OLTP – Lazy Write 8k–256K Random Write Primarily transactional

OLTP – Checkpoint 8k–256K Random Write Primarily transactional

Read ahead (DSS, table/index scans) 8k–256K Sequential Read Primarily throughput

Bulk insert 8k–128K Sequential Write Primarily transactional


HBA Performance Tuning Best PracticesHBA Parameters that Impact HBA Performance

Microsoft Exchange I/O Characteristics

HBA Parameters that Impact HBA PerformanceThe following table lists the programmable HBA driver and HBA parameters (also called NVRAM parameters). Modifying these parameters may impact HBA performance. This section provides a brief description of the parameters, and shows their default values and range. The following sections discuss several of these parameters in more detail.

The following sections focus on tuning HBA performance in Windows and VMware ESX environments.

HBA Performance Tuning in Microsoft WindowsIn Microsoft® Windows® environments, the following three HBA parameters affect HBA performance: Execution Throttle, Frame Size, and Fibre Channel Data Rate. Of these, the Frame Size and Fibre Channel Data Rate default settings are pre-set to 2112 bytes (2048 bytes + headers) and auto-negotiate to provide the best possible performance in any environment. Therefore, Execution Throttle is the only HBA parameter that you can tune to improve HBA performance in a Windows environment.

Table 3. Microsoft Exchange I/O Characteristics

Subsystem I/O Block Size Access Type I/O Type Workload Type

Database Store .EDB 4KB Random Equally split reads and writes Transactional

Database Store .STM 32–64KB Random Equally split reads and writes Throughput

Database Store .LOG 4KB Sequential Write Transactional

HBA Parameter Definition Default Value Range OS Support

Execution Throttle Specifies the maximum number of I/O commands allowed to execute on a HBA port.

256 1–256 Windows

Frame Size Specifies the size of a Fibre Channel frame per I/O. 2048 512–2048 All

Fibre Channel Data Rate

Specifies the HBA adapter data rate. When set to Auto, the adapter auto-negotiates the data rate with the connecting SAN device.

Auto 1 (Auto), 2 (1Gb),3 (2Gb),4 (4Gb)

All

Maximum Queue Depth

Specifies the maximum number of I/O commands allowed to execute/queue on a HBA port.

32 1-65535 VMware ESX

Maximum Scatter Gather List Size

Specifies the size of the list of DMA items that are reported to SCSI mid-level per I/O request.

32 1-255 VMware ESX

Maximum Sectors Specifies the maximum number of disk sectors that are reported to SCSI mid-Level per I/O request.

512 512, 1024, 2048

VMware ESX


HBA Performance Tuning Best PracticesHBA Performance Tuning in Microsoft Windows

Execution Throttle

What is Execution Throttle?Execution throttle is a HBA parameter that controls the maximum number of I/O commands executing on any one HBA port. When a port’s execution throttle is reached, no new commands are executed until the current command finishes executing. The valid options for this setting are 1–256. For Microsoft Windows environments, this parameter is set to the default value of 16 or to an OEM specific value. With this value, at any point in time, one HBA port can have a maximum of 16 I/O commands executing. Any more I/O commands will have to wait before they can be scheduled in the HBA. The value of the Execution Throttle parameter can be increased to allow the HBA to execute more commands or decreased to decrease the amount of I/O an HBA port can put on the wire.

Tuning Execution ThrottleIn a SAN configuration with three or more servers accessing the same storage array, QLogic recommends changing the default Execution Throttle value for each HBA. By default all QLogic EMC 4Gb FC HBAs have their Execution Throttle value set to maximum; if you decide to change this from its default value, use the guidelines below to derive a new value. To calculate the new execution throttle, first determine if all servers carry the same I/O load.

If all servers carry the same I/O load, calculate the value by dividing 250 by the number of servers in the SAN. Set each HBA in the SAN to the calculated value. For example, in a four-server configuration, divide 250 by 4 to arrive at 62.5. The Execution Throttle value for each HBA is 62. Assign the value of 62 to all HBAs.

If some of the servers carry heavier I/O loads, first calculate the Execution Throttle value by dividing 250 by the number of servers, and then adjust the values so that servers with higher I/O loads have higher Execution Throttle values and servers with lower I/O loads have lower execution throttle values. For example, in a four-server configuration, you can assign the value of 72 to the HBAs in the server with the highest I/O load, the value of 52 to the HBAs in the second server, and the value of 62 to the HBAs in the remaining two servers.

When calculating the server Execution Throttle value, consider the following:

EMC recommends setting all HBAs in a server to the same Execution Throttle value.

When adding the values, treat each server as having a single value. If the adapters in the first server are set to 72, the adapters in the second server are set to 52, and the adapters in the third and fourth servers are set to 62, the sum is 248.

In most environments, it is OK to set the value of Execution Throttle to its maximum for all HBAs in the SAN.

In general, increasing the value of the execution throttle will have a more profound increase in performance for transaction-oriented applications as compared to throughput-oriented applications.

The effects of changing the execution throttle of a QLogic HBA are most noticeable when the application workload type is transaction oriented.


HBA Performance Tuning Best PracticesHBA Performance Tuning in Microsoft Windows

How to Change the Execution ThrottleThe Execution Throttle value for each port of an HBA can be easily changed with the QLogic SANsurfer FC HBA Manager application (or the SANsurfer command line interface (CLI)) on Windows environments (see the figure below).

Key Notes on Tuning Execution ThrottleCAUTION: When experimenting with Execution Throttle, consider:

There is a risk of flooding the storage array with I/O from just one HBA port that has a high Execution Throttle. This flooding will neither allow nor delay access to this storage device from other HBA ports. If the Execution Throttle value is too high, attached servers may indicate an I/O timeout issued by the QLogic Fibre Channel device driver in the data section of the Windows Event Viewer System log.

In many cases, the speed of the HBA is limited by speed of the storage array. Do not increase the Execution Throttle to a value such that the I/O overwhelms the storage array.


HBA Performance Tuning Best PracticesHBA Performance Tuning in VMware ESX Server

HBA Performance Tuning in VMware ESX ServerA virtual machine, which is no different than a physical server, hosts an operating system and runs applications. Tuning a physical server is similar to tuning a virtual machine environment, but there are a few considerations and special options. This section describes the QLogic HBA parameters that apply to the VMware ESX Server that can impact the performance of the virtual machines hosted by the server. There are three such HBA parameters:

Maximum Queue Depth

Maximum Sectors

Maximum Scatter Gather Entries

HBA Queue Depth

What is HBA Queue Depth?HBA queue depth is an HBA driver parameter that refers to the number of HBA SCSI command buffers that can be allocated by an HBA port. It relates to the maximum number of outstanding commands that can execute on any one HBA port. When the QLogic driver loads, it registers the value of the queue depth with the SCSI mid-level, which notifies the SCSI layer above the HBA driver how many commands the HBA driver is willing to accept and process. The default for ESX 3.0.0 is 32 and 16 for ESX 2.5.x. The valid queue depth range is from 1–256, but the best value for your environment depends on the following factors:

The total number of LUNs exposed through the storage array (target) ports

The queue depth of the storage array (target) port

The effects of changing the HBA Max Queue Depth parameter are most noticeable when the application workload type is transaction oriented.

Tuning HBA Queue Depth for your EnvironmentWhen determining the right value of HBA queue depth for your environment, consider the effect of the HBA queue depth on other SAN components. This section describes the relationship between the HBA queue depth and the target device queue depth, and discusses whether increasing the HBA queue depth can increase performance without adversely affecting other SAN components.

Parameter Description Denoted in Driver by Default in ESX 2.5.x Default in ESX 3.0.x Range

HBA Max Queue Depth ql2xmaxqdepth 16 32 1–256



I/O Loading FundamentalsThe section presents the fundamentals of I/O loading using EMC Disk Array examples. This model should extend to other devices in the SAN as well. Loading is presented from the target device port loading perspective, with host and HBA factors that affect the storage array’s optimal utilization. The concepts and variables presented are standard SCSI terminology. The terminology may change depending on the host operating system. For example, in Solaris, the QLogic driver queue depth per LUN is the variable sd_max_throttle; in VMware, it is the variable ql2xmaxqdepth.

When determining loading in any mass storage SAN setup, is necessary to understand the factors used; these factors are at the SCSI layer level in both the initiator (host) and the target (device). These factors provide an I/O flow control mechanism between the initiator and the target. The host flow control variable is the queue depth per LUN; the device flow control variable is target port queue depth. The above diagram shows the placement of these variables in the I/O architecture.

The following five main variables assess the loading factor on a device port:

P = The number of host paths connected to that array device port.

q = Queue depth per LUN on the hosts (for the host port); i.e., the maximum I/Os outstanding per LUN from the host at any given time. This is a QLogic FC driver/SCSI driver parameter.

L = The number of LUNs configured on the array device port as seen by a particular host path.

T = The maximum queue depth per array device port, which signifies the maximum number of I/Os outstanding (that can be handled) on that port at a given time. The value for T is typically 2048 for EMC arrays.

Q = Queue depth per HBA port; i.e., the maximum I/Os that an HBA can have outstanding at an instant. This parameter is needed only if q (queue depth per LUN) is not available.



These variables relate to each other through the following equation:

T => P * q * L

where: The target queue depth should be greater than or equal to the product of host paths, HBA queue depth, and the number of LUNs configured.

For heterogeneous hosts connected to the same port on the device, use the following equation:

T => Host OS 1 (P * q * L) + Host OS 2 (P * q * L) + ............ + Host OS n (P * q * L)

Key Considerations

The target port queue depth value (2048 for many EMC arrays) used in the equations above is on a per-port basis; that is, 2048 outstanding I/Os (simultaneous) at a time. Therefore, one EMC controller with four ports can have 2048 x 4 = 8192 outstanding I/Os at a time handled by the EMC controller without any port overloading. Two controllers can have 16384 outstanding I/Os. Check the specifications of your EMC array to get the exact number.

Not abiding by the above equations can result in either under utilizing (T < actual queue depth) or flooding (T > actual queue depth) the target device. Flooding the target device queue results in a QFULL condition. A QFULL is an I/O throttling command that is sent by the Storage Array SCSI layer to an HBA port to notify the port that its I/O processing limit has been reached and it cannot accept any more I/Os until it completes its current set.

In VMware ESX, when a QFULL condition is received from the target, the HBA driver typically decreases the LUN’s maximum queue depth to the minimum value (usually 1). This value throttles I/O to the target port. When the target stops issuing QFULL conditions, the HBA driver starts gradually increasing the LUN queue depth value, thereby slowly increasing I/O to the target port. Excessive QFULL commands will drastically decrease the performance of your SAN connection.

If you suspect flooding, you can enable the Extended Error Logging flag in the HBA parameters to view extended logging and see if a QFULL command is being received by the HBA.

Determining HBA Queue UtilizationUse the VMware ESX tool esxtop to view the current HBA queue utilization while I/O is active on the HBA ports. Navigate your way through esxtop to find disk statistics. A man esxtop command issued from the ESX console provides detailed information on its usage.



The figure below shows the output of the storage statistics section of esxtop while I/O is active. LQLEN shows the current HBA queue depth set in the QLogic HBA driver. A value of LOAD > 1 indicates that the host application is placing more data in the HBA queue than its current size can handle. A system with this issue can benefit from an increase in the HBA queue depth. Note that any increase in the HBA queue depth should be guided by the equation discussed in “I/O Loading Fundamentals” on page 32.

The figure below shows the effect of increasing the HBA queue depth. This result of esxtop has been captured after increasing the HBA queue depth from its default value of 32 to 64. Note that the LOAD is < 1 and there is a significant increase in the READS/s operations, which means that performance has increased.



How to Change the HBA Queue DepthTo change the queue depth of a QLogic HBA in VMware ESX, follow these steps:

1. Log on to the VMware ESX Console as root.

2. Create a copy of /etc/vmware/esx.conf so you have a backup copy.

3. Edit the file /etc/vmware/esx.conf in your favorite editor.

4. Locate the following entry:/vmkmodule[0002]/module = "qla2300_707.o" /vmkmodule[0002]/options = ""

5. Modify the entry as shown, where xx is the queue depth value:/vmkmodule[0002]/module = "qla2300_707.o" /vmkmodule[0002]/options = "ql2xmaxqdepth=xx"

6. Save the file.

7. Reboot the VMware ESX Server.



How to Verify the Queue Depth ChangeThe current QLogic HBA queue depth and any changes to it can be verified by one of the methods described in the following paragraphs.

1. View the entries for the QLogic FC adapter driver in the proc File System.

a. Execute the command cat /proc/scsi/qla2300/x at the VMware ESX console, where x is the port number of the QLogic HBA. The port number is typically 0 for a single port HBA, and 0 and 1 for a dual port HBA.

b. Look for the string Device queue depth. The value assigned to this field is a hexadecimal representation of the currently configured queue depth for the HBA port specified in the command.



2. Look at the VMware kernel log files.

a. View the file /var/log/vmkernelin your favorite editor after logging on to the VMware ESX console.

b. Look for the string Queue depth. The value assigned to this field is a decimal representation of the currently configured queue depth for the specified HBA port.

Key Notes on Tuning the HBA Queue DepthConsider the following when tuning the HBA queue depth:

For QLogic driver version 6.04, the default setting for the HBA queue depth is 16; for versions 6.07 and 7.x (in Virtual Infrastructure 3), the default is 32.

Increasing the queue depth in the QLogic Fibre Channel HBA above 65 has little effect, because the maximum parallel execution (or queue depth) of SCSI operations is 64. Experiment with your SAN setup and determine the value that best suits your environment.

Increasing the HBA queue depth will likely result in an increased number of I/O operations that can be placed in the HBA queue for servicing, consequently increasing the CPU utilization of the host.

Consider the I/O performance of disks or another I/O device that is in between the storage controller port and disks. Array controller I/O queue clearance depends on these devices, which allow outstanding I/Os to be processed from the host side. The number of disks, cache settings, RAID type used, etc., should be considered with EMC arrays and the I/O performance of the QLogic HBA.



Maximum Scatter/Gather List

What is Scatter/Gather List?Scatter/gather is a QLogic driver feature that processes DMA data transfers that are written to noncontiguous areas of memory. A scatter/gather list is a list of vectors, each of which gives the location and length of one segment in the overall read or write request. QLogic Fibre Channel adapters have a DMA controller on each adapter, which is programmed to support scatter/gather operations.

Any I/O operation performed by an application to a disk first reaches a block device driver in the host operating system; the block device driver stores the data contained in the I/O into locations in memory. The data for a single I/O operation processed from an application could span across multiple memory locations. The block device driver passes these memory locations and their length to the HBA driver via the SCSI layers. A scatter/gather list is stored as a table in host kernel memory; the memory has a list of all these memory locations and the length of each such location. The HBA driver then passes this table to the firmware on the physical adapter, which in turn does a DMA operation to retrieve the data.

The QLogic HBA parameter HBA Max SG List defines the size of this table. The larger the table, the more entries it can store and process. The larger the I/O operation, the more scatter/gather entries are needed to store the I/O data.

The following table specifies the VMware ESX specific parameter name, defaults, and range. This HBA driver parameter is not available on Windows operating systems.

Tuning the Maximum Scatter/Gather ListBased on the maximum scatter/gather list concept, conceptually larger size I/O operations (throughput based) will benefit from an increased value of the HBA Max SG List parameter. However, changing the size of the scatter/gather list to a very large value (even for large I/O block sizes) may adversely effect the performance, as a very large scatter/gather list may put a undue load on the HBA firmware, host memory, and the DMA engine. QLogic recommends experimenting by increasing the scatter/gather list for large size I/Os or decreasing the list for small size I/Os, then measuring the performance gain. Do not expect a big performance jump, as the default value for the scatter/gather list is pre-optimized for most application environments. QLogic does not recommend setting the value of the scatter/gather list to its maximum value (255) for any application environment.

Parameter Description Denoted in Driver by Default in ESX 2.5.x Default in ESX 3.0.x Range

HBA Max SG List ql2xmaxsgs 32 32 1–255



How to Change Maximum Scatter/Gather ListTo change the HBA Max SG List parameter of a QLogic HBA in VMware ESX, perform the following steps:

1. Log on to the VMware ESX console as root.

2. Create a copy of /etc/vmware/esx.conf so you have a backup copy.

3. Edit the file /etc/vmware/esx.conf in your favorite editor.

4. Locate the following entry:/vmkmodule[0002]/module = "qla2300_707.o" /vmkmodule[0002]/options = ""

5. Modify as the entry as follows, where xx is the new HBA Max SG List value: /vmkmodule[0002]/module = "qla2300_707.o" /vmkmodule[0002]/options = "ql2xmaxsgs=xx"

If you need to change more than one option, separate them with spaces in the options field. For example:/vmkmodule[0002]/module = "qla2300_707.o" /vmkmodule[0002]/options = "ql2xmaxsgs=xx q12xmaxqdepth=64"

6. Save the file.

7. Reboot the VMware ESX Server.

How to Verify the Max Scatter/Gather List ChangeAny change in the QLogic HBA Max SG List parameter can be verified by looking at the VMware kernel log files, as follows:

View the file /var/log/vmkernel in your favorite editor after logging on to the VMware ESX console.

Look for the string sg_tablesize. The value assigned to this field is a decimal representation of the currently configured size of the scatter/gather list, as shown in the following screenshot.


HBA Performance Tuning Best PracticesMonitoring Performance

Maximum Sectors

This QLogic HBA driver parameter defines the maximum number of disk sectors for a LUN that are reported by the HBA driver to the SCSI mid-level. The value of this parameter is used by the block device driver to obtain a DMA transaction size for each I/O operation.

The following table specifies the HBA Max Sectors parameter name, defaults, and range. This HBA driver parameter is not available on Windows operating systems.

QLogic recommends experimenting with increasing the HBA Max Sectors parameter for large size I/Os or decreasing the parameter for small size I/Os, then measuring for a performance gain. Do not expect a big performance jump, as the default value for the HBA Max Sectors parameter is pre-optimized for most application environments. QLogic does not recommend setting the value of HBA Max Sectors above 2048 in any application environment.

The recommended procedure for changing and verifying the Max Sector HBA parameter is similar to the procedure for changing the HBA Max SG List detailed in “HBA Performance Tuning in VMware ESX Server” on page 31. When verifying for a successful change, look for the string max_sectors in the vmkernel file.

Monitoring PerformanceTo determine where a performance problem exists or where a potential problem will happen, it is important to gather data from all the components of the storage solution. It is not uncommon to be misled by a single piece of information and misdiagnosing the cause of poor system performance, only to realize that another component of the system is causing the problem. This section looks at what utilities, tools, and monitors are available to help you analyze what is happening within your SAN environment.

Transaction performance is generally perceived to be poor when the following conditions occur:

Random read/write operations are exceeding 20ms (without write cache).

Random write operations are exceeding 2ms with the cache enabled.

I/Os are queuing up in the operating system I/O stack (due to a bottleneck).

Throughput performance is generally perceived to be poor when the disk capability is not being reached. Causes of this can stem from the following situations:

With read operations, read-ahead is being limited, preventing higher amounts of immediate data available.

I/Os are queuing up in the operating system I/O stack (due to a bottleneck).

Most operating systems, Fibre Channel adapters, Fibre Channel switches, and disk arrays provide a host of utilities (add-on and built-in) to help you monitor I/O performance.

Parameter Description

Denoted in Driver by

Default in ESX 2.5.x

Default in ESX 3.0.x Range

HBA Max Sectors ql2xmaxsectors 512 512 512, 1024, 2048



QLogic recommends breaking down the measurement into the following three parts:

Gathering host server data.

Gathering fabric network data.

Gathering EMC storage server data.

Gathering Host Server Data

Most operating systems (OSs) have several built-in utilities to measure I/O performance and the effect of this performance on other host server components (CPU consumption, memory utilization, etc.). The following table summarizes many of the commonly used tools.

Tool OS Description Key Statistics Usage

SANsurfer Windows, Linux, Solaris

QLogic HBA performance monitor HBA port IOPS, MBps, error statistics


perfmon Windows Windows built-in real time performance monitoring tool. Ability to set alerts for various thresholds.

CPU, disk, memory, network

Menu under Administrative tools → Performance or by entering perfmon at the command line

esxtop VMware Display ESX server resource utilization statistics in real time.

CPU, disk, memory, network

/usr/bin/esxtop

Virtual Center/Virtual Infrastructure Client

VMware Performance tabs for these VMware applications display performance charts per VM and host server

Virtual machine CPU, disk, memory, network statistics

http://<ip address of host>/

vmkusage VMware 2.x Historical graphs that show physical server and virtual machine system statistics

Virtual machine CPU, disk, memory, network statistics

http://<ip address of host>/vmkusage




Gathering Fabric Network Data

One of the easiest ways of measuring and monitoring I/O performance in a SAN is to look at I/O statistics at the heart of the SAN — the Fibre Channel switch. These statistics provide a view of SAN performance from both the initiator and target. QLogic recommends looking at the FC switch statistics as the first step in any performance monitoring exercise. QLogic Fibre Channel switches have a built-in performance monitor agent. Use the SANsurfer Performance Viewer to connect to a Fibre Channel switch and monitor I/O performance for one or more switch ports of your choice. The viewer provides real time graphs of current I/O workloads for easy comparison.


HBA Performance Tuning Best PracticesChoosing the Optimal PCI Slot for Your HBA

Gathering Disk Array Performance DataAll EMC storage arrays have built-in capabilities for performance monitoring and measurement. These tools provide a wealth of information about how I/O is being processed by the storage array, which disks are taking the most I/O hits, what is the cache hit percentage, what is the split between read and write operations across the array, etc. QLogic recommends installing and using the EMC Navisphere Manager for CLARiiON arrays to supplement any performance monitoring done in your SAN. Navisphere Manager fully integrates the Navisphere Analyzer, which provides detailed real-time information about your array and system performance. There is an easy-to-use data record and playback facility for reviewing the captured data. Navisphere Analyzer lets you automatically record data metrics from selected arrays, LUNs, and storage processors. You can set the schedule of when to record the data, where to store the results, and from which hosts to gather data. These features allow you to investigate performance patterns and trends before problems occur, and provide a powerful tool for capacity planning and intelligent load balancing.

Explore http://germany.emc.com/pdf/products/navisphere/navisphere_analyzer.pdf for more information about the EMC Navisphere Analyzer.

Choosing the Optimal PCI Slot for Your HBAA typical enterprise server may have several add on cards — SCSI, VGA, LAN, HBA, etc., as well as a variety of slots to house these cards. In a shared bus environment like PCI, it is imperative that the right PCI slot be chosen to host the QLogic HBA. The type and placement of the slot makes a significant difference in the HBA performance. The following table matches each QLogic HBA to the appropriate slot. This section will help you choose the right slot in your server with multiple prospective slots.

Table 4. EMC-Supported QLogic HBAs

HBA PCI Slot Bus Length Power Slot Key

QLA2460-E PCI-X 2.0 64-bit 3.3V 3.3V

QLA2462-E PCI-X 2.0 64-bit 3.3V 3.3V

QLE2460-E PCI Express 1.0a x4 lane 3.3V n/a

QLE2462-E PCI Express 1.0a x4 lane 3.3V n/a


http://germany.emc.com/pdf/products/navisphere/navisphere_analyzer.pdf


A typical motherboard I/O diagram is depicted below; it shows the maximum available bandwidth in each direction for various slot types and the difference between these slot types.

General Rules for PCI and PCI-X HBAsA PCI bus operates at the clock rate of the slowest card installed on that bus. Therefore, for optimum performance, PCI/PCI-X cards of different speeds should not be installed on the same bus.

Many PCI/PCI-X slots share the bus bandwidth; closely spaced PCI slots most likely share the same PCI bus and bridge (the figure below shows an example). QLogic recommends that you obtain the I/O Diagram for your host server from its manufacturer to understand which PCI/PCI-X slots share a PCI bridge and bus. For example, on an HP Integrity rx5670 server, a common 66-MHz bus is shared between slots 4 and 5, slots 6 and 7, and slots 8 and 9. To maximize PCI bus bandwidth, populate only one slot on each bus and leave the second slot empty. For example, install cards in slots 4, 6, and 8, but leave slots 5, 7, and 9 for non-performance critical adapters.



Match the slots speeds with device/adapter speeds; slower or non-performance critical should take up the slower PCI/PCI-X slots.

HBAs without a slot key can be inserted into any PCI slot; you can insert a 64-bit HBA into a 32-bit PCI slot if no 64-bit PCI-X slots are available. In this case, however, the data transmission rate is limited to standard PCI speed. Avoid such HBA placements.

General Rules for PCI Express (PCIe) HBAsUnlike PCI/PCI-X which divides bandwidth between all devices on the bus, PCI Express provides each device with its own dedicated data pipeline. Data is sent serially in packets through pairs of transmit and receive signals called lanes, which enable 250 MBps bandwidth per direction, per lane. Multiple lanes can be grouped together into x1 (“by-one”), x2, x4, x8, x12, x16, and x32 lane widths to increase bandwidth to the slot.

This allows a QLogic PCIe HBA to be placed in any available PCIe slot on a server that can fit the HBA connector without any performance impedances.

A QLogic PCIe x4 FC HBA will not fit in an x1 or an x2 slot, but will fit into an x4, x8 and x16 slot. An x4 slot is the right slot for this kind of a HBA. Placing an x4 HBA into an x8 or x16 slot will not make it run faster, such a placement only wastes a high performance slot which could be used for a higher performance device.

The bar chart below compares Fibre Channel HBA capabilities with PCI technologies hosting the HBA. Use this as a reference to see if you are imposing any limitations on the performance of the HBA by putting it in a slower slot. 8Gb data is a future projection. PCI Express denotations are for Gen1 (1.0a)

FC Interfaces Bar Chart


HBA Performance Tuning Best PracticesGeneral SAN Considerations for Performance

General SAN Considerations for Performance

Fencing High Performance Applications

It is good practice to fence out a high performance demanding application away from other applications by assigning it a dedicated SAN fabric. If this fencing is not implemented, high performing applications can overwhelm other applications competing for the same storage resource over the same port.

Minimize ISL

QLogic recommends that you locate the HBA and the storage ports it will access on the same switch. Otherwise, try to minimize ISLs and decrease the number of hops. The more hops the data takes, the slower it reaches its destination.

Upgrade to 4Gb

QLogic recommends replacing all 2Gb Fibre Channel adapters with QLogic 4Gb adapters, as 4Gb Fibre Channel adapters have many advantages over 2Gb adapters. Besides performance advantages, they offer new features, High Availability, and better troubleshooting capabilities.

Set Data Rate to Auto-negotiate

EMC recommends that all Fibre Channel ports (HBAs, switches, and storage ports) be set to auto-negotiate the Fibre Channel speed. Any other setting results in a slow connection. All QLogic HBAs come factory set to auto-negotiate.

Fan-In Considerations

Consider the ratio of storage ports bound to a single QLogic HBA port; this ratio is called the Fan-In. The Fan-In ideally is 1:1 — one storage port is bound to a single HBA. If the application accessing the storage ports is not bandwidth intensive, then the Fan-In can be relaxed to 3:1 or further, depending on the application and time-of-the-day load. Visit powerlink.emc.com to review EMC recommended Fan-In and Fan-Out ratios for popular operating systems and EMC Arrays.

Avoiding RSCNs

Assign a unique Fibre Channel switch zone between the server initiator and the storage controller processor ports. That is, for each WWN (each initiator HBA), regardless of the number of ports, there should be one unique zone. This zoning strategy isolates an initiator from Register State Change Notification (RSCN) disruptions, which can hamper performance.


HBA Performance Tuning Best PracticesI/O Load Balancing

Linear Scaling of HBAs

It is possible to achieve higher performance by scaling the number of HBAs in a host. When installing more than one HBA on a server, QLogic recommends that you follow the slot identification guidelines in “HBA Parameters that Impact HBA Performance” (page 28) to take advantage of the full capacity of two or more QLogic HBAs. The graph below shows that total bus throughput increases with number of devices for different interfaces.

I/O Load BalancingYou can maximize the performance of an application by spreading the I/O load across multiple paths, arrays, and device adapters in the storage unit.

When trying to balance the load within the storage unit, placement of application data is the determining factor. The following resources, in relative order of importance, are the most critical to balance:

Activity to the RAID disk groups. Use as many RAID disk groups as possible for critical applications. Many performance bottlenecks occur because a few disks are overloaded. Spreading an application across multiple RAID disk groups ensures that as many disk drives as possible are available.

Activity to the device adapters. When selecting RAID disk groups within a cluster for a critical application, spread them across separate device adapters.


HBA Performance Tuning Best PracticesI/O Load Balancing

Activity to the Fibre Channel ports. EMC recommends the use of an EMC automated non-disruptive solution — PowerPath — to load balance I/O between paths in your SAN. Visit www.emc.com to learn more about PowerPath. PowerPath is not available for VMware ESX Server.

NOTE: With PowerPath, servers are continually tuned to adjust to changing application loads. This is because channel directors/storage processors write to cache instead of disk.


http://www.emc.com

Fibre Channel Security Best Practices

As more business critical data is being consolidated into SANs, protecting this data has come to the forefront as one of the key best practices that every SAN administrator must implement. To effectively protect a Storage Area Network, it is important to understand what actions increase security and what actions create potential loopholes. It is also important to understand the impact of these actions on the rest of the environment.

As with any other best practice, the "best" approach to implementing security is dependent on the business and the current needs of the organization, along with an assessment of risk to the data being protected. This section defines a series of best practice recommendations for enhancing the security of Fibre Channel SANs. The focus is on QLogic Fibre Channel Host Bus Adapters, associated software, and how security relates to generic SAN best practices.

NOTE: Fibre Channel security is as strong as the weakest component in the SAN; typically, hosts that are connected to a SAN constitute the weakest link. Therefore, protecting hosts from intruders should have priority over protecting other components. Today's hosts potentially host hundreds of different applications, OSs, and driver components, many of them with their own security considerations. It is essential to keep yourself and your host abreast with the latest security patches for all components of the host.

Setting a Password for QLogic SANsurfer FC HBA ManagerQLogic SANsurfer FC HBA Manager software runs on a host server and allows you to install, configure, troubleshoot, and analyze an installed QLogic HBA. Many SANsurfer features are password protected. The default password for these features is “config”, and is known by almost anybody who manages a SAN or has access to the internet and a search engine. Leaving the SANsurfer password at its default value means that anyone with access to the host server could easily change key HBA parameters and potentially breach the security of the SAN.

QLogic recommends that you set both host and application access passwords for SANsurfer by performing the following steps:

1. In the SANsurfer FC HBA Manager main window HBA tree, select the host for which you want to set the application access password.

2. Click the Security tab. The Security tabbed page displays, as shown in the following figure.


Fibre Channel Security Best PracticesGeneric Fibre Channel SAN Security

Generic Fibre Channel SAN SecurityThe EMC white paper Security for Fibre Channel Storage Area Networks provides a series of best practice recommendations for enhancing the security of a Fibre Channel SAN; the paper focuses on general SAN technologies as opposed to specific products. EMC recommends that you tailor these security guidelines for your environment. You can find this white paper at http://powerlink.emc.com/.



Appendix A – QLogic HBA Tasks and Tools Sheet

This appendix provides:

Task: Information (page 51)

Task: Monitoring/Diagnostics (page 52)

Task: Configuration (page 53)

Task: Maintenance (page 54)

Task: Other (page 54)

Task: Information

Subtask

SANsurfer FC HBA Manager

SANsurferFC CLI

Windows InstallWizard FlasUTIL FAST!UTIL EfiUTIL

Linux Tools

Operating System Windows, Solaris, NetWare, Mac, Linux

Windows, Solaris, NetWare, Mac, Linux

Windows Any Any Any Linux

Serial number ✓ ✓ ✓ ✓ ✓

Model/type ✓ ✓ ✓

OptionROM (BIOS/FCode/EFI) version

✓ ✓ ✓ ✓

Firmware version ✓ ✓

Driver version ✓ ✓

Fibre Channel specific data: WWN, loop ID

✓ ✓ ✓ ✓ ✓

Target information ✓ ✓ ✓

VPD displays ✓ ✓ ✓

Graphic SAN topology ✓



Task: Monitoring/Diagnostics

Subtask


SANsurferFC CLI


Linux Tools

Real-time status of the HBAs

✓ ✓

On-demand Fibre Channel link status

✓ ✓

Read/write buffer test ✓ ✓ ✓

Loopback test ✓ ✓ ✓ ✓a

SFP health ✓

Verifying NVRAM, flash ✓ ✓

Email notification of fabric events

✓ ✓

Error/alarm reporting ✓ ✓

Statistics ✓ ✓

Snapshot ✓b

Low-level disk commands

✓

LUN state transition ✓c

NoteaWith loopback plugbUse Linux tool ql-hba-snapshot.shcUse Linux tool ql-lun-state.sh



Task: Configuration

Subtask


SANsurferFC CLI


Linux Tools

Driver settings ✓ ✓ ✓

NVRAM settings ✓ ✓

Persistent binding ✓ ✓

LUN masking ✓ ✓ ✓ ✓

Updating factory NVRAM defaults

✓ ✓ ✓

BOOT from SAN device

✓ ✓ ✓ ✓a

Scan for Fibre Channel devices

✓ ✓ ✓b

Remote HBA management

✓ ✓ ✓c

SCSI command timeout value

Failover ✓ ✓

Device replacement ✓ ✓

Option ROM config ✓ ✓

iiDMA setting ✓ ✓

NoteaUse Linux tool ql-scan-lun.shbCapability limited to Linux tools ql-lun-state.sh and ql-set-cmd-timeout.shcUse Linux tool ql-set-cmd-timeout.sh



Task: Maintenance

Task: Other

Subtask


SANsurferFC CLI


Linux Tools

Driver installation/update

✓ ✓

Change boot code/Option ROM (BIOS, EFI, FCode)

✓ ✓ ✓

Copying the current flash and NVRAM contents

✓ ✓ ✓

Installation reports ✓

Subtask


SANsurferFC CLI


Linux Tools

Scripting ✓

Generate XML ✓

Generate a summary of current configuration of the local host/HBA/devices

✓


Appendix B – QLogic HBA LED Scheme

HBA Scheme

QLA2460-E QLA2462-E QLE2460-E QLE2462-E OFF OFF OFF Power Off

On On On Power On (before firmware init.)

Flashing Flashing Flashing Power ON (after firmware init.)

Firmware Error

Flashing OFF Flashing Beacon

On/Flashing OFF OFF Online, 4 Gbps link / I/O activity

OFF On/Flashing OFF Online, 2 Gbps link / I/O activity

OFF OFF On/Flashing Online, 1 Gbps link / I/O activity

Green LED(2 Gbps)

Amber LED(1 Gbps)

Activity

Yellow, Green, and Amber LEDs flashing alternately

Yellow LED (4 Gbps)


Appendix B – QLogic HBA LED Scheme


References

EMC CLARiiON High Availability (HA) Best Practices, July 2005 at http://powerlink.emc.com/

EMC Fibre Channel and iSCSI with QLogic Host Bus Adapter in the Linux v2.6.x kernel Environment and the v8.x-Series Drivers, Rev A04 at http://powerlink.emc.com/

EMC Fibre Channel and iSCSI with QLogic Host Bus Adapter in the Windows Environment, Rev A13 at http://powerlink.emc.com/

EMC Network Storage Topology Guide, Rev A05 at http://powerlink.emc.com/

EMC Security for Fibre Channel Storage Area Networks Best Practices Planning, August 2006 at http://powerlink.emc.com/

QLogic Corporation EMC Approved Software Website section at http://support.qlogic.com/support/oem_emc.asp

QLogic SANblade 2200 and 2300 Series Troubleshooting Guide for EMC, April 2003 at http://www.qlogic.com/

VMware Performance Tuning Best Practices for ESX Server 3, Technical Note at http://vmware.com

VMware SAN System Design and Deployment Guide, March 2007 at http://vmware.com/

VMware Technology Network (VMTN) at http://vmware.com/

http://www.techtarget.com/ - Several storage connectivity related articles.





http://vmware.com/


http://www.techtarget.com/

http://www.qlogic.com/


http://vmware.com/

http://vmware.com


References


QLogic Press Review

The QLogic Press authors value your feedback. We are especially interested in situations where a QLogic Press publication has made a difference. Please review the document, addressing value, subject matter, structure, depth, and quality as appropriate.

Please use one of the following methods to provide feedback so we can make this document better suit your needs:

■ Visit the QLogic website at http://www.qlogic.com/go/qlogic-press-review and complete the online form; or

■ Print both pages of this Press Review form and fax completed pages to: (949) 389-6114.

Your Contact InformationThe data you provide here may be used to provide you with information from QLogic about our products, services, or activities.

Please identify yourself as belonging to one of the following groups:

QLogic Press Document Title

Please rate your overall satisfaction with the document:

Name:

Title:

Company Name:

Email Address:

Phone Number:

Address:

City, State, Zip:

Customer Business Partner

Solution Developer QLogic Employee

Other (please specify):

Very Satisfied Satisfied Somewhat Satisfied Not Satisfied


http://www.qlogic.com/go/qlogic-press-review

QLogic Press Review

Review Comments

What other subjects would you like to see QLogic Press address?


QLogic Company Information

Powered by QLogicQLogic is a leading supplier of high-performance storage networking solutions, producing the controller chips, host bus adapters (HBAs), and fabric switches that are the backbone of storage networks for most Global 2000 corporations. The company delivers a broad and diverse portfolio of products that includes Fibre Channel HBAs, blade server embedded Fibre Channel switches, Fibre Channel stackable switches, iSCSI HBAs, iSCSI routers, and storage services platforms for enabling advanced storage management applications. The company is also a leading supplier of InfiniBand switches and InfiniBand host channel adapters for the emerging High Performance Computing Cluster (HPCC) market. QLogic products are delivered to small-to-medium businesses and large enterprises around the world via its channel partner community. QLogic products are also powering solutions from leading companies like Cisco, Dell, EMC, Hitachi Data Systems, HP, IBM, NEC, Network Appliance, and Sun Microsystems. QLogic is a member of the S&P 500 Index.

To learn more about QLogic, visit the QLogic website at www.qlogic.com or refer to the contact information on the following page.


http://www.qlogic.com

QLogic Company Information

Contacting QLogic

Partner Programs

Sales Education and Technical Training

Sales Information800.662.4471

Technical Support952.932.4040 [email protected]

Corporate Headquarters QLogic Corporation 26650 Aliso Viejo Parkway Aliso Viejo, CA 92656 Phone: 949.389.6000 Fax: 949.389.6009

Germany–Sales Office QLogic Corporation Terminalstr. Mitte 18 85356 Muchen Germany (49) 89 97007-427

UK–Sales Office QLogic Corporation Surrey Technology Centre 40 Occam Road Guildford GU2 5YG Surrey, UK Phone: (44) 1483-295825 Fax: (44) 1483-295827

QLogic Asia–Sales Office QLogic Corporation 23F, 105 Dun Hua S. Road Sec 2, Taipei 106, Taiwan, R.O.C. 886-2-2755-0000 Ext. 501

Channel Programs 877.975.6442 [email protected]

Interoperability Testing [email protected]

Business Alliances Programs 949.389.6557 [email protected]

Technical Training [email protected]

Sales Training [email protected]


mailto:[email protected]






B e s t P r a c t i c e s

QLogic FC HBA in an EMC Environment

Best Practices

About QLogic

QLogic is a leading supplier of high-performance storage networking solutions, producing the controller chips, host bus adapters (HBas) and

fabric switches that are the backbone of storage networks for most Global 2000 corporations. the company delivers a broad and diverse portfolio

of products that includes Fibre channel HBas, blade server embedded Fibre channel switches, Fibre channel stackable switches, iscsi HBas,

iscsi routers and storage services platforms for enabling advanced storage management applications. the company is also a leading supplier of

infiniBand switches and infiniBand host channel adapters for the emerging High Performance computing cluster (HPcc) market. QLogic products

are delivered to small-to-medium businesses and large enterprises around the world via its channel partner community. QLogic products are also

powering solutions from leading companies like cisco, Dell, eMc, Hitachi Data systems, HP, iBM, Nec, Network appliance and sun Microsystems.

QLogic is a member of the s&P 500 index. For more information go to http://www.qlogic.com.

recent accolades include:

- s&P 500 index

- Forbes’ Best 200 Best small companies

- Fortune’s 100 Fastest Growing companies

- Business Week Hot Growth company

- Network computing Well connected

- 2005 storage Magazine “saN Product of the Year”

- 2005 saNbox 5602 infostor and asNP “Most Valuable Product Finalist”

QLogic FC HBA in an EMC Environment JULY 2007

corporate Headquarters QLogic corporation 26650 aliso Viejo Parkway aliso Viejo, ca 92656 949.389.6000

europe Headquarters QLogic (UK) LtD. surrey technology centre 40 Occam road Guildford surrey GU2 7YG UK +44 (0)1483 295825

Fc0054602-00 rev a 7/07

QLogic FC H

BA in an EMC Envi ron m

ent

JULY 2007

JULY 2007QLogic Press V1

plugin-user%27s guide - qlogic fc hba in an emc environment

Documents