sr-iov configuration guide - intel · revision 1.0 333876-001 sr-iov configuration guide intel®...

Revision 1.0333876-001

SR-IOV Configuration GuideIntel® Ethernet CNA X710 & XL710 on VMware* vSphere* 6Technical Brief

Networking Division (ND)

February 2016

SR-IOV Configuration GuideIntel® Ethernet CNA X710 & XL710 on VMware vSphere 6

2 333876-001

Revision History

Revision Date Comments

1.0 February 9, 2016 Initial release (Intel public).

333876-001 3


Contents

1.0 Introduction ............................................................................................................. 51.1 Intel® Ethernet Controller XL710 .......................................................................... 51.2 I/O Virtualization ................................................................................................ 6

1.2.1 Hardware Requirements ................................................................................. 61.2.2 Software Requirements .................................................................................. 71.2.3 Supported Guest Configurations ...................................................................... 71.2.4 Guest OS...................................................................................................... 71.2.5 VF Driver in the Guest OS............................................................................... 7

2.0 Installation and Configuration .................................................................................. 82.1 Server Setup...................................................................................................... 82.2 VF Creation........................................................................................................ 8

2.2.1 VF Creation via CLI Console Configuration......................................................... 82.2.2 VF Creation via Host Profile Configuration ........................................................10

2.3 VF Assignment to Guest Virtual Machines ..............................................................152.4 VF Use in Guest Operating Systems......................................................................17

2.4.1 Driver Support .............................................................................................172.4.2 Verification ..................................................................................................182.4.3 Traffic Monitoring of NICs in SR-IOV Mode .......................................................19

2.5 Advanced Topics................................................................................................19

3.0 Summary .................................................................................................................20

4.0 Customer Support ....................................................................................................20

5.0 Product Information ................................................................................................20


4 333876-001

NOTE: This page intentionally left blank.

333876-001 5


1.0 IntroductionSince its inception in 1973, Intel has been vital to the development of the Ethernet, and continues to be the Industry leader. For over 40 years, the Ethernet has been growing to accommodate increasing bandwidth needs for complex multi-media; streaming video, music and voice data, for example. Beginning with One Gigabit Ethernet, expanding to 10 Gigabit, and now introducing 40 Gigabit Ethernet, computing and storage resource needs continue to grow.

1.1 Intel® Ethernet Controller XL710The 40 Gigabit XL710 Controller is designed for flexibility, with configurable port speeds of up to 2 x 40 GbE, or 4 x 10 GbE, ensuring a smooth transition to 40 GbE, It also provides a 222% increase in Gigabits per Watt in adapter power for approximately half the power cost when compared to using two previous generation dual-port adapters.

The XL710 offers the following features:

• 10/40 GbE Controller (Dual and Single 40 GbE, Quad and Dual 10 GbE configurations).

• PCI Express* (PCIe) 3.0, x8 including Direct I/O optimizations via TLP Processing Hints (TPH).

• Intelligent Off-load to enable high-performance with Intel® Xeon® servers.

• Network Virtualization off-loads including VXLAN and NVGRE.

• Industry-leading I/O virtualization innovations and performance with broad hypervisor and standards support.

• Intel® Ethernet Flow Director (for hardware application traffic steering).

• Excellent small packet performance for network appliances and NFV.

• Intel® Data Plane Developer Kit Optimize.

• Unified Networking providing a single wire for LAN and storage: NAS (SMB,NFS) and SAN (iSCSI).

The following are the Intel 40 Gigabit XL710 Controller-based Dual and Quad Adapter offerings:

Note: These boards do NOT ship with optics installed. Optics must be purchased separately.

• Intel® Ethernet Converged Network Adapter X710-DA4

— X710DA4FH, XL710DA4FHBLK (Retail, Quad Port FH)

— X710DA4FHG1P5 (OEM Gen, Quad Port FH)

— X710DA4G1P5 (OEM Gen, Quad Port LP)

• Intel® Ethernet Converged Network Adapter X710-DA2

— X710DA2, XL710DA2BLK (Retail, Dual Port)

— X710DA2G1P5 (OEM Gen, Dual Port)

• Intel® Ethernet Converged Network Adapter XL710-QDA2

— XL710QDA2, XL710QDA2BLK (Retail, Dual Port)

— XL710QDA2G1P5 (OEM Gen, Dual Port)


6 333876-001

• Intel® Ethernet Converged Network Adapter XL710-QDA1

— XL710QDA1, XL710QDA1BLK (Retail, Single Port)

— XL710QDA1G1P5 (OEM Gen, Single Port)

Power efficiency is critical to IT specialists as energy consumption is a real concern in data center operations. The Intel Ethernet Controller provides a low-power interface to eliminate the need for additional power. It also offers the manageability IT personnel require for remote control and alerting.

This controller provides multiple interface options, a smaller footprint for reduced infrastructure and cabling costs, lower power consumption, and intelligent off-loads that do not require disabling key features and flow direction to balance high volume traffic flows.

1.2 I/O VirtualizationThe Intel® Ethernet Server Adapter X710 & XL710 family of adapters delivers numerous industry-leading features that are helping data center administrators implement innovative solutions for difficult and challenging connectivity problems. I/O Virtualization is one of the fastest growing usage models within the data center.

The X710 & XL710 family of adapters provides the ability to create Virtual Functions (VFs) that are lightweight instantiations of the Physical Functions (PFs). VFs are capable of providing up to 10 GbE or 40 GbE connectivity to Virtual Machines (VMs) within a virtualized operating system framework. The Intel® Ethernet Server Adapter X710 supports up to 32 VFs per port, for a total of up to 128 VFs per adapter. The Intel® Ethernet Server Adapter XL710 supports up to 64 VFs per port, for a total of up to 128 VFs per adapter.

In VMware* vSphere* 6, SR-IOV enables administrators to scale the lower latency and higher throughput performance benefits of VMware DirectPath I/O* pass-through technology to multiple guests VMs without increasing physical port count. To learn more about VMware's implementation of SR-IOV, including restrictions on Guest OS and feature support, refer to VMware's documentation at the following address:

https://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-6-pubs.html

This document shows how to make use of these VFs using VMware vSphere 6.

1.2.1 Hardware Requirements

The server platform must have:

• Compatibility with the VMware vSphere 6 release.

• An Intel® Xeon® processor E5 or E7 family.

• Support for an Input/Output Memory Management Unit (IOMMU) with IOMMU enabled in the Server BIOS.

• Support for Intel® Virtualization Technology for Directed I/O (Intel® VT-d) and the PCI-SIG* Single Root I/O Virtualization and Sharing (SR-IOV) specification.

• Support for SR-IOV enabled in the Server BIOS. Contact the server vendor to determine whether the server platform supports SR-IOV.

• An available PCIe slot: x8 5.0 Gb/s (Gen2) or x8 8.0 Gb/s (Gen3).

• An Intel® Ethernet Converged Network Adapter X710 or XL710.

Note: To verify compatibility of physical hosts and NICs with vSphere releases, refer to the VMware Compatibility Guide.

https://www.vmware.com/support/pubs/vsphere-esxi-vcenter-server-6-pubs.html

https://www.vmware.com/resources/compatibility/search.php

333876-001 7


1.2.2 Software Requirements

Following are the software requirements:

• VMware vSphere 6 (Commonly known as VMware ESXi 6).

• VMware vCenter* Server 6.

• Intel® Ethernet Converged Network Adapter X710 or XL710 Linux Drivers for Virtual Function

http://sourceforge.net/projects/e1000/files/

• Intel® Ethernet Converged Network Adapter X710 or XL710 Windows Drivers for Virtual Function

https://downloadcenter.intel.com

• Intel® Ethernet Converged Network Adapter X710 or XL710 VMware ESXi 6 Drivers for Physical Function

https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI55-INTEL-I40E-1345&productId=353

1.2.3 Supported Guest Configurations

For supported guest configurations, refer to the VMware documentation.

1.2.4 Guest OS

• Red Hat Enterprise Linux 7.x (64-bit).

• SuSE Linux Enterprise Server 12.x (64-bit).

• Microsoft Windows Server 2012 (64-bit) and newer.

1.2.5 VF Driver in the Guest OS

• Must be compatible with the NIC.

• Must be supported on the Guest OS release according to the technical documentation from the NIC vendor.

• Must be Microsoft WLK or WHCK certified for the Windows virtual machines.

• Must be installed on the OS.

The OS release contains a general, default driver for the NICs. For specific NICs, download and install the appropriate driver provided by the NIC's manufacturer.


https://downloadcenter.intel.com

https://my.vmware.com/web/vmware/details?downloadGroup=DT-ESXI55-INTEL-I40E-1345&productId=353


8 333876-001

2.0 Installation and Configuration

2.1 Server SetupThis section shows various setup and configuration steps for enabling SR-IOV on Intel® Ethernet CNA X710 or XL710 server adapters.

1. Install Intel® Ethernet CNA X710 or XL710 server adapter in an available PCIe x8 slot. (Ensure that the x8 slot is electrically connected as x8, some slots are physically x8 but electrically support only x4. Verify this with your server manufacturer or system documentation.)

2. Power up the server.

3. Enter the server's BIOS setup and make sure the Virtualization Technology, Intel® VT-d and SR-IOV features are enabled.

4. Install VMware ESXi 6 on the server..

5. Install and configure vCenter Server 6 as per VMware's vSphere 6 documentation. vCenter Server is required for managing ESXi hosts.

Note: VMware vSphere Web Client is the recommended tool for managing ESXi hosts. VMware's standalone vSphere client application is not supported for configuring and managing ESXi hosts.

2.2 VF CreationThere are two ways to create VFs on Intel Ethernet Adapters on VMware ESXi 6:

• via command line console configuration.

• via host profile configuration.

Both of these methods are described below.

2.2.1 VF Creation via CLI Console Configuration

1. Enable ESXi Shell, either remote or direct console, for CLI configuration. Refer to the VMware documentation for more details on shell enablement.

2. Log into ESXi Shell.

3. Before you begin, make a note of the server adapter port numbering on the PCI bus.

~ # lspci | grep -i intel | grep -i 'ethernet\|network'

4. Set i40 max_vfs module parameter to create VFs:

~ # esxcfg-module i40e -s max_vfs=Y,Z

or:

~ # esxcli system module paramaters set -m i40e -p max_vfs=Y,Z

333876-001 9


max_vfs is a comma-separated list, where each number in the list corresponds to the number of virtual functions enabled on a physical port. The numbers are ordered according to the order of the NIC ports on the PCI bus.

Note: Use lspci | grep -i intel | grep -i 'ethernet\|network' to see port ordering on the PCI bus. For example, max_vfs=0,10 corresponds to 10 VFs on adapter 1 port 2.

The maximum number of VFs supported per ESXi host is a function of the maximum number of MSI-X vectors available for VF use. Intel Ethernet CNAs require three MSI-X vectors per VF. VMware ESXi 6 hosts have at most 128 MSI-X vectors available for VF use. The actual limit depends on your server configuration. Refer to the VMware ESX 6 Networking Guide for maximum supported VF number.

5. Check i40e module parameters to verify settings:

~ # esxcfg-module -g i40e

Example output:

6. Reboot the ESXi host for VF settings to take effect.

7. Check that VFs are created on the PCI bus through the shell commands and in the vSphere Web Client GUI under Host > Manage > Settings > PCI Devices.


10 333876-001

2.2.2 VF Creation via Host Profile Configuration

For completeness, instructions for configuring SR-IOV via Host Profiles in ESXi are included in this section. Most of this information is also included in the VMware vSphere 6 documentation. For more information about host profiles, see the vSphere Host Profiles documentation.

1. From vSphere Web Client, click on the Host Profiles icon.

333876-001 11


2. Creating a new Host Profile is accomplished by extracting the designated reference host's configuration. Click on the + icon in Host Profile's Objects tab.

3. Select a Host to extract designated reference host's configuration.

4. Assign a unique profile name and click the Finish button. This action extracts the designated reference host's configuration and populates the resulting host profile in Host Profile's Objects tab.


12 333876-001

5. The Host Profile can now be edited. Select Edit Settings from the Actions drop-down menu.

333876-001 13


6. In the Host Profile edit screen, navigate to General System Settings > Kernel Module Configuration > Kernel Module > i40e > Kernel Module Parameter > max_vfs. Adjust the number of VFs to the desired value and click the Finish button.

7. Attach the profile to the host. Return to the Hosts & Clusters window, right-click on the host, and select Host Profile > Attach Host Profile. Select the SR-IOV enabled profile from the list. Set the host in Maintenance Mode and apply the Profile.


14 333876-001

8. ESXi host must be rebooted for the changes to take effect.

9. Check that the VFs are created in the vSphere Web Client GUI. on the Host page, select Settings on the Manage tab, then select Hardware > PCI Devices. The numbers are ordered according to the order of the NIC ports on the PCI bus.

333876-001 15


2.3 VF Assignment to Guest Virtual Machines1. Power off the virtual machine.

2. Select the VM that is intended for VF assignment and select Edit Settings from the Actions drop-down menu.

3. Expand the Memory section and click on the Reserve all guest memory (All locked) check box in the Reservation option.

4. Select Network device from the New Device drop-down menu and click the Add button. vSphere Web Client populates a new network device within the window. Expand the newly added Network Device.


16 333876-001

a. Select SR-IOV passthrough device in the Adapter Type section.

b. Select the appropriate physical function that has SR-IOV enabled and has VFs associated with it. The vSphere Web Client GUI only shows devices that have SR-IOV enabled in the Physical Function field.

c. Select the Allow or Disallow option from the Guest OS MTU Change field to allow or disallow the Virtual Machine to change VF's MTU size.

Assigning one VF to multiple VMs is not supported. vSphere Web Client GUI manages the VFs for a given Physical Function (PF) and keeps track of VF assignments.

Note: VM compatibility mode should be for at least ESX 5.5, (VM version 10 and above).

333876-001 17


VMware limits support for SR-IOV to specified Guest operating systems only. As an extension of DirectPath I/O, all limitations and rules that apply to VMware DirectPath I/O may also apply to SR-IOV in ESXi 6. Refer to the VMware documentation for OS support and supported VF functionality.

5. Complete the configuration change and start the VM to begin using the VF in the Virtual Machine (also called Guest OS).

2.4 VF Use in Guest Operating Systems

2.4.1 Driver Support

For VF support with Microsoft Windows Server 2012 R2 guests, download and install the latest network adapter drivers from Intel.com.

https://downloadcenter.intel.com/product/82947/Intel-Ethernet-Controller-X710-Series

For Red Hat Enterprise Linux 7.x guests, VF drivers are part of the distribution. Updating the VF driver to the latest version is strongly recommended. VF drivers are available for download from the following:

https://downloadcenter.intel.com/download/24411/Network-Adapter-Driver-for-PCI-E-40-Gigabit-Network-Connections-under-Linux-


https://downloadcenter.intel.com/product/82947/Intel-Ethernet-Controller-X710-Series





18 333876-001

2.4.2 Verification

Once the VFs have been assigned to VMs and the VMs are loaded with a VMware SR-IOV supported Guest OS, confirm the presence of the VF in the Guest OS.

1. In Microsoft Windows Server 2012 R2 guest console, open Device Manager or Network Connections to view Virtual Function assigned.

2. Verify network connections.

333876-001 19


3. In Red Hat Enterprise (RHEL) 7.x guest console, use the following to show Virtual Function:

lspci | grep -i intel | grep -i 'ethernet\|network'

Or, use the following to show Virtual Function driver and its properties.

2.4.3 Traffic Monitoring of NICs in SR-IOV Mode

When a network adapter is in SR-IOV mode, its physical function is visible to ESXi host monitoring tools. Network traffic monitoring in VMware vSphere 6 Web Client is not possible for Virtual Functions.

2.5 Advanced Topics• In Red Hat Enterprise Linux 7.x, the Linux channel bonding driver is agnostic to the difference

between a network interface on a PF and one on a VF. This enables bonding multiple VFs from multiple unique ports to the same VM. However, since teaming assigns a single MAC Address to multiple ports, MAC Address anti-spoofing security blocks traffic when bonded ports are configured for load balancing or fail-over.

• SR-IOV and Storage over Ethernet: While iSCSI is not supported over VFs in vSphere 6 (ESXi 6), an interesting use case for Intel's dual/quad port CNAs is to use one port of the CNA to provide iSCSI storage to the ESXi host, while the other port(s) is divided into VFs using SR-IOV, providing low latency/high throughput traffic to guest VMs.

• If using server platforms with the Intel® Xeon® E5 processor family, the network adapter to be configured for SR-IOV must be in a PCIe slot provided by the Intel Integrated I/O PCI root complex directly off the processor.

• SR-IOV is not supported on server platforms with the Intel® Xeon® E3 processor family.


20 333876-001

3.0 SummaryIntel's best-of-breed 40 GbE solutions are now available with I/O Virtualization capabilities. Customers get world-class Ethernet support along with I/O virtualization support in mainstream Linux distributions in a single adapter.

4.0 Customer SupportIntel® Customer Support Services offers a broad selection of programs, including phone support and warranty service. For more information, contact us at:

support.intel.com/support/go/network/ adapter/home.htm

Service and availability may vary by country.

5.0 Product InformationTo see the full line of Intel Network Adapters for PCI Express, visit www.intel.com/go/ethernet.

To speak to a customer service representative regarding Intel products, please call 1-800-538-3373 (U.S. and Canada) or visit support.intel.com/support/go/network/contact.htm for the telephone number in your area.

support.intel.com/support/go/network/ adapter/home.htm

www.intel.com/go/ethernet

support.intel.com/support/go/network/contact.htm

support.intel.com/support/go/network/contact.htm

333876-001 21


NOTE: This page intentionally left blank.

22 333876-001

LEGAL

No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.This document contains information on products, services and/or processes in development. All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest forecast, schedule, specifications and roadmaps.The products and services described may contain defects or errors which may cause deviations from published specifications.Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or by visiting www.intel.com/design/literature.htm.Intel and the Intel logo are trademarks of Intel Corporation in the U.S. and/or other countries.* Other names and brands may be claimed as the property of others.© 2016 Intel Corporation.

sr-iov configuration guide - intel · revision 1.0 333876-001 sr-iov configuration guide intel®...

Documents