Slide 1

Slide 2

Implementing Efficient RSS Capable Hardware and Drivers for Windows 7 Rade Trimceski Program Manager Windows Core Networking radetrim@microsoft.com

Slide 3

Agenda Receive-Side Scaling (RSS) and system scalability RSS enhancements in Windows Server 2008 Dynamic MSI-X mapping Processor selection algorithm RSS enhancements in Windows 7 Support for more than 64 processors Per interface indirection table Support for 2-tuple hash NUMA support Load balancing algorithm New DDIs for processor topology and RSS information Summary Q&A

Slide 4

Scalability on Highly Parallel Systems Systems are getting larger and more parallel Devices that handle lots of data should use processors, memory, and I/O links efficiently for optimal system scalability Real, measurable performance differences are available to drivers that implement scalability techniques RSS remains the best solution for addressing the scalability challenges of networking subsystems

Slide 5

RSS Enhancements in Windows Server 2008 MSI-X support Not present in Windows Server 2003 RSS Dynamic MSI-X resource configuration Changing the processor affinity of an RSS queue at runtime More robust load balancing algorithm The activity level of a connection is taken into account Manageability Using netsh interface to turn RSS on/off

Slide 6

Configuring MSI-X Resources Available starting with Windows Server 2008 MSI-X messages vs. MSI-X table entries Miniport driver can allocate an MSI-X message for each processor Initial mapping of MSI-X table entries to MSI-X messages Remapping MSI-X table entries at runtime Setting any arbitrary affinity at run-time Following RSS indirection table Also available: interfaces to mask/unmask an MSI-X table entry

Slide 7

RSS Enhancements in Windows 7 Per-interface indirection table Removing limitations on using a global table Driver workarounds can be removed Number of RSS processors is always less than or equal to the number of RSS queues No need to choose a subset of processors in the indirection table Support for more than 64 processors In all RSS and processor-related DDIs NUMA support Fallback to 2-tuple hash when necessary More robust load balancing algorithm used by stack Processor topology and RSS information made available to drivers

Slide 8

System Processor Topology New DDI returns the processor topology NdisGetProcessorInformationEx Can be called by NDIS 6.20 miniport and protocol drivers NdisGetProcessorInformation is deprecated for NDIS 6.20 drivers Group and processor number Supporting more than 64 processors Socket, core, and hyper-thread information NUMA Node ID for each processor NUMA distance of the NIC to each processor RSS processor information is returned by another DDI - NdisGetRssProcessorInformation Formerly returned by NdisGetProcessorInformation

Slide 9

Per-Interface RSS Information New DDI to return the RSS processor information NdisGetRssProcessorInformation Can be used by NDIS 6.20 miniport and protocol drivers Returns per interface RSS information RSS base processor Number of RSS processors NUMA node of the NIC if known RSS processor set for the interface is sorted based on preference

Slide 10

Initial Per-Interface RSS Processor Set RSS base processor Default is zero Can be set by a standard registry key Maximum number of RSS processors Limited by the number of the RSS queues Can further be controlled by a standard registry key Default is based on MSI-X capabilities and speed NDIS using only one processor from each core Not using hyper-threaded processors on the same core NUMA systems NDIS sets the preference index

Slide 11

More than 64 Processor Support Changes to RSS parameters Include both processor group and processor numbers RSS keywords Support both group and processor number Processor-related DDIs Queue DPC /current processor DDIs RSS information DDIs MSI-X resource allocation and configuration RSS on processors in different groups

Slide 12

Using Two-Tuple Hash Scenario: IPsec packets Ideal solution is to support IPsec task offload Ideally do IPsec decryption before RSS hash calculation Scenario: Unknown options and extension headers NIC can fall back to calculating hash on source and destination IP only TCP will revert to using 2-tuple hash also Using 4-tuples is still preferred Using 2-tuples does not always provide the desired granularity in hash value

Slide 13

RSS Configuration Global keywords affect all interfaces Standard keywords per interface Set the RSS base processor group and number Set the number of RSS Processors Default RSS parameters based on: Speed of the NIC Number of NIC queues MSI-X support Enable/disable RSS through netsh netsh int tcp set global rss=enabled/disabled

Slide 14

Diagnostics and Tools Per-processor performance counters Interrupts DPCs CPU cycles spent in miniport driver, stack, or NDIS Packets received/transmitted NICSTATS tool Display RSS capabilities Display current RSS parameters Display MSI-X capabilities Enable/disable RSS through netsh netsh int tcp set global rss=enabled/disabled

Slide 15

Summary As systems become larger and more parallel, efficient RSS implementation becomes even more important Windows 7 provides a more robust RSS support Improved load-balancing algorithm > 64 processor support Using two-tuple hash when necessary Windows 7 makes it easier to develop RSS-capable drivers Per-interface indirection table Provides information on system processor topology and RSS processor set Windows 7 makes it easier to configure and manage RSS Per-interface reserved registry keywords, diagnostic tools, perfmon counters

Slide 16

Resources Windows 7 WDK will be available as of WinHEC Windows Logo Program Web site: http://www.microsoft.com/whdc/winlogo/default.mspx http://www.microsoft.com/whdc/winlogo/default.mspx NDIS 6 Feedback alias: ndis6fb@microsoft.com ndis6fb@microsoft.com

Slide 17

Related Sessions SessionDay / Time Virtual Machine Queue Architecture ReviewTues. 2:45-3:45 Virtual Machine Queue Driver DevelopmentTues. 4-5 Driver ScalabilityMon. 11-12 and Tues. 11-12 Developing TCP Chimney Drivers for Windows 7Tues. 11-12 NDIS 6.20 OverviewMon. 9:45-10:45 and Tues. 9:45-10:45 Windows Logo Program Tests for NDISMon. 11-12 and Wed. 9:45-10:45 Discussion: Porting Drivers to NDIS 6.20Wed. 8:30-9:30 Lab: How to Debug an NDIS 6.20 DriverMon. 2:45-3:45 and Wed. 9:45-10:45