Windows Server Scalability Windows Server Scalability And Virtualized I/O Fabric And Virtualized I/O Fabric For Blade ServerFor Blade ServerChris PetteyChris PetteyCTO, NextIOCTO, NextIO

Son VoBaSon VoBaProgram Manager, Microsoft CorporationProgram Manager, Microsoft CorporationWindows Server Platform ArchitectureWindows Server Platform Architecture

AgendaAgenda

What is Shared I/O?What is Shared I/O?

Value of Shared I/OValue of Shared I/O

Architecture for I/O VirtualizationArchitecture for I/O Virtualization

User experienceUser experience

What Is Shared I/O?What Is Shared I/O?

Dedicated I/ODedicated I/O

Independent Operating Independent Operating Systems (OS)Systems (OS)

Each OS owns Each OS owns a physical adaptera physical adapter

Each OS owns Each OS owns the system components the system components to communicate to communicate to the adapterto the adapter

Example: Chipset and PCI Example: Chipset and PCI Express busExpress bus

OS #2 OS #3

I/OController

I/OController

OS #1

I/OController

Shared I/OShared I/O

Independent Operating Independent Operating Systems (OS)Systems (OS)

Each OS owns a virtualized Each OS owns a virtualized I/O controllerI/O controller

Corresponds to a shared Corresponds to a shared physical I/O controllerphysical I/O controller

Virtualization Enablers Virtualization Enablers control access control access to shared devicesto shared devices

I/OController

I/O

Controller

OS #2 OS #3OS #1

Virtualization Enablers

Characteristics Of Shared I/OCharacteristics Of Shared I/O

Multiple, independent operating systemsMultiple, independent operating systemsOSs do not coordinate with each otherOSs do not coordinate with each other

Each OS has a virtual set of resources to controlEach OS has a virtual set of resources to controlVirtual resources behave and interact in manners resembling Virtual resources behave and interact in manners resembling physical resourcesphysical resources

Single resources accessed by one or more OSsSingle resources accessed by one or more OSsSystem resources, e.g., Chipset, act as access points to I/OSystem resources, e.g., Chipset, act as access points to I/O

PCI Express acts as the connectivity from CPU and Chipset to I/OPCI Express acts as the connectivity from CPU and Chipset to I/O

I/O devices are simultaneously accessed by each OSI/O devices are simultaneously accessed by each OS

Virtualization Enablers control shared componentsVirtualization Enablers control shared componentsIsolate each OS for protectionIsolate each OS for protection

Provide virtual view of physical shared resourceProvide virtual view of physical shared resource

Manage functions for physical devicesManage functions for physical devices

Value Of Shared I/OValue Of Shared I/O

Shared I/O = Lower CostShared I/O = Lower Cost

CPU

CPU

PC

IeCPU

CPU

PC

IeEthernet

Fibre Channel Ethernet Switch

Fibre Channel SwitchCPU

CPU

PC

IeCPU

CPU

PC

Ie

Ethernet

Fibre Channel

CPU

CPU

PC

IeCPU

CPU

PC

Ie

Ethernet

Fibre Channel

InfiniBand

InfiniBand Switch

InfiniBand

InfiniBand

Shared I/O = Lower CostShared I/O = Lower Cost

Lower Acquisition costLower Acquisition cost

Lower TCOLower TCO

Lower Maintenance CostLower Maintenance Cost

Lower Upgrade CostLower Upgrade Cost

CPUCPU

CPUCPU PCIe

PCIe

CPU

CPU

PC

Ie

CPUCPU

CPUCPU PCIe

PCIe

CPU

CPU

PC

Ie

CPUCPU

CPUCPU PCIe

PCIe

CPU

CPU

PC

Ie

Shared I/OPCI Express

Switch

Shared Ethernet Controller

Shared Fibre Channel Controller

Shared InfiniBand Controller

New Technology?

Dedicated I/O Blade ServersDedicated I/O Blade Servers

10 Ethernet LOM chips10 Ethernet LOM chips

10 Fibre Channel 10 Fibre Channel oror InfiniBand Daughter InfiniBand Daughter CardsCards

4 Switches: 2 Ethernet Switches and 2 Fibre Channel 4 Switches: 2 Ethernet Switches and 2 Fibre Channel oror 2 InfiniBand 2 InfiniBand

Shared I/O Blade ServersShared I/O Blade Servers

2 Shared Ethernet chips2 Shared Ethernet chips

2 Shared Fibre Channel chips2 Shared Fibre Channel chips

2 Shared InfiniBand chips2 Shared InfiniBand chips

2 Shared I/O PCI Express Switches2 Shared I/O PCI Express Switches

Flexibility = Future ProofFlexibility = Future Proof

CPUCPU

CPUCPU PCIe

PCIe

CPU

CPU

PC

Ie

CPUCPU

CPUCPU PCIe

PCIe

CPU

CPU

PC

Ie

CPUCPU

CPUCPU PCIe

PCIe

CPU

CPU

PC

Ie

Shared I/OPCI Express Switch

Shared 1G Ethernet Controller

Shared 4G Fibre Channel Controller

Shared InfiniBand Controller

New Technology?

Shared 10G Ethernet Controller

Shared SAS Controller

Architecture For Architecture For I/O VirtualizationI/O Virtualization

Dedicated I/O Blade Server DesignDedicated I/O Blade Server Design

CPUCPU

CPUCPU PC

IeP

CIeCPU

CPU

Ethernet NIC

Fibre Channel HBA

Ethernet SwitchPC

Ie

CPUCPU

CPUCPU PC

IeP

CIeCPU

CPUFibre Channel HBA

Ethernet NIC

CPUCPU

CPUCPU PC

IeP

CIeCPU

CPU

Ethernet NIC

Fibre Channel HBABlade Server ChassisBlade Server Chassis

EnterpriseNetwork

Multiple intra-chassis fabricsMultiple intra-chassis fabrics

Fixed I/O configuration at orderFixed I/O configuration at order

Internal versus external switch Internal versus external switch compatibility concernscompatibility concerns

Management tools mustManagement tools mustcontemplate multiple fabricscontemplate multiple fabrics

PC

IeP

CIe Fibre Channel

Switch

FCFCFabricFabric

Shared I/O Blade Server DesignShared I/O Blade Server Design

CPUCPU

CPUCPU

CPU

CPU

CPUCPU

CPUCPU

CPU

CPU

CPUCPU

CPUCPU

CPU

CPU

Increased FlexibilityIncreased Flexibility

Simplified ManagementSimplified Management

Higher PerformanceHigher Performance

Lower CostLower Cost

Standard Standard PCI Express PCI Express

Mid-planeMid-plane Shared Ethernet NIC

Shared Fibre Channel HBA

Shared I/OPCI Express

Switch

Enhanced Enhanced PCI Express PCI Express

ProtocolProtocol

PC

IeP

CIe

PC

Ie

EnterpriseNetwork

FCFCFabricFabric

Blade Server ChassisBlade Server Chassis

Shared I/O In BladesShared I/O In Blades

Blades run independentlyBlades run independentlySoftware for OS, applications, etc. is unique for each bladeSoftware for OS, applications, etc. is unique for each blade

Blades have independent PCIe hierarchiesBlades have independent PCIe hierarchiesPCIe hierarchy is virtualizedPCIe hierarchy is virtualized

Root Complex, BIOS, etc. utilize PCIe as if it was dedicatedRoot Complex, BIOS, etc. utilize PCIe as if it was dedicated

Shared I/O components present virtualized componentsShared I/O components present virtualized componentsSwitches present multiple, virtual switchesSwitches present multiple, virtual switches

Controllers present multiple, virtual controllersControllers present multiple, virtual controllers

I/O Sharing through hardware componentsI/O Sharing through hardware componentsSwitches and Controllers combine to enable sharingSwitches and Controllers combine to enable sharing

Shared I/O In Virtual MachineShared I/O In Virtual Machine

Each Each OSOS runs independently runs independently

Virtual I/OVirtual I/O adapters appear adapters appear as physical componentsas physical components

Virtualization EnablerVirtualization Enabler is a combination of software is a combination of software and hardwareand hardware

Hypervisor softwareHypervisor software

PCIe Root Complex enablement, PCIe Root Complex enablement, e.g., Address Protection and e.g., Address Protection and Translation Table (ATPT) technologyTranslation Table (ATPT) technology

Physical SystemPhysical System is PCIe is PCIe I/O solutionI/O solution

Physical I/OPhysical I/O comprised of comprised ofSingle PCIe fabricSingle PCIe fabric

Dedicated I/O adaptersDedicated I/O adapters

Virtual Machine #1

Virtualization Enabler

Physical System

Virtual Machine #2

OS #2OS #1

Virtual I/O Virtual I/O

Physical I/O

Blade & Virtual Machine RelationshipBlade & Virtual Machine Relationship

SimilaritiesSimilaritiesEach OS is independentEach OS is independent

Legacy OSs Legacy OSs are fully supportedare fully supported

PCIe I/O systemPCIe I/O system

Single PCIe adapter Single PCIe adapter used by multiple OSsused by multiple OSs

Virtual adapters appear Virtual adapters appear as PCIe devicesas PCIe devices

DifferencesDifferencesVirtual Machine utilizes Virtual Machine utilizes software Hypervisor software Hypervisor for virtualizationfor virtualization

Blades utilize PCIe Switch Blades utilize PCIe Switch for virtualizationfor virtualization

Virtual Machine may use Virtual Machine may use traditional or Shared I/O traditional or Shared I/O enabled adaptersenabled adapters

Blades use Shared I/O Blades use Shared I/O enabled adaptersenabled adapters

Blade & Virtual Machine CombinationBlade & Virtual Machine Combination

Blade and Virtual Machine I/O Sharing are complementaryBlade and Virtual Machine I/O Sharing are complementary

Component solutions for Blades benefit Virtual MachineComponent solutions for Blades benefit Virtual Machine

Management model can service both Blades and Virtual MachinesManagement model can service both Blades and Virtual Machines

CPUCPU

CPUCPU

CPU

CPU

CPUCPU

CPUCPU

CPU

CPU

Shared Ethernet NIC

Shared FibreChannel HBA

Virt

ualiz

atio

n E

nabl

ers

OS

#2

OS

#3

OS

#1

Virt

ualiz

atio

n E

nabl

ers

OS

#2

OS

#3

OS

#1

PC

IeP

CIe

Shared I/OPCI Express

Switch

EnterpriseNetwork

FCFCFabricFabric

Blade Server ChassisBlade Server Chassis

Component Impact for Shared I/OComponent Impact for Shared I/O

BladeBlade Virtual MachineVirtual Machine

ChipsetChipset No impactNo impact ATPT functionsATPT functions

PCIe SwitchPCIe Switch Virtualized switchVirtualized switch No impactNo impact

I/O AdapterI/O Adapter Virtualized adapterVirtualized adapterNo impactNo impact

*Virtualized adapters provide *Virtualized adapters provide increased performanceincreased performance

OSOS No impactNo impact Hypervisor enabled OSHypervisor enabled OS

Guest OSGuest OS No impactNo impact No impactNo impact

User ExperienceUser Experience

Legacy OS and ApplicationsLegacy OS and Applications

Shared I/O fully supports legacy OS Shared I/O fully supports legacy OS and application softwareand application software

Applies equally to Blades and Virtual MachinesApplies equally to Blades and Virtual Machines

Blade solution fully supports all Microsoft OSBlade solution fully supports all Microsoft OS

No guest OS software introducedNo guest OS software introducedLegacy software is fully containedLegacy software is fully contained

No new “intermediate” drivers or 3No new “intermediate” drivers or 3rdrd party software party software

Guest OS and applications function normally Guest OS and applications function normally as in stand-alone serveras in stand-alone server

Migration from single server to Shared I/O is seamlessMigration from single server to Shared I/O is seamless

Devices And DriversDevices And Drivers

Blades support legacy devices dedicated to a single bladeBlades support legacy devices dedicated to a single blade

Virtual Machine enables software virtualization of legacy devicesVirtual Machine enables software virtualization of legacy devices

Blades and Virtual Machine enable Shared I/O enabled devices Blades and Virtual Machine enable Shared I/O enabled devices to be shared by multiple OSsto be shared by multiple OSs

Shared FibreChannel HBA

Shared Ethernet NIC

Dedicated Ethernet NIC

Dedicated Fibre Channel HBA

EnterpriseNetwork

FCFCFabricFabric

CPUCPU

CPUCPU

CPU

CPU

CPUCPU

CPUCPU

CPU

CPU

Virt

ualiz

atio

n E

nabl

ers

OS

#2

OS

#3

OS

#1

Virt

ualiz

atio

n E

nabl

ers

OS

#2

OS

#3

OS

#1

PC

IeP

CIe

Shared I/OPCI Express

Switch

Blade Server ChassisBlade Server Chassis

Management ModelManagement Model

Each OS manages its own virtual environmentEach OS manages its own virtual environmentBlades = Chipset, Virtual PCIe switch, and Virtual AdaptersBlades = Chipset, Virtual PCIe switch, and Virtual Adapters

Virtual Machine = Virtual Chipset and Virtual AdaptersVirtual Machine = Virtual Chipset and Virtual Adapters

Single control point manages common componentsSingle control point manages common componentsBlades = Switch Firmware for Switch and Shared I/O AdaptersBlades = Switch Firmware for Switch and Shared I/O Adapters

Virtual Machine = Hypervisor for all hardware componentsVirtual Machine = Hypervisor for all hardware components

Blade and Virtual Machine management can stageBlade and Virtual Machine management can stageBlade management partitions Switch and Adapters into virtual Blade management partitions Switch and Adapters into virtual PCIe systemsPCIe systems

Hypervisor partitions virtual PCIe system and chipset Hypervisor partitions virtual PCIe system and chipset into Virtual Machines into Virtual Machines

DeviceManagement

Tool

DeviceManagement

Tool

SwitchManagement

Tool

Common ManagementCommon Management

CPUCPU

CPUCPU

CPU

CPU

CPUCPU

CPUCPU

CPU

CPU

Shared I/OAdapter

Shared I/OAdapter

PC

IeP

CIe

Shared I/OPCI Express

Switch

Virtual Machine #1

Virtualization Enabler

Physical System

Virtual Machine #2

OS #2OS #1

Virtual I/O Virtual I/O

Physical I/O

Common Management CapabilitiesCommon Management Capabilities

Discovery of devices and capabilitiesDiscovery of devices and capabilitiesDiscovery, enumeration, etc. of PCIe devicesDiscovery, enumeration, etc. of PCIe devices

Partitioning of resourcesPartitioning of resourcesAssigning virtual resources to OSAssigning virtual resources to OS

Programming of shared functionsProgramming of shared functionsSetting operational parameters for physically Setting operational parameters for physically shared functionsshared functions

E.g., 10/100/1000 Ethernet link speedE.g., 10/100/1000 Ethernet link speed

Device specific functionDevice specific functionSupport for vendor specific device Support for vendor specific device management functionsmanagement functions

Common Management InterfaceCommon Management Interface

WS-ManagementWS-ManagementA DMTF Preliminary A DMTF Preliminary Standard defining Standard defining a web services a web services based protocolbased protocol

Is data model neutralIs data model neutral

A suitable management A suitable management protocol for both virtual protocol for both virtual devices and switchdevices and switch

DMTF CIMDMTF CIMProvides a consistent Provides a consistent abstraction of devices abstraction of devices and their virtual and their virtual environmentsenvironments

Independent of the Independent of the I/O Virtualization I/O Virtualization techniques and techniques and implementationsimplementations

Shared I/O and Industry StandardsShared I/O and Industry Standards

PCI SIG is standardizing elements PCI SIG is standardizing elements of I/O Virtualizationof I/O Virtualization

Virtual Machines and BladesVirtual Machines and Blades

Focused on I/O devices for PCIeFocused on I/O devices for PCIe

Microsoft and NextIO are active participants Microsoft and NextIO are active participants

Shared I/O leverages existing standardsShared I/O leverages existing standardsProtocols for Ethernet, FC, SAS, Protocols for Ethernet, FC, SAS, etc. unchangedetc. unchanged

Management models for DMTF, Management models for DMTF, etc. leveragedetc. leveraged

WinHEC Microsoft PavilionWinHEC Microsoft PavilionWinHEC Technical SessionWinHEC Technical Session

PCIe Switch In Blade PCIe Switch In Blade Server DesignsServer Designs

Technology DemoTechnology Demo

Shared I/O solution for Blade ServersShared I/O solution for Blade ServersPCI Express Switch with virtualization supportPCI Express Switch with virtualization support

Virtualized Fibre Channel HBAVirtualized Fibre Channel HBA

Unmodified Blade ChassisUnmodified Blade ChassisDell PowerEdge 1855 (Intel Xeon)Dell PowerEdge 1855 (Intel Xeon)

FSC PRIMERGY BX630 (AMD Opteron)FSC PRIMERGY BX630 (AMD Opteron)

Legacy supportLegacy supportNo change to OS or legacy driver No change to OS or legacy driver

No change to Chipset, Blades or ChassisNo change to Chipset, Blades or Chassis

Fibre ChannelSwitch

Technology DemoTechnology Demo

CPU

CPU

CPU

CPU

CPU

CPU

Fibre ChannelController

Fibre ChannelController

Fibre ChannelController

…Insert PCIe pass-thru cardsinstead of Fibre Channel Cards

PCI ExpressPass-thru

PCI ExpressPass-thru

PCI ExpressPass-thru

Note: The Fibre Channel midplanebecomes a PCI Express midplane

Hit EnterHit Enter

PC

IeP

CIe

PC

Ie

EthernetSwitch

EthernetController

EthernetController

EthernetController

NextIO PCIe Switch

Shared 4G Fibre Channel

PowerEdge 1855 or PRIMERGY BX630PowerEdge 1855 or PRIMERGY BX630

Shared I/O ValueShared I/O Value

Reduced component costReduced component cost

Reduced TCOReduced TCO

Increased flexibility Increased flexibility

Increase performanceIncrease performance

Zero impact to Legacy SoftwareZero impact to Legacy Software

Evolutionary solution starting in 2007Evolutionary solution starting in 2007

Call To ActionCall To Action

Plan for shared I/O in your next Blade Server designPlan for shared I/O in your next Blade Server design

Visit NextIO demo at the Microsoft PavilionVisit NextIO demo at the Microsoft Pavilion

Attend virtualization and management sessionsAttend virtualization and management sessionsDevice Virtualization Architecture (VIR040)Device Virtualization Architecture (VIR040)

How to Use the WMI Interfaces with Windows Virtualization (VIR043)How to Use the WMI Interfaces with Windows Virtualization (VIR043)

Hypervisor, Virtualization Stack, and Device Virtualization Architectures (VIR047)Hypervisor, Virtualization Stack, and Device Virtualization Architectures (VIR047)

PCIe Address Translation Services and I/O Virtualization (VIR071)PCIe Address Translation Services and I/O Virtualization (VIR071)

Windows Virtualization Best Practices and Future Hardware Directions (VIR124)Windows Virtualization Best Practices and Future Hardware Directions (VIR124)

Storage Management Directions (STO085)Storage Management Directions (STO085)

Windows Server Manageability Directions and Updates (SER120)Windows Server Manageability Directions and Updates (SER120)

Additional ResourcesAdditional Resources

Web ResourcesWeb ResourcesPCI-SIG (PCI-SIG (www.pcisig.orgwww.pcisig.org))

Distributed Management Task Force (Distributed Management Task Force (www.dmtf.orgwww.dmtf.org))

© 2006 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.

The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation.

MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.