WEB344WEB344Microsoft.com: Understanding Microsoft.com: Understanding IIS 6.0 Architecture Strategies for IIS 6.0 Architecture Strategies for High Availability High Availability

Casey Jacobs – Group Mgr, OperationsCasey Jacobs – Group Mgr, OperationsWayne King – Group Mgr, Web OperationsWayne King – Group Mgr, Web OperationsJeff Stucky – Debug Mgr, OperationsJeff Stucky – Debug Mgr, Operations

Microsoft.com OperationsMicrosoft.com OperationsMicrosoft CorporationMicrosoft Corporation

AgendaAgenda

Microsoft.com IntroductionMicrosoft.com Introduction

High Availability ArchitectureHigh Availability ArchitectureGlobal Solutions & NetworkingGlobal Solutions & Networking

Web & Database Hosting ModelsWeb & Database Hosting Models

64-Bit Adoption – Case Study64-Bit Adoption – Case StudyOverview of AdoptionsOverview of Adoptions

Key Takeaways of MigrationKey Takeaways of Migration

Lessons LearnedLessons Learned

Microsoft.com IntroductionMicrosoft.com Introduction

Vision, Strategy & InitiativesVision, Strategy & Initiatives

Vision:Vision: Achieve the highest availability on the Achieve the highest availability on the Internet while showcasing MS Technologies Internet while showcasing MS Technologies

Strategy:Strategy: Early Adopt Microsoft solutions, Early Adopt Microsoft solutions, provide positive feedback loop with Product provide positive feedback loop with Product Teams and showcase our Operational abilitiesTeams and showcase our Operational abilities

Do ItDo It

Prove ItProve It Share ItShare It

Know ItKnow It

Internet Hosting – Quick FactsInternet Hosting – Quick Facts

3 Internet Data Centers 3 Internet Data Centers

2 Content Delivery Networks2 Content Delivery NetworksAkamai & Savvis vendor relationshipsAkamai & Savvis vendor relationships

Cache Downloads & Product UpdatesCache Downloads & Product Updates

1606 Data Center servers1606 Data Center servers

500+ servers in Lab (Dev 500+ servers in Lab (Dev PPE) PPE)

109 Web Sites, 1069 DBs, 1000’s Apps109 Web Sites, 1069 DBs, 1000’s Apps

80+ Gigabit/sec Bandwidth80+ Gigabit/sec Bandwidth

Enterprise Web Solutions at ScaleEnterprise Web Solutions at Scale

Microsoft.com Microsoft.com 13M UUsers/Day & 70M Page Views/Day13M UUsers/Day & 70M Page Views/Day10k Req/sec & 300K ConConnections10k Req/sec & 300K ConConnectionsIIS ConfigurationsIIS Configurations

350 Virtual Roots350 Virtual Roots190 IIS Web Applications190 IIS Web Applications12 Application Pools12 Application Pools

Windows Update Windows Update 150M UClient scans/day (Interactive & AU)150M UClient scans/day (Interactive & AU)12k ASP.NET Req/sec12k ASP.NET Req/sec4 Gb/sec Web site egress (Web pages only…4 Gb/sec Web site egress (Web pages only…doesn’t include the downloads!)doesn’t include the downloads!)

Web Site Availability BenchmarkingWeb Site Availability BenchmarkingMeasured by Keynote Systems, Inc.Measured by Keynote Systems, Inc.

 Year To DateYear To DateRank Site Avail %

1 Microsoft.com 99.81

2 WindowsUpdate 99.80

3 AOL 99.77

4 Google 99.75

5 IBM 99.56

6 Yahoo 91.50

7 Oracle 87.67

8 SUN 80.88

Q:Q: Who is Keynote? Who is Keynote?

A:A: They provide monitor They provide monitor

agents worldwide which agents worldwide which

‘ ‘ping’ systemsping’ systems

Q:Q: How do we use the data? How do we use the data?

A:A: RealTime & Trending RealTime & Trending

* 3 Yrs at #1 for Microsoft.com* 3 Yrs at #1 for Microsoft.com

** Top 10 MSCOM web sites out ** Top 10 MSCOM web sites out

ranked Benchmarks in 2004ranked Benchmarks in 2004

High Availability ArchitectureHigh Availability Architecture

Data Center 1 Data Center 2

Cisco Guard Devices- Packet Filtering

- Anomaly Tracking

Routing Rule Sets - ACL’s (80 & 443 Only)

- Dedicated LANs for Unique Req’s

Internet

`

Clients

CDN Partnerships - Edge Cache: Akamai & Savvis - Cluster Load Balancing - FirstPoint (Akamai) - ITM (Savvis)

High Availability ArchitectureHigh Availability ArchitectureGlobal Solutions & NetworkingGlobal Solutions & Networking

High Availability Architecture High Availability Architecture Global Solutions & NetworkingGlobal Solutions & Networking

Global SolutionsGlobal SolutionsAkamai & Savvis – Content CachingAkamai & Savvis – Content Caching

Global Load Balancing – Web Cluster Level MgmtGlobal Load Balancing – Web Cluster Level Mgmt

Network InfrastructureNetwork InfrastructureCisco Guards = Anomaly Detection & DOS FiltersCisco Guards = Anomaly Detection & DOS Filters

Attack Rates > 70x Normal TrafficAttack Rates > 70x Normal Traffic

4k Req/Sec vs Attack 280k Req/Sec/Server4k Req/Sec vs Attack 280k Req/Sec/Server

Cisco Router ArchitectureCisco Router Architecture

HTTP/S Only – Exceptions Require ReviewHTTP/S Only – Exceptions Require Review

““Cookie Cutter” - Simple, Scalable & ManageableCookie Cutter” - Simple, Scalable & Manageable

Agility - Quickly Swap LANs between Web SitesAgility - Quickly Swap LANs between Web Sites

Data Center 1 Data Center 2

Online Online

Read OnlyNLB

Online Offline

Read/WriteNLB

Log Ship

(Synch’d Copies)

WebServicesNLB Cluster

WebServicesNLB Cluster

Log ShipSecondary

Log Ship Secondary

Log Shipping Synch’s (Contingency Tactics)

Online OnlineDuplicates

Cluster 1 Cluster 5 Cluster 6Cluster 3Cluster 2 Cluster 4

ASP.NET, ASP, HTML NLB Load Balancing 3 Servers Min/Cluster Synch’d Code & Content

High Availability ArchitectureHigh Availability ArchitectureWeb & Database HostingWeb & Database Hosting

High Availability Architecture High Availability Architecture Web & Database HostingWeb & Database Hosting

Network Load Balancing (NLB) ClustersNetwork Load Balancing (NLB) ClustersServer Cluster Sizes: 3 – 8 Servers/ClusterServer Cluster Sizes: 3 – 8 Servers/Cluster

Positives: Easy Mgmt & Free with Windows SKU’sPositives: Easy Mgmt & Free with Windows SKU’s

Challenges: Switch Overhead & Connection AffinityChallenges: Switch Overhead & Connection Affinity

Server ConfigurationsServer ConfigurationsBaseline O/S, IIS, ASP.NET Configurations IdenticalBaseline O/S, IIS, ASP.NET Configurations Identical

Application Code & Content Unique per SiteApplication Code & Content Unique per Site

Standard Hosting ModelsStandard Hosting ModelsAgility = Quickly reallocate from System to SystemAgility = Quickly reallocate from System to System

Efficiency = Less Staffing & Equipment RequiredEfficiency = Less Staffing & Equipment RequiredConsistent ConfigurationsConsistent Configurations

Repeatable Infrastructure ArchitectureRepeatable Infrastructure Architecture

64-bit Adoption – Case Study64-bit Adoption – Case Study

Early Adoption ObjectivesEarly Adoption Objectives1 Year Runtime prior to Win2k3 SP11 Year Runtime prior to Win2k3 SP1

Began May 21, 2004Began May 21, 2004

100% Adopted by April 2005100% Adopted by April 2005

Identify Memory Management ImprovementsIdentify Memory Management Improvements

System & App Migration Gotcha’sSystem & App Migration Gotcha’s

Historical ChallengesHistorical ChallengesWin2k/IIS5 – Memory limits hosting ASP & ASP.NET Win2k/IIS5 – Memory limits hosting ASP & ASP.NET with no memory based recycling for protectionwith no memory based recycling for protection

Win2k3/IIS6 – App Pools helped Isolation protection, Win2k3/IIS6 – App Pools helped Isolation protection, but memory recycling intervals continually shortenedbut memory recycling intervals continually shortened

Windows Server 2003 x64 Edition Windows Server 2003 x64 Edition Adoption OverviewAdoption Overview

64-bit Migration Strategy64-bit Migration Strategywww.Microsoft.com Hardwarewww.Microsoft.com Hardware

HardwareHardware1.1. Proof of concept testing with AMD supplied reference Proof of concept testing with AMD supplied reference

hardwarehardware

2.2. Purchased 64-bit capable HardwarePurchased 64-bit capable Hardware

Ability to host both x86 & 64-bit Windows O/SAbility to host both x86 & 64-bit Windows O/S

Analyze vendor components: Proc’s & MemoryAnalyze vendor components: Proc’s & Memory

3.3. Install x86 Windows (Windows 2003)Install x86 Windows (Windows 2003)

Incrementally test S/W stack on new H/W platformIncrementally test S/W stack on new H/W platform

4.4. Migrated to 64-bit WindowsMigrated to 64-bit Windows

Use same H/W as (2) and (3) were validatedUse same H/W as (2) and (3) were validated

Allowed for phased migration while testing App’s Allowed for phased migration while testing App’s on the new 64-bit H/W without impacton the new 64-bit H/W without impact

64-bit Migration Strategy64-bit Migration Strategywww.Microsoft.com Softwarewww.Microsoft.com Software

Software ApproachSoftware Approach1.1. Rely on x86 code not yet ported to 64-bitRely on x86 code not yet ported to 64-bit2.2. Expect Zero App Dev – x86 IIS on WoW64 Expect Zero App Dev – x86 IIS on WoW64

allows for Transparent Migrationallows for Transparent MigrationASP.NET 1.1 App’s, x86 ISAPI Extensions & FiltersASP.NET 1.1 App’s, x86 ISAPI Extensions & FiltersKB 895976: KB 895976: “How to run 32-bit Worker Processes on 64-“How to run 32-bit Worker Processes on 64-bit”bit”

3.3. Target 4GB VM per ProcessTarget 4GB VM per Process32-bit Processes under WoW64 get full 4 GB of VM32-bit Processes under WoW64 get full 4 GB of VMFull benefits of 64-bit Kernel if running 32-bit Apps Full benefits of 64-bit Kernel if running 32-bit Apps in WOW64in WOW64

* Helped solve VM crunch we were running into on x86* Helped solve VM crunch we were running into on x86** Don’t Sacrifice Kernel Memory Pools for extra Virtual** Don’t Sacrifice Kernel Memory Pools for extra Virtual Memory into UserModeMemory into UserMode

Hardware Platform Hardware Platform Comparative Study: x86 vs x64Comparative Study: x86 vs x64

Key Take Away'sKey Take Away'sHardware Costs – lower cost per server for x64Hardware Costs – lower cost per server for x64

Hardware capable of running both x86 & x64 O/SHardware capable of running both x86 & x64 O/S

Greater CPU and Memory capacities lend towards Greater CPU and Memory capacities lend towards further Consolidation of systems or ReAllocationsfurther Consolidation of systems or ReAllocations

x86x86 x64x64

Hardware ModelHardware Model HP DL580HP DL580 HP DL585HP DL585

ProcessorProcessor 4 x 2.0 GHz (Intel)4 x 2.0 GHz (Intel) 4 x 2.2 GHz (AMD)4 x 2.2 GHz (AMD)

RAMRAM 4 GB4 GB 16 GB16 GB

OS VersionOS VersionWindows Server Windows Server 2003 Enterprise 2003 Enterprise Edition SP1Edition SP1

Windows Server Windows Server 2003 Enterprise 2003 Enterprise x64 Editionx64 Edition

CPU Utilization per Platform CPU Utilization per Platform Comparative Study ResultsComparative Study Results

x86x86 x64x64

HTTP Req/SecHTTP Req/Sec CPU %CPU % HTTP Req/SecHTTP Req/Sec CPU %CPU %

222222 65%65% 216216 35%35%

Key TakeAway'sKey TakeAway'sHuge Gains due to 64-bit H/W & Windows PlatformsHuge Gains due to 64-bit H/W & Windows PlatformsSeamless migration provided with WoW64 Seamless migration provided with WoW64 Enabled Enabled www.Microsoft.comwww.Microsoft.com to leverage saved to leverage saved infrastructure to enable Data Center Redundancyinfrastructure to enable Data Center RedundancyApp Pool Recycles Eliminated – Enjoying the new 4GB App Pool Recycles Eliminated – Enjoying the new 4GB VM address space running under WoW64!!VM address space running under WoW64!!Enabled more App Pools driving further Isolation of Enabled more App Pools driving further Isolation of Code & Content in shared hosting modelsCode & Content in shared hosting models

Average Server Response TimesAverage Server Response TimesComparative Study ResultsComparative Study Results

x86x86 x64x64

RequestRequestTypeType Req/secReq/sec ResponseResponse

Time (ms)Time (ms) Req/secReq/sec ResponseResponseTime (ms)Time (ms)

ASPASP 7.857.85 244244 7.417.41 5353

ISAPIISAPI 110.85110.85 248248 125.43125.43 1818

StaticStatic 41.941.9 135135 31.0131.01 33

Static (cached)Static (cached) 47.1147.11 11 54.5154.51 11

Key Take Away'sKey Take Away'sSignificantly Improved Response Times/RequestsSignificantly Improved Response Times/Requests

Explanation – Combo due to relieved CPU burdens Explanation – Combo due to relieved CPU burdens and increased Physical and Virtual Memoryand increased Physical and Virtual Memory

Worst Performing Web PagesWorst Performing Web PagesComparative Study ResultsComparative Study Results

x86 (Seconds)x86 (Seconds) X64 (Seconds)X64 (Seconds) Performance GainPerformance Gain

79.3 79.3 5.1 5.1 15.5x15.5x

53.5 53.5 4.7 4.7 11.3x11.3x

49.4 49.4 2.82.8 17.7x17.7x

47.7 47.7 2.72.7 17.4x17.4x

44.8 44.8 2.62.6 17.4x17.4x

Key Take Away'sKey Take Away'sHUGE drop in Page delivery for worst App’sHUGE drop in Page delivery for worst App’s

Systems no longer required to queue or keep open Systems no longer required to queue or keep open connections for poor performing App’sconnections for poor performing App’s

* Server Performance Advisor (SPA) used to analyze data* Server Performance Advisor (SPA) used to analyze data

32-bit vs 64-bit Performance32-bit vs 64-bit Performance

64bit Adoption – www.Microsoft.com64bit Adoption – www.Microsoft.comTop Lessons LearnedTop Lessons Learned

WoW64 Redirection BehaviorsWoW64 Redirection BehaviorsFile Sys Redirection for 32-bit Apps File Sys Redirection for 32-bit Apps windows\windows\system32 vs. windows\syswow64system32 vs. windows\syswow64

Registry Redirection Registry Redirection HKLM\Software\Wow6432NodeHKLM\Software\Wow6432Node

Script Host Gotcha’s (cscript.exe/wscript.exe)Script Host Gotcha’s (cscript.exe/wscript.exe)Scripts depending on x86 com objects will require Scripts depending on x86 com objects will require x86 scripting hosts (located in \SysWow64)x86 scripting hosts (located in \SysWow64)Scripts using x86 scripting host need to be aware of Scripts using x86 scripting host need to be aware of WoW64 Redirection behaviors (see Above)WoW64 Redirection behaviors (see Above)

64bit Adoption – www.Microsoft.com64bit Adoption – www.Microsoft.comTop Lessons Learned (cont’d)Top Lessons Learned (cont’d)

Verification of 3Verification of 3rdrd Party App Compatibility Party App CompatibilityCommon Admin Tools: Regmon, Filemon, etc.Common Admin Tools: Regmon, Filemon, etc.

Required 64-bit flavor of device drivers or filter Required 64-bit flavor of device drivers or filter drivers that run in the kerneldrivers that run in the kernel

Antivirus SoftwareAntivirus Software

Backup SoftwareBackup Software

Imaging/Deployment SoftwareImaging/Deployment Software

www.Microsoft.com 64-bit Adoption www.Microsoft.com 64-bit Adoption Future DirectionFuture Direction

ASP.NET 2.0 Beta 2 – Early AdoptionASP.NET 2.0 Beta 2 – Early Adoptionhttp://www.Microsoft.com/Downloadshttp://www.Microsoft.com/DownloadsASP.NET introduces 64-bit flavorASP.NET introduces 64-bit flavorASP.NET 1.1 is the current dependency for ASP.NET 1.1 is the current dependency for having to run 32-bit IIS in WoW64.having to run 32-bit IIS in WoW64.Looking forward to start testing native 64bit Looking forward to start testing native 64bit configuration (running full 64-bit IIS)configuration (running full 64-bit IIS)

Getting started with SQL server deploymentsGetting started with SQL server deploymentsWorking on getting some SQL 2000 servers Working on getting some SQL 2000 servers online to leverage 4 GB Virtual Memory benefitsonline to leverage 4 GB Virtual Memory benefitsYukon Early Adoption: Full native 64-bit SQL Yukon Early Adoption: Full native 64-bit SQL setup with SQL Server 2005setup with SQL Server 2005

ResourcesResources

Online Webcasts: http://IISWebCastSeries.com

TechNet Online: http://Microsoft.com/Technet

ASP.NET Sites: http://MSDN.Microsoft.com

http://GotDotNet.com

http://ASPNET.com

Your FeedbackYour Feedbackis Important!is Important!Please Fill Out a Survey forPlease Fill Out a Survey forThis Session on CommNetThis Session on CommNet

© 2005 Microsoft Corporation. All rights reserved.This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.