WINDOWS AZUREACID TEST!

DDD SYDNEY, 2 JULY 2011

Matt Done, expanz Pty. Ltd.

Agenda

Presentation Overview What is Azure and SQL Azure Goals of the Load Test Testing Setup Test Results Cost Comparisons Live Demo Questions

Presentation Overview

1. Load testing in Windows Azure and the various benefits it brings2. Horizontally Scalable, Stateful computing.3. High performance WCF (Windows Communications Foundation)

Topics

Intended Audience

1. Enterprise Architects, Solution Architects2. Developers involved in large scale applications3. People who love Azure!!

What is Windows Azure• Microsoft Cloud Based Platform as a Service (PaaS)

What is SQL Azure• Microsoft Cloud Based Database as a Service (DBaaS)

Goals of the load test• To prove the efficiency of the expanz Platform under load

of 10000+ concurrent users.• Have an environment to tweak performance settings and

to find potential bottlenecks.• To assess how the Azure platform performs under load.• To assess how SQL azure performs under load.

Application Architecture

Firewall (Internal Endpoints)

Firewall (Access to DB is configurable)

SQL AzureCluster

Application Server Cluster

(Session Manager)

Application Server Load

Balancer(Site Manager)

Web Services(WCF)

Azure Load Balancer

Web Role Load Balancing Layer (Port 80/443)

Application Server Load Balancing (net.tcp endpoints)

Azure Table Storage

Sticky Sessions Once

EstablishedSession Creation

Load Testing Architecture

Azure Cloud

Banking Application

Load Test Client

Load Test Co-ordinator

Service

Load Test Load Balancer

Load Testers

Europe

Load Testers

South Asia

Load Testers

Central US

Load Test Load Balancer

Load Testers

Central Asia

Load Test Load Balancer

Load Test Load Balancer

Azure Load Test Costs

Azure Service Geographic LocationInstance

TypeTotal Cores Database

GB of Trans

Compute Cost

Trans Cost DB Cost

Hours Run

Total Cost

Platform South Central US Medium 76 1 1 $ 9.12 $ 0.10 $ 0.33 1 $ 9.55 Test Instance1 Southeast Asia Medium 12 0 1 $ 1.44 $ 0.10 $ - 1 $ 1.54 Test Instance2 North Europe Medium 12 0 1 $ 1.44 $ 0.10 $ - 1 $ 1.54 Test Instance3 East Asia Medium 12 0 1 $ 1.44 $ 0.10 $ - 1 $ 1.54 Test Instance4 Anywhere US Medium 12 0 1 $ 1.44 $ 0.10 $ - 1 $ 1.54

Totals 124 5 $ 11.04 $ 0.50 $ 0.33 Total $ 15.71

Windows Azure Pricing

Compute Instance

CPU (GHz) Memory Storage Bandwidth Instance Cost

Trans per GB

App Fabric per

1,000,000XSmall 1 768mb 20GB 5 Mbps $ 0.06 $ 0.10 $ 1.99 Small 1.6 1.7GB 225GB 100 Mbps $ 0.12 $ 0.10 $ 1.99 Medium 2 x 1.6 3.5GB 490GB 200 Mbps $ 0.24 $ 0.10 $ 1.99 Large 4 x 1.6 7GB 1000GB 400 Mbps $ 0.48 $ 0.10 $ 1.99 XLarge 8 x 1.6 14Gb 2040GB 800 Mbps $ 0.96 $ 0.10 $ 1.99

SQL Azure Pricing

Database SizePrice Per

Day1GB $ 0.33 5GB $ 1.66 10GB $ 3.66 20GB $ 7.33 30GB $ 10.99 40GB $ 14.65 50GB $ 18.31

Load Testing Setup Cost

Total Cost = $15.71 per hour

Local Debug

Local/Remote Debug

1. Install SSH server (Putty, BitVise)2. Start azure compute emulator3. Start local client4. Start SSH client and port forward to Web service port on remote

machine5. Start testing

Problem

• Need to test and locally debug• Azure will not accept remote connections

Solution

(Very) Simple Banking App

Load Test SetupProcess 1 Process 2 Process 3 Process 4 Process 5

Display Transactions Transfer Money Pay New Biller Pay Existing Biller Update User Details

Select display transactions Select transfer money menu item Select Pay Bill menu item Select Pay Bill menu item Select update address details menu item

Select an account Check the from account is selected Select from account Select from account Select account to change details for

Return 50 transactions for the selected account Check the to account is entered Enter biller id Enter biller id Check if logged in user can change

address details

Return 100 Transactions for a selected account

Check the "To Description" field is entered and the value conforms to a given mask and is a max length (15 chars)

Validate newly entered biller Select existing biller from list User updates address details

EndCheck the "From Description" field is entered and the value conforms to a given mask and is a max length (15 chars)

Enter Biller Reference code Enter Numeric amount System validates postcode/state/country

Check amount is entered and is numeric Enter Numeric amount Execute pay bill System updates address

User clicks transfer Execute pay bill Validate Amount and reference for numeric and length Return new address details and refresh

System checks if user is within daily limit Validate Amount and reference for numeric and length Check if user within daily limit End

System checks user has enough money Check if user within daily limit Check if has enough money

System withdraws money from account Check if has enough money Remove money from account

Reference number (GUID) is assigned Add biller to users billers Return transaction details

Return transaction details. Remove money from account End

End Return transaction details

End

Database Setup

Table Name # Records Reserved (KB) Data (KB) Indexes (KB)dbo.Accounts 20,003 1,672 1,608 16

dbo.Billers 1,002 64 48 16

dbo.BillingAddressBooks

20,015 2,128 1,624 304

dbo.Transactions 4,001,404 728,224 539,752 188,232

dbo.Users 10,003 1,224 1,184 16

Table Sizing

Indexes• Non Clustered Index on [dbo].[Transactions], Columns account_id, [transactiondate]• Non Clustered Index on [dbo].[Transactions], Columns [billingaddressbook_id]• Non Clustered Index on [dbo].[Transactions], Columns [account_id]• Non Clustered Index on [dbo].[BillingAddressBooks], Columns [user_id]

Considerations

• Make sure the Database resides at the same location as your application

Load Test Execution• Record trace of required processes• Allocate percentage execution for each process• Determine wait time figure (100ms or 1 sec)? • Each execution waits a random amount of time, based on processExecutionDelay.

Thread.Sleep(myRandom.Next(processExecutionDelay)); • Each User session is a single thread. 2500 users = 2500 threads.

<LoadTester> <LoadTest WebServerURI="http://esaloadtester.cloudapp.net/LoadTester.svc/binary" UserIDStart="4" UserIDEnd="2500" CreateSessionGUID="c97c9dda-8148-49f2-9085-7460338d07d3" ReleaseSessionGUID="6c81b864-d653-439a-afb7-e6a875b62e7a" TotalUserIdChars="6" RuntimeSeconds="3600" RampUpStep="250" RampUpInterval="60" ProcessExecutionDelay="1000"> <Processes> <Process Name="Display Transactions" GUID="9481c4e3-3fcb-47b5-a8f0-2987cff3caac" Percentage="32"/> <Process Name="Transfer Money" GUID="4b1f1659-bae2-4d7b-b2b4-7e78ad18d864" Percentage="22"/> <Process Name="Pay an New Biller" GUID="cd2243e2-4645-4956-bdc0-84087390fabd" Percentage="23"/> <Process Name="Pay an Existing Biller" GUID="2c9af3cb-da1f-4b6e-91ef-b382c4764bad" Percentage="12"/> <Process Name="Update User Address" GUID="9b337b9f-6032-4dd3-b07d-9c0c2fec6454" Percentage="11"/> </Processes> </LoadTest> <LoadTest WebServerURI="http://esaloadtester2.cloudapp.net/LoadTester.svc/binary" UserIDStart="2501" UserIDEnd="5000" CreateSessionGUID="c97c9dda-8148-49f2-9085-7460338d07d3" ReleaseSessionGUID="6c81b864-d653-439a-afb7-e6a875b62e7a" TotalUserIdChars="6" RuntimeSeconds="3600" RampUpStep="250" RampUpInterval="60" ProcessExecutionDelay="1000"> <Processes> <Process Name="Display Transactions" GUID="9481c4e3-3fcb-47b5-a8f0-2987cff3caac" Percentage="32"/> <Process Name="Transfer Money" GUID="4b1f1659-bae2-4d7b-b2b4-7e78ad18d864" Percentage="22"/> <Process Name="Pay an New Biller" GUID="cd2243e2-4645-4956-bdc0-84087390fabd" Percentage="23"/> <Process Name="Pay an Existing Biller" GUID="2c9af3cb-da1f-4b6e-91ef-b382c4764bad" Percentage="12"/> <Process Name="Update User Address" GUID="9b337b9f-6032-4dd3-b07d-9c0c2fec6454" Percentage="11"/> </Processes> </LoadTest> </LoadTester>

Web Service Request/Response Capture

Design Objectives

• Do not log to SQL Azure database• Do not log to file system• Preserve thread pool threads for load test execution• Preserve bandwidth and CPU as little as possible

Solution

• Use SQL Azure table storage• Capture Request/Responses and send to Site Manager• Site Manager will process and commit records• Use actual threads and not thread pool threads

Issues

• Batch table storage requests in lots of 20-50 and use SaveChangesWithRetries

• Use Thread.Start instead of  ThreadPool.QueueUserWorkItem • Datetime.Now only accurate to 15-20ms. Use StopWatch class

Capturing Diagnostics in Azure• Azure has an API for logging diagnostic information to

TableStorage• Counter names are normal windows counter. Eg @"\

Processor(_Total)\% Processor Time" , @"\Memory\Available MBytes"

public void StartDiagnostics() { TimeSpan perfSampleRate = System.TimeSpan.FromSeconds(5); DiagnosticMonitorConfiguration diagConfig = DiagnosticMonitor.GetDefaultInitialConfiguration(); diagConfig.PerformanceCounters.ScheduledTransferPeriod = TimeSpan.FromSeconds(30); foreach (string diag in GetDiagnostics()) { //Add diagnostics element diagConfig.PerformanceCounters.DataSources.Add( new PerformanceCounterConfiguration() { CounterSpecifier = diag, SampleRate = perfSampleRate }); } DiagnosticMonitor.Start(dsm.myStorageAccount, diagConfig); }

• Data can be viewed with Cerebrata Diagnostics Studio (Demo)

Load Test Results (First Cut)

• 5 Large Web Services• 1 Large Site Manager• 5 Large Session ManagersLarge Instance = 4 x 1.6 GHz cpu’s and 7 GB of ram

• 1 Extra Large Instance• 4 Geographic locations

Run duration = 32 MinutesTotal Transactions = 1.89 MillionAverage Number of Transactions per Minute = 59767Average WS Response Time = 70msAverage CPU Utilisation (App Servers) = 30%Min Memory (App Servers) = 4Gb

Test Result Summary

Application Instances Setup Load Test Instance Setup

NOT GOOD ENOUGH. NEED MORE!!!

Load Test First Cut Investigation

• WCF Web services are not multithreaded by default[ServiceBehavior(ConcurrencyMode = ConcurrencyMode.Multiple)]

• When enabling multiple concurrency, throttling is important <serviceThrottling maxConcurrentSessions="400"  maxConcurrentInstances="400" maxConcurrentCalls="400" />

• When using Channel Factories, max number of open concurrent connections is 10.

• Adjust max concurrent connections when using TCP or Named pipe bindings, is 2.<system.net>   

  <connectionManagement>       <add address="*" maxconnection="500"/>     </connectionManagement> </system.net>

• Azure load balancer is not throttling connections. Throttle Connections using ServicePointManager.DefaultConnectionLimit in Web Role

• Avoid closing factory connections in a finally block. Or do both.

The channel factory, factory

• Limit of 10 open connections.• For singleton to singleton operations, comms is problematic• Aborted and faulted channels

Problem

Solution

• Create a factory of channel factories (pooling)• Needs to be Muti-threaded.

Couple of Cool and not so cool things (Code)

• Parallel Linq (PLinq)• To get a Service Channel Guid Use the following

((IClientChannel)channel).SessionId • Concurrent Collections. Not so cool.

Run 2,3,4…

• 10 Large Web Services• 1 Extra Large Site Manager• 8 Large Session ManagersLarge Instance = 4 x 1.6 GHz cpu’s and 7 GB of ram

• 5 Medium Instances• 4 Geographic locations

Run duration = 30 MinutesTotal Transactions = 7.8 MillionAverage Number of Transactions per Minute = 161245Max Transactions per Minute = 211589Error Rate = 0.014%Average Web Service response time = 39msAverage CPU Utilisation (App Servers) = 70%Min Memory (App Servers) = 4Gb

Test Result Summary

Application Instances Setup Load Test Instance Setup

Total Cost = $12.88 per hour

Headline Results

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 330

5

10

15

20

25

30

35

40

45

50Average Response Times (ms)

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 330

50000

100000

150000

200000

250000

Transactions per minute

Time lineDefault WCF Settings. 5 Instances

Default WCF Settings. 10 Instances

Multithreaded Service. Per Session Context

Create Channel Factory Factory

Raise tcp Connections limit

Adjust Instance Counts

1 2 3 4 5 60

50000

100000

150000

200000

250000

Demo

• Live Diagnostics• Live Table Storage• Live working client

• Please find full recording of my session atwww.youtube.com/expanzTV

Thank you for following my presentation!

Please take the time to have a look at the next slide with an exclusive invitation for DDD attendees from expanz as a DDD sponsor.

During our Private Beta Program expanz are looking for thought leading developers for trialling an exclusive preview of their next generation Line Of Business (LOB) Application Development Platform, the ‘expanzPLATFORM 2011’

As a DDD Melbourne, Sydney or Adelaide attendee, you qualify for participating in our Private Beta program. To join and receive your free perpetual license after launch, please email us at beta@expanz.com with ‘DDD Attendee’ in your subject line before 22 July 2011.

BETA

Perpetual License exclusively for

DDD Attendees