automated performance profiling with continuous integration

W11 Concurrent Class

10/2/2013 1:45:00 PM

"Automated Performance

Profiling with Continuous

Integration"

Presented by:

Ivan Kreslin

Mitchell International Inc

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Ivan Kreslin

Mitchell International

Ivan Kreslin is a principal software development engineer at Mitchell International, North

America's leading provider of property and casualty claims technology solutions for most major

insurance providers and collision repair shops. Ivan leads the test automation development

efforts in the auto casualty solutions division, creating tools and architecting and integrating the

tools into automation frameworks and libraries shared among the different feature teams. Ivan

brings more than twenty-one years of software industry experience, most focused on quality and

automation.

Automated Performance Profiling 9/20/2013

1

Automated Performance Profiling in Continuous Integration

Ivan Kreslin Principal Software Development Engineer

Confidential and Proprietary. ©2013 Mitchell International, Inc.

Automated Performance Profiling

Overview • Historically performance tests

have been run too late

• Learn how to capture regressions in performance during Continuous Integration (CI) by using Microsoft’s Performance Profiling tools functionality integrated into a test automation framework.

– A test framework that enables existing and new test automation to be used for both functional testing and performance profiling purposes.

http://msdn.microsoft.com/en-us/library/ms182401.aspx


2



Performance Profiling in Visual Studio • Before attempting to automate performance

profiling, try it out within Visual Studio.



Performance Profiling w/Command Line • For test automation, the command line tools are used.


3



Data Collection Method • Three main ways for collecting performance

profiling data:

– CPU Sampling

– Instrumentation (i.e. trace).

– Concurrency

• Our automation framework uses Instrumentation in order to obtain detailed timing data each time a function gets called.



Binary Instrumentation • Performance Profiling requires

instrumented binaries.

• The Profiling Tools instrument binaries by injecting timing info at start and end of each function.

– Related symbol files are also required.

– Setting of Environment Variables on test machine:

• PATH, NT_SYMBOL_PATH


4



Visual Studio Profiling (VSP) Data Files

• A vsp file is generated by the profiling tools after each test run.

– The vsp file collects function timing details.

• Vsp files can get relatively large, growing whenever functions are called.

– Running of our current tests result in files

ranging from 25MB to 500MB

• If changes are made to a test affecting the total performance then a new baseline needs to be captured for usage in any future test runs.



Baseline Data • To determine if regressions in performance have

occurred we need a baseline to compare against.

– To capture baseline data: We run the test same way as we plan to do later in continuous integration.

• The baseline vsp gets stored in TFS and can then be used in then test runs.


5



Determining Success of a Test Run

• Time spent within calls to instrumented binaries is compared against the baseline.

• If total time is greater than a specified % allowed by the test -> test will be logged as having failed.

– We chose this logic as some relatively high percent diffs can often be just fractions of a ms.



Creating the Automated Tests • Automation Engineers create tests as they would

any functional test.

– All referencing a common Automation Framework.

• During test initialization, the tests config xml is checked to determine if performance profiling is to be used, the xml also specifies additional needed info such as:

– TFS branch location of code being tested

– List of binaries to be instrumented

– Path to baseline vsp file for the test…


6



Running the Tests • Tests are run using the latest CI release build.

• Tests can be run like any other framework referencing Automated test.

– From Visual Studio, MTM, mstest.exe or Part of the build process as is the case with our tests



Profiling Test Execution Overview

• Automation framework creates and executes 2 bat files used during by the process

(1) Binary instrumentation

(2) Runs Perf Profiling commands

& launches application/service

• Diff report is inserted into a sql table

• Timing Data is analyzed and success/fail is determined

• Build/Test completes and report is generated


7



Profiling Tool Command Details

• 1st step instruments the binaries, e.g.

– VSInstr AnalyzeBridge.dll…

• 2nd step launches app and runs Perf tools, e.g.

– VSPerfCLREnv /TraceOn

– VSPerfCmd /Start:Trace /Output:<VSP>

• ASP.NET apps use VSPerfAspNetCmd.exe instead

– <Application.exe> ( or net start <ServiceName>)

– VSPerfCmd /shutdown

– VSPerfReport /output & /diff



Test Environment

• State of machines/db’s… used to run profiling tests needs to be constant between test runs.

• Use Physical machines if VM’s are inconsistent

– Testing with VM’s showed performance varying up to 20% run to run vs. 2% for physical machines.

• Time to complete running test depends on:

– Amount & number of times functions are executed.

– Our libraries enable running a test multiple times to obtain a median time & reduce false positives.


8



Reports • After a test run has completed a report is automatically

generated and sent out with details of the run.

– Report lists functions with the greatest difference vs. baseline along with recent TFS check-in info.

– Any regressions get assigned to appropriate developers

• Reports can be customized whichever

way teams wish/need them to be.



Sample Report Table


9



Steps To Get Started 1. Validate your application can be profiled

2. Integrate performance profiling tools execution and data/TFS handling into test automation framework

3. Create/Identify:

– Automated tests for profiling usage

– List of Application binaries to be instrumented

– Baseline vsp file(s).

– Test environment: machine/db… as needed

– TFS build definition used to build & deploy the

application + build & run the tests.



Demo


10



Questions? • Contact: [email protected]

mailto:[email protected]

automated performance profiling with continuous integration

Technology

test test

performance tests

performance profiling

total performance

performance profiling

test confidential

test framework

test automation framework