testing, mathias olausson

QWise software engineering – refactored! www.qwise.se

Testing, testingA first-look at the new testing

capabilities in Visual Studio 2010

Mathias OlaussonCallista Knowledgebase | QWise

QWise software engineering – refactored! www.qwise.se

About Mathias...

QWise software engineering – refactored! www.qwise.se

About the Presentation...

• What’s on the menu?– Test planning and execution– Issue tracking with rich bugs – Eliminating the "works on my machine / cannot

reproduce" scenario– Automated UI Testing– Virtual Lab Management

QWise software engineering – refactored! www.qwise.se

TEST PLANNING AND EXECUTION

QWise software engineering – refactored! www.qwise.se

Agile Testing

QWise software engineering – refactored! www.qwise.se

Core Practices

QWise software engineering – refactored! www.qwise.se

Agile Testing Quadrants

Functional Tests Exploratory Tests

Unit TestsComponent Tests

Performance TestsLoad Tests

Security Tests”-ilitets” Tests

Business-facing

Technology-facing

Supp

ortin

g th

e te

amCritisizing the product

Automated &Manual

Manual

AutomatedAutomated

QWise software engineering – refactored! www.qwise.se

Test TypesManual Tests

UI Tests

Acceptance Tests

Unit Tests / Component Tests

QWise software engineering – refactored! www.qwise.se

Testing in an Iteration

sprint plan

sprint plan

write tests for us1

write tests for us1

write tests for us2

write tests for us2

sprint plan

sprint plan

build 1build 1 build 2build 2 build 3build 3

implement user story ‘1’

(us1)

implement user story ‘1’

(us1)

test us1 & file bugs

test us1 & file bugs

Implement US2

Implement US2

build 4build 4

fix bugsfix bugs

build 6build 6 build 7build 7

regress impacted

tests

regress impacted

tests

build 5build 5

implement us2

fix bugsfix bugs

test us2 & file bugs

test us2 & file bugs

verify fixesverify fixes

verify fixesverify fixes

QWise software engineering – refactored! www.qwise.se

Microsoft Test Manager and Visual Studio

Team Foundation Server Reporting

Development (Teams)Testing (Teams)

Web Test

Load Test

Coded UI Test Unit Testing

Test Case Management

Lab Management

Microsoft Test Runner

Fast Forward For Manual Testing

GeneralistSpecialistEngineering Support

Data Collectors (IntelliTrace, Test Impact, Metrics)

QWise software engineering – refactored! www.qwise.se

The Fundamentals

QWise software engineering – refactored! www.qwise.se

Defining and Running Tests

QWise software engineering – refactored! www.qwise.se

ISSUE TRACKING WITH RICH BUGS

QWise software engineering – refactored! www.qwise.se

Should We Do Bug Tracking?

Pros– Tracability– Priority of work– Visibility

Cons– Overhead

Should we log bugs within an iteration?Source: Pixar ”A bugs’ life”

QWise software engineering – refactored! www.qwise.se

Defect Tracking in MTM and MTR

QWise software engineering – refactored! www.qwise.se

No-more-no-repro

• Automatic data collection

• Test settings– Configuration of

diagnostic data collectors

– Assigned to test plan• Or per test run

QWise software engineering – refactored! www.qwise.se

Diagnostic Data Adapters

QWise software engineering – refactored! www.qwise.se

No-more-no-repro

QWise software engineering – refactored! www.qwise.se

AUTOMATED UI TESTING

QWise software engineering – refactored! www.qwise.se

Why Automated Testing?

Källa: Tools for Agility, Kent Beck

QWise software engineering – refactored! www.qwise.se

iteration 1iteration 1 iteration 2iteration 2 release iterationrelease iteration

feature afeature a feature bfeature b feature cfeature c feature dfeature d

unit testing

manual testing

regression testing

performance testing

feature afeature a

Automation Strategy

QWise software engineering – refactored! www.qwise.se

Coded UI Tests

• Unit test for UI• High-level framework• Create test from– Recording– Test result (action

log)– Using API

QWise software engineering – refactored! www.qwise.se

Automation Platform Support

QWise software engineering – refactored! www.qwise.se

Coded UI Tests

QWise software engineering – refactored! www.qwise.se

LAB MANAGEMENT

QWise software engineering – refactored! www.qwise.se

Motivation

QWise software engineering – refactored! www.qwise.se

Lab Management Aims To...

QWise software engineering – refactored! www.qwise.se

Lab Management Infrastructure

Team Foundation Server (TFS)

Lab Management

Test Case management

Build management

Work Item Tracking

Source Control

Hyper-V HostsVirtual

Machine1

Virtual Machine2

Virtual Machinen

Library SharesLibrary Share1

Library Share2

Library Sharen

System Center Virtual Machine Manager (SCVMM)

Team Foundation Server (TFS)

Lab Management

Test Case management

Build management

Work Item Tracking

Source Control

Hyper-V HostsVirtual

Machine1

Virtual Machine2

Virtual Machinen

Library SharesLibrary Share1

Library Share2

Library Sharen

System Center Virtual Machine Manager (SCVMM)

QWise software engineering – refactored! www.qwise.se

Environments

• An environment is a set of machines with different roles

• An environment can– Be deployed, saved, removed– Started, stopped– Snapsshot

VM1

Lab Agent

VM2

Lab Agent

QWise software engineering – refactored! www.qwise.se

Manual Tests in Lab

• Testing from the outside– Easy for web applications– Limitations in data collection

• Testing from the inside– Run application in lab via environment viewer

QWise software engineering – refactored! www.qwise.se

Automatic Tests in Lab

• Testing in VM– Test case connected to test environment– Test component comes from selected build

• Testing in build– Test case connected to build definition– Runs test list or test component

QWise software engineering – refactored! www.qwise.se

Lab Management Build Workflow

QWise software engineering – refactored! www.qwise.se

Build and Deploy

QWise software engineering – refactored! www.qwise.se

Summary

• Structured test planning with MTM• Integrated test execution• Effective defect tracking with rich bugs• Seamless integration with automated tests• Powerful lab management

QWise software engineering – refactored! www.qwise.se

Questions?

mathias.olausson@callista.sehttp://msmvps.com/blogs/molausson