From Quality Control to Quality Assurance…and Beyond

Alan PageMicrosoft

IntroductionMy testing career started in 1993

Testing always lastTesting often skippedAnyone can test“Testing in” quality

My role in testing has changed significantly in the past 13 yearsTesting as a whole has matured Disclaimer – I know what I know and don’t claim to know more (and are not representative of microsoft)

Software testing 13 years agoAnyone can do it

Technical knowledge not important

Test is a stepping stone to development (often still true today)

Testing occurs late in the product cycle

Software testing todayMany teams using automation to produce consistent and frequent test results

Often heavy emphasis on automation

Testers are generally focused on:Test authoring: writing test cases and automated testsTest execution: running automated or manual test casesFunctional and non-functional testingOther bug finding activities

Some analysisData analysis is often extensive, but is primarily based on defect data

Software testing tomorrowToo many bugs getting to customers

Bugs need to be found sooner

Where are we going?

Better bug prediction

More testable software

Defect prevention

Inspections

…

How do we get there?

More testable software

Testability is the degree to which components and systems are designed and implemented to make it easier for tests to achieve complete and repeatable code path coverage and simulate all usage situations in a cost efficient manner. In other words, testability determines how inexpensive it is to test.

or

What is the cost of testing?

Testability

Testability is a design time attribute…but testers often drive testability into the design

Ask questions and take action:How are we going to efficiently test this?Add testability sections to design and functional specificationsHold testability reviews

SOCK for testability

Testability: SOCKSimple components are less expensive to test.

Can the hardware be simulated?

Can we test a multi-machine scenario on a single machine?

Observable behavior and results are essential in order to determine pass or fail

Can I determine if the internal structures were updated correctly?

Can I tell which path the code took?

Control: can I test every nook and cranny?

Can I change timeout values or other thresholds to simulate failures?

Can I simulate every failure that a customer may hit?

Knowledge is needed to understand expected results and compare with actual results

What should happen in error cases?

What circumstances cause this error?

SOCK for testability

Defect analysis

Known defects need to be analyzed

Deep analysis can tell a lot, but it’s expensiveToo lightweight of an analysis may not tell you enough

Ask whyAnalyze to the point of action: Ask “why” until you have a sufficient root cause

Why did the program crash? The filename was longer than expected.

Why did the program not expect a longer filename?

The programmer was unfamiliar with file name limits

Why was the programmer unfamiliar with the limits?

They were new to windows programming and hadn’t been trained…or…

Why was this missed during code review?Code review rules were often skipped or rushed.

Ask Why?

Defect prevention

Root cause analysis leads to defect prevention

What could have been done to keep this defect from ever happening?

Examples:Change compiler settings or use code analysis tools or scriptsBuild code often (continuous integration)Automatically run unit tests or verification tests at check-in

Software Inspections

Not typically considered a “Test task” – but testers are well suited to drive this effort

Extremely effective method of removing defects, but…

Cost (training and time) tends to scare teams away

What happens when you inspect?

Example of changes from inspection:

Inspection MetricsExisting

codeRelease

ARelease

BRelease

CInspection Effort 0 hrs 406 hrs 276 hrs 691 hrsTotal LOC 400K 7777 10904 49343Inspection hrs / KLOC 0 52 25 14Total Effort n/a 1062 hrs 1101 hrs 2169 hrsPercent in inspection 0% 38% 25% 32%Total hrs / KLOC n/a 137 101 44Defects / KLOC (after check-in) 8.2 1.8 1 0.7

Measurements were performed on existing code

base, and again on subsequent releases

How to make inspections less scaryTeam members do not know how to inspect

Train the team; appoint a moderator

Inspections perceived to slow down the project

Schedule inspections; measure and track progress

Team members fearful of inspectionsNon-confrontational meetings; remove threats; use as a learning experience for author

What am I trying to say here?We need to stop waiting for bugs!

It’s great to find bugs before the customers do, but we need to find them earlier

We need more emphasis on early detection and ultimately, prevention

How can this be achieved?Be an equal member of the engineering team

Software quality is a tough problemTechnical, creative people are needed to solve this problem

We need clearly defined career paths for both managers and non-managers in order to keep these types of people in test

Provide learning opportunities when needed

Key pointsSoftware testing is a maturing professionNeed to move beyond quality control (writing test cases and running tests)Test needs to do more to drive quality improvements throughout the engineering processTechnical skills are important in order to drive quality initiatives

Resources

Software InspectionsTom Gilb and Dorothy Graham, Addison-Wesley

The Practical Guide to Defect Prevention Harry Emil et al. Microsoft Press

Random thoughts on testinghttp://blogs.msdn.com/alanpa/

Questions?