qa club kiev #17 measuring quality by volodymyr prymakov

Measuring QualityRelease Readiness Criteria and Quality Reporting

22

11 Years general working experience in IT/QA (30+ projects)

----------

•QA Consultant at Ciklum Interactive

•QA Manager (Manual)

•Automated Testing Manager

•Load Testing Manager

•Project Manager

----------

•Hobbies: Job, Wife and 2 children, Sport

----------

Skype: vladimir.primakov

Linkedin: ua.linkedin.com/in/vladimirprimakov/

Email: [email protected]

Some Words About Me

Volodymyr Prymakov (Vladimir Primakov)

http://ua.linkedin.com/in/vladimirprimakov/

mailto:[email protected]

33

The presentation will cover the following items:

• Release Readiness criteria as a part of a master test plan, their types and details. (~ 30 min)

• Release readiness criteria appliance and areas of influence. (~ 5 min)

• Quality report and quality metrics + their connection to release readiness criteria. (~ 15 min)

• Concluding release readiness. (~1 min)

• Questions (~ 10 min)

Presentation’s Structure

Presentation Plan

44

Release Readiness Criteria as a part of Master Test Plan

Release Readiness Criteria

Master Test Plan

55

Categories of Release Readiness Criteria

Analyzing Quality

(RESULTS)Catching Quality

(ACTIVITIES)

1. New Functionality Testing

2. Regression Testing

3. Test Environments Testing

4. Unit & Integration testing

5. Special Types of Testing

66

Any Ideas?

Criteria for New Functionality Testing

77

Short Example


Test Environment Priority

Coverage of Test Environments

Tested Functionality Priority

Minimum Test Deepness

Minimum Functionality Coverage

MaximumFail Rate %

The Highest one 100% Features of Any Priority

Detailed Exploratory testing

100% 0%

High 100% (all high priority test environments)

All priorities Happy path Exploratory testing

100% 0%

Medium Please See test environments related criteria

Low Please See test environments related criteria

88

Short Example







MaximumFail Rate %



100% 0%



100% 0%



99

Short Example







MaximumFail Rate %



100% 0%



100% 0%



1010

Short Example







MaximumFail Rate %



100% 0%



100% 0%



1111

Short Example







MaximumFail Rate %



100% 0%



100% 0%



1212

Short Example







MaximumFail Rate %



100% 0%



100% 0%



1313

Short Example







MaximumFail Rate %



100% 0%



100% 0%



1414

Expanded Example







MaximumFail Rate %

The Highest one 100% High (Blocker/Critical)


100% 0%

Medium (Major) Detailed Exploratory testing

100% 0%

Low (Normal/Trivial) Happy path Exploratory testing

100% 0%



100% 0%



1515

Test Deepness







MaximumFail Rate %



100% 0%


100% 0%


100% 0%



100% 0%



Test Deepness

1616

Test Deepness


TEST DEEPNESS / DETALIZATION

Test Case Based Thought Out Test

Detailed Exploratory Test

Happy Path Exploratory test

Smoke test

Nothing

1717

Test Coverage







MaximumFail Rate %



100% 0%


100% 0%


100% 0%



100% 0%



Test Coverage

1818

Test Coverage


Test Coverage

Test Coverage is 71 %, But should be 100%

1919

MaximumFail Rate %







MaximumFail Rate %



100% 0%


100% 0%


100% 0%



100% 0%



Fail Rate %

2020

Fail Rate %


Functionality PriorityMaximum Allowable Bugs Number

(Story, Improvement, New Feature)

BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs

High (Blocker / Critical) 0 0 0 * *

Medium (Major) 0 0 0 * *

Low (Normal/Trivial) 0 0 * * *

PASS CRITERIA

*if too many bugs are in place for the functionality – the bugs number can be considered for any specific case separately

2121

Fail Rate %




BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs




PASS CRITERIA


2222

Fail Rate %




BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs




PASS CRITERIA


2323

Fail Rate %




BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs




PASS CRITERIA

2424

Fail Rate %




BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs

Features of Any Priority 0 0 * * *

PASS CRITERIA (short and not so strict)

2525

Fail Rate %




BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs



Status: FAIL RATE %

Blocked

2626

Fail Rate %




BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs



Status: FAIL RATE %

Blocked

Failed Release Readiness Criteria

2727

Fail Rate %




BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs



Status: FAIL RATE %

Blocked


Status: Pass Rate %

Failed With Acceptable Bugs

2828

Fail Rate %




BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs



Status: FAIL RATE %

Blocked


Status: Pass Rate %


2929

Fail Rate %




BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs



Status: FAIL RATE %

Blocked


Status: Pass Rate %


3030

Fail Rate %




BlockerBugs

CriticalBugs

Major Bugs

Normal Bugs

Trivial Bugs



Status: FAIL RATE %

Blocked


Status: Pass Rate %


Passed

3131

Fail Rate %


Test Coverage

Fail Rate is 29%, But should be 0%

3232

Test Environment Coverage







MaximumFail Rate %



100% 0%


100% 0%


100% 0%



100% 0%



Test Environments Coverage

3333



Test EnvironmentsType OS Browser/Device Resolution Priority

PC Win 7 Chrome 1920*1080 Highest

PC Win 8 (metro) IE 10 1920*1080

HighMac MacOS Firefox 1920*1080

iPad iOs7 iPad 4 2048*1536

Mac MacOS Safari 2880*1800

MediumPC Win 7 IE 11 1366*768


PC Linux Firefox 1366*768Low

3434





PC Win 8 (metro) IE 10 1920*1080







There is only 1 highest

priority environment so the

coverage, which it gives for

the priority, is 100%.

3535





PC Win 8 (metro) IE 10 1920*1080







1 environment of 3 high priority

environments gives ~ 33% coverage.

All these environments together

gives 100% coverage for the priority.

3636

Test Environment Coverage - Example


3737

What Else?


3838

What Else?


3939

DEFINTION OF DONE (EXAMPLE)

User story reviewed & adjusted correspondingly

Unit-test created and passed

Integration test created and passed

Pear review passed

UX-review passed

Manual Test Passed

End User Test Passed…

Definition of Done


4040

Criteria for Regression Testing

4141


Test Deepness

4242


Test Coverage

Test Deepness

4343


Test Coverage

Test Deepness

Fail Rate

4444


Test Coverage

Test Deepness

Fail Rate

Test Environments

4545

Example




Tested Scenario Priority



MaximumFail Rate %

The Highest one 100% High Detailed Exploratory testing

100% 0%

Medium (That might be affected by recent product changes)

Happy path Exploratory testing

100% 0%

Other Medium Happy path Exploratory testing

50% 0%

High See test environments related criteria

Medium See test environments related criteria

Low See test environments related criteria

4646

Example







MaximumFail Rate %


100% 0%



100% 0%


50% 0%




4747

Example







MaximumFail Rate %


100% 0%



100% 0%


50% 0%




4848

Criteria for Test Environment TestingPossible Test Environment Ingredients

Internet Speed Antivirus, Firewalls

Plugins

Integration with other

applications and API

User Rights

RuntimesExternal Devices

Screen Resolutions and Modes

Hardware and Software Platforms and Modes

Web-browsers

Mobile Software Platforms

Mobile Software Platforms

External Libraries

Hardware

Characteristics and

Capacity

Robots (HD,

Software,

Capabilities)

4949

Criteria for Test Environment Testing

Product-wide Test Deepness

5050


Product-wide Test

Deepness

Full Regression

Full Regression for

functionality of certain

priority

Main product Flows test /

User Acceptance Test /

Cover test

Selective Regression

Smoke Test

5151


Coverage of

Test Environments


5252


Coverage of

Test Environments


Fail Rate

5353


Test Environment Priority Target (functionality) Minimum Test Deepness

Minimum Coverage of Test environments

MaximumFail Rate %

Highest Please see the release readiness-criteria for new functionality and oldfunctionality

High New functionality Please see release readiness-criteria for newfunctionality

Old (Regression-related) functionality

Selective Regression and/or Main product Flows test

100% 0%

Medium New and Old (Regression-related) functionality

Product-wide Smoke test

75%* 0%

Low New and Old (Regression-related) functionality


50%* 0%

Example

*Not covered environments may be covered next release and so on

5454




MaximumFail Rate %





100% 0%



75%* 0%



50%* 0%

Example


5555




MaximumFail Rate %





100% 0%



75%* 0%



50%* 0%

Example


5656

Criteria for Unit & Integration Testing

Test Coverage

Fail RateGUI-LESS

Functionality

GUI

Functionality*

Unit & Integration Testing

Test Coverage

Fail Rate

*Criteria may be not so strict, because GUI functionality can be also manually tested

5757

Criteria for Unit & Integration TestingExample

Integration Testing or Unit-TestingPriority of functionality


MaximumFailRate %

High 100% 0%*

Medium 0% 0%*

*Depends on severity of the problems found in

integration/unit testing

5858

Criteria for Special Types of testing

Type of testing* Sufficient?

(yes/no)

Passed? Conclusions

Load/Performance

testing (client/server

side)

NOT READY

Security Testing NOT READY

UX-Testing READY

(Accessibility Testing) NOT READY

GENERAL CONCLUSSION: NOT READY

* Usually has separate release readiness criteria, but in any case it should be sufficient

and passed.

Special types of testing, which are obligatory to be conducted and passed

5959

Release readiness criteria appliance and areas of influence

6060

Release readiness criteria appliance and areas of influence

Release readiness criteria

Defines testing activities and product quality sufficient for release

Influences testing strategy, testing approaches, and activities needed

Defines QA Resources requirements

Defines Quality reporting structure and content

Helps making conclusion about release readiness

6262

For those who don’t sleep yet

thank you for being cheerful!

6363

Quality Reporting and its connection to release readiness criteria

Quality Reporting

6464

1. Number of Bugs and their severity

2. New Functionality Testing & Results

3. Regression Testing & Results

4. Test Environments Testing & Results

5. Unit & Integration testing & Results

6. Special Types of Testing & Results

7. Concluding Release Readiness

Quality Report Structure

Quality Reporting

6565

1. Number of Bugs and their severity

2. New Functionality Testing & Results

3. Regression Testing & Results

4. Test Environments Testing & Results

5. Unit & Integration testing & Results

6. Special Types of Testing & Results

7. Concluding Release Readiness

Quality Report Structure

Quality Reporting

6666

All Not Closed Bugs

Number of Bugs and their severity

6767

All Not Closed Bugs


6868

All Not Closed Bugs – By Component


6969

All Not Closed Bugs – Not-Assigned to any Sprint


7070

Number of Bugs and their severityAll Not Closed Bugs – Not-Assigned to any Sprint

7171

Bugs List (for All not-closed bugs) – That fails release readiness criteria


Bugs List (for All not-closed bugs) – That fails release readiness criteria

Assigned to sprints

Not-Assigned to sprints

7272


Make a separate release readiness conclusion for this criterion type

7373

New Functionality Statuses (From Tasks Management System)

New Functionality Testing & Results

Features Statuses (Current release)

7474



Features Statuses (Current release)

Not-Resolved Features By Components(Current release)

7575



1

Test Environments

7676



1

2

7777



1

2

3

7878



1

2

3

4

7979



1

2

3

4

Failed Functionalities Details (Optionally)

8080



8181

Regression testing results for the highest priority environment

Regression Testing & Results

By Components

8282

Regression testing results for the highest priority environment


Failed Functionalities Details (Optionally)

8383



8484

Test Environments Testing & Results

Test plans for test environments of different priorities

8585

Test Environments Testing & Results


8686

GUI-less functionality

Unit & Integration testing & Results

Type of testing Coverage Fail Percentage Conclusions

Unit-test 50% 0% READY

Integration test 100% 0% READY

GENERAL RELEASE READINESS CONCLUSSION: READY

8787







8888







8989







9090

Special Types of Testing & Results


(yes/no)

Passed? Conclusions

Load/Performance


side)

NOT READY


UX-Testing READY


GENERAL RELEASE READINESS CONCLUSSION: NOT READY


and passed.

9191



(yes/no)

Passed? Conclusions

Load/Performance


side)

NOT READY


UX-Testing READY




and passed.

9292



(yes/no)

Passed? Conclusions

Load/Performance


side)

NOT READY


UX-Testing READY




and passed.

9393



(yes/no)

Passed? Conclusions

Load/Performance


side)

NOT READY


UX-Testing READY




and passed.

9494

Concluding Release Readiness

Criterion Status

1. Number of Bugs and their severityNOT READY

2. New Functionality Testing & ResultsREADY

3. Regression Testing & ResultsREADY

4. Test Environments Testing & ResultsNOT READY

5. Unit & Integration testing & ResultsREADY

6. Special Types of Testing & ResultsNOT READY


9595

Concluding Release Readiness

Criterion Status

1. Number of Bugs and their severityNOT READY

2. New Functionality Testing & ResultsREADY

3. Regression Testing & ResultsREADY

4. Test Environments Testing & ResultsNOT READY

5. Unit & Integration testing & ResultsREADY

6. Special Types of Testing & ResultsNOT READY


9696

Instruments Facilitating Quality Reporting

CSV

CSV

As Gadgets, iFrames,

Screenshots, Links

plugins

9797

Thank you! ;)

QUESTIONS?

qa club kiev #17 measuring quality by volodymyr prymakov

Education

test environments testing4

new functionality testing2

short examplecriteria

highest one100

special types of testing

quality metrics

qa consultant

following items