K1

Keynote

5/7/2014 8:30:00 AM

Principles Before Practices:

Transform Your Testing by

Understanding Key Concepts

Presented by:

Randy Rice

Rice Consulting Services, Inc.

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Randy Rice

Rice Consulting Services, Inc.

A leading author, speaker, and consultant with more than thirty years of experience in the field of software testing and software quality, Randy Rice has worked with organizations worldwide to improve the quality of their information systems and optimize their testing processes. He is coauthor (with William E. Perry) of Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems. Randy is an officer of the American Software Testing Qualifications Board (ASTQB). Founder, principal consultant, and trainer at Rice Consulting Services, Randy can be contacted at riceconsulting.com where he publishes articles, newsletters, and other content about software testing and software quality. Visit Randy’s blog.

4/29/2014

1

PRINCIPLES BEFORE

PRACTICES

HOW TO TRANSFORM YOUR TESTING PRACTICES WITH BASIC AND ADVANCED PRINCIPLES

RANDALL W. RICE, CTAL

RICE CONSULTING

SERVICES, INC.

© 2013, Rice Consulting Services, Inc.

2

A LITTLE BACKGROUND

• Many years ago, I heard a great

analogy about conveying and

applying knowledge.

• If I were to teach someone how to

wash dishes, I could:

• Teach all the variations needed for

every type of dish, cookware and

utensil, or&

• I could teach principles and let the

person adapt the methods to

whatever they needed to wash.

4/29/2014

2

3

THINGS LIKE…

• Rinse off the big foodstuff first.

• Save the really messy dishes until the

end so you don't get everything else in

the sink messy too.

• Use hot water, but not too hot or else

you will scald yourself.

• Be careful with sharp knives in sudsy

water"

4

SIMILARLY, IN TESTING…

• Take some sample tests early and find

where the big problem areas seem to be.

• Don't test the really complex areas at first.

Get your bearings first.

• Have strong tests, but if you make every

test strong, you may not have time to

finish.

• Early testing is good, expect when the

thing you are testing isn't ready even for

early testing.

• Test automation is not automatic.

4/29/2014

3

5

COMMON QUESTIONS

• Which tools do we use in

certain situations?

• Which techniques are “best” for

a project?

• How do we adjust tools and

techniques?

6

WHAT WE WANT

No Knowledge

Mastery

4/29/2014

4

7

REALISTIC VIEW

No Knowledge

Mastery

“The Dip”

Years

8

TESTING IS VERY NUANCED

• There are small to large variables on any project that require

you to:

• Select certain techniques

• Apply a technique differently

• In essence, all testing is context-driven

• And, there are small things that really make a big difference in

how we may apply a technique.

4/29/2014

5

9

FROM CLASSROOM TO PRACTICE

Learning Objectives

Retention

Instruction

Exercises

Visual Aids

Application

10

BUT, IT GETS WORSE

Learning Objectives

Instruction

Visual Aids

ExercisesRetention for multiple people

Application bymultiple people

Passage of time. Needs change, organizations change

4/29/2014

6

11

WHY?

• Let’s look at the lowest four levels of the Bloom

Taxonomy:

K4 (Analyze) - AssimilateK3 (Do) - PracticesK2 (Understand) –

Principles, concepts, theory

K1 (Remember) - Facts

The great majorityof Foundation Levellearning objectives

4 Foundationlearning objectives

12

7 GENERAL TESTING PRINCIPLES

• Testing shows the presence of defects

• Exhaustive testing is impossible

• Early testing

• Defect clustering

• Pesticide paradox

• Testing is context dependent

• Absence-of-errors fallacy

Source: ISTQB Foundation Level Syllabus

4/29/2014

7

13

THERE ARE MANY OTHER IMPLICIT

PRINCIPLES

• Not every failure is a defect.

• Not every test can or should be

automated.

• Test automation doesn’t replace the

need for manual testing

• It doesn’t matter how good your test

is, if you are testing the wrong

version.

14

CLIMBING K2

• People normally dislike this level of

learning.

• Conceptual

• Theoretical

• Hands-off

• Requires deeper understanding

• Takes longer to build the basis of

understanding

4/29/2014

8

15

ANOTHER CHALLENGE

• “Just show us how to use the tool.”

• The students

• “We want a course with all practical

application and little or no theory.”

• Management

16

A LITTLE TIME GOES BY…

• “We don’t know really get how to

design tests for our applications. We

just learned on the case study

example.”

• Students

• “The students passed the exam but we

are still struggling in how to implement

the techniques.”

• Management

4/29/2014

9

17

AN EXAMPLE FROM MUSIC

• It is possible to learn to play an

instrument from mimicking someone else.

• The Suzuki Method

• However, you are able to play only specific

songs.

• Professional musicians, even many rock

stars, understand music theory!

• Blues, heavy metal and other styles are all

based on different types of scales and

tunings.

18

WE SEE THE SAME

THING IN SPORTS

• Why do professional baseball

players, golfers, football players need

coaches?

• Objectivity

• Accountability

• Teaching from a broader background

of experience

4/29/2014

10

19

WHAT SETS APART PROFESSIONAL

MUSICIANS AND ATHLETES?

• Practice!

• Professional golfers are on the driving

range every day and play most days.

• Football players practice in extreme

conditions.

• Musicians may practice hours for one

show.

20

HOW DO TESTERS

PRACTICE?

• Try new approaches

• Delve deeper into an approach or

technique you already know

• Beta test a new technology

• Play mental games

4/29/2014

11

TWO EXAMPLES OF

TEST TECHNIQUES

THAT REQUIRE

DEEPER

UNDERSTANDING

22

EXAMPLE 1 – PAIRWISE TESTING

• Key Principles

• Used when there are distinct numbers of conditions that can be

tested together, but the testing of all combinations would be

infeasible.

• Instead of testing all combinations of conditions, all unique pairs

of conditions are tested.

• Finds defects when:

• The interaction between TWO conditions are incorrect

• The conditions make sense to relate to each other

• Primarily applied by using either orthogonal arrays or pairwise

tools.

• This can be a very interesting and effective way to deal with

testing high numbers of combinations.

4/29/2014

12

23

Teacher, why do we need to know this?

24

AN EXAMPLE

The IE Tools>AdvancedOptions window

53 binary conditions1 condition with 3 options1 condition with 4 options

253 = 9,007,199,254,740,992

X 12

108,086,391,056,891,904

Possible combinations ofconditions!

Source: James Whittaker, How to Break Software

4/29/2014

13

25

THIS WOULD REQUIRE…

At one second per test execution:

108,086,391,056,891,904

360300,239,975,158,033.067= hours

12,509,998,964,918.04 days

34,273,969,766.9 years

To test all possible combinations!

26

WITH TEST CASE OPTIMIZATION

• The number of tests that

represent unique pairings = 7

• Obtained from using the AllPairs

tool

4/29/2014

14

27

WHAT RESEARCH INDICATES

• There have been many studies done

where defects were researched after

the fact.

• Research has shown that the highest

likelihood of an integration defect

relates to interaction between pairs of

items or parameters.

• www.pairwise.org

28

EXAMPLE OF RESEARCH FINDINGS

• [Evaluating FDA recall class failures in

medical devices we established that] [...]

out of the 109 reports that [were] detailed

[enough], 98% showed that the problem

could have been detected by testing the

device with all pairs of parameter settings.

• D.R. Wallace and D.R. Kuhn, 2001

• http://csrc.ncsl.nist.gov/staff/kuhn/final-rqse.pdf

4/29/2014

15

29

THE LIKELIHOOD OF TRIPLE-

MODE OR HIGHER DEFECTS

Condition B Condition DCondition CCondition A Condition E

Double-mode defects are more likely&

than Triple-modeor higher defects.

30

EXAMPLE – AN OA.4.3.2.2 ARRAY

011Test case 4

101Test case 3

110Test case 2

000Test case 1

Condition 3Condition 2Condition 1

4/29/2014

16

31

EXAMPLE – WITH CONDITIONS

PoolFarmRuralTest case 4

PondHighriseRuralTest case 3

PondFarmUrbanTest case 2

PoolHighriseUrbanTest case 1

Condition 3Condition 2Condition 1

Some paired conditions may make no sense, or may be infeasible

32

ONE MORE SPIN ON

THIS

• A client was using orthogonal arrays to optimize the testing of

stock trades.

• 15 broker types

• 3 trade types

• 2 customer types

• 7 amount levels (bands)

• 4 account types

• This would yield 15*3*2*7*4 = 2,520 tests for all combinations.

• We noticed that the 15 broker types could be safely divided into

5 equivalence classes.

• 5*3*2*7*4 = 840

• This resulted in a 67% reduction in test cases before any pairwise

analysis!

4/29/2014

17

33

WHAT WE’VE SEEN

• Instead of going immediately to a tool or technique with a set of categories and conditions, let’s look first to see:

• If the technique makes sense in the situation

• How do we deal with the downsides?

• In this example, can they be used as negative tests?

• What do the stakeholders think of the technique?

• I’ve seen some get very excited

• I’ve seen some get very nervous!

• Are we willing to accept the trade-offs of pairwise testing?

• It becomes very important which categories and conditions we test together.

• We still need an oracle to know the correct outcome.

• Maybe there is a provision for a highrise building on a farm!

34

EXAMPLE #2 – RISK-BASED

TESTING

• Key Principles

• Used when you can’t test the basic layer of functionality and/or when you need to prioritize testing to meet delivery schedules

• The higher the risk, the more intense the test should be

• Helps by:

• Focusing on areas that might experience high levels of loss

• Giving lesser attention to trivial or less important functions

• Tracing and mitigating risks identified during testing

• Can be used in just about any application as long as you are able to intelligently identify and measure risk.

• It is possible to miss some risks and some defects.

• Risk is a possibility, not a certainty

• Our knowledge is limited

• Some things will likely not get tested

4/29/2014

18

35

Teacher, isn’t all testing risk-based?

36

Universe of Test Cases

Case 1

Case 2

Case 3

Case 4

Case 5

Case n

Case 6

Case 7

Case 11

Case 8

Case 12

Case 13

Case 10

Case 14

Case 9

CONSIDER THE FOLLOWING

4/29/2014

19

37

Universe of Test Cases

Case 1

Case 2

Case 3

Case 4

Case 5

Case n

Case 6

Case 7

Case 11

Case 8

Case 12

Case 13

Case 10

Case 14

Case 9

High-risk cases

Low-risk cases

Moderate-risk cases

THE TEST CASE UNIVERSE

SEGMENTED BY RISK

38

Universe of Test Cases

Case 1

Case 2

Case 3

Case 4

Case 5

Case n

Case 6

Case 7

Case 11

Case 8

Case 12

Case 13

Case 10

Case 14

Case 9

High-risk cases

Low-risk cases

Moderate-risk cases

TEST CASES THAT SPAN RISK LEVELS

In testing this transaction, youspan all levels of risk. How do youprioritize risk in this scenario?

4/29/2014

20

39

HOW TO APPLY

• At various levels:

• Product risk

• Functional risk

• Attribute risk – Usability, performance, etc.

• Project risk

• But what about?:

• Residual risks

• Safety critical (when everything is a “high” risk)

• When risks are:

• Difficult to identify

• Change often

• Are an excuse for reduced testing

• Difficult to mitigate and track

This is the practice level!

40

A TIP TO HELP IDENTIFY YOUR

PRACTICES

Area/Principles Implications

Test Automation

• You can’t automate everything

• Automation doesn’t replace people

• The ROI of test automation is repetition

• Select tests carefully to automate

• Consider which skills will need to be built

• To get payback, start with simple tests

that are performed a lot.

Test Design

• Test conditions form the basis of test

cases

• Many things can serve as a basis for test

design, including experience

• Before writing test cases, have a smart

way to identify test conditions.

• If you don’t have a good written basis for

testing, you can create one.

Test Evaluation

• If you can’t observe the outcome of a

test, you can’t evaluate it

• Testing is measurement

• If we can’t define a way to see, report, or

measure the outcome of a test, it does

little good to plan it.

• Since measurement of something

(defects, performance, etc.) is the

deliverable of testing, we should do it

well.

4/29/2014

21

HOW TO GAIN

UNDERSTANDING

GOING FROM PRINCIPLE TO PRACTICES

42

A FEW EXPECTATIONS

• This is not easy, automatic or instant

• This is a mental process more than

anything else

• This is also the result of practice and

experience

• This is not a classroom deliverable

• Mentoring and coaching can help you see

blind spots, but they can’t make you

change.

4/29/2014

22

43

WHAT REALLY HAPPENS

Instructor says, “Do it like this&” You say&”OK”But think, “I’m not so sure” or“I like my way better.”

44

However, I know I need to improve my game. (I think)

4/29/2014

23

45

ACTUALLY, THAT’S FINE

• Each of us know internally

the way we work best.

• An external coach can

provide tips and advice, but

you know what works for you.

• A great resource for this is

the series, “The Inner Game”

by Timothy Gallwey.

46

WHAT WE ARE TALKING ABOUT IS

CHANGE

No Knowledge

Mastery

“The Dip”

Years

Change is hard!

We often fail

4/29/2014

24

47

COMPETENCE LEVELS

You do things badly but don’t know it.

You do things badly but do know it.

You do things well but have to think carefully about what you are doing.

You do things well and really don’t think much about the mechanics.

48

Source: Will Taylor

4/29/2014

25

49

IN TAKING EXAMS

• Don’t overthink the questions.

• Read it carefully and think it through, but

don’t go back and forth too much on one

question.

• Don’t rely on practice exams.

• They can be HUGELY misleading.

• Instead, ask “Can I demonstrate the

learning objectives?”

• Read the book “Choke”.

• It explains why we get all flustered and

don’t do well on exams.

50

IMPLEMENTING PRACTICES

• Pilot projects are great.

• Learn in the small before going to

the larger scale

• Use the “planned organic” approach

• Have senior management support

• Let others copy your success!

4/29/2014

26

51

FINAL THOUGHT

• “the 10,000hr rule is a definite key in

success” ― Malcolm Gladwell, Outliers: The Story of

Success

52

4/29/2014

27

53

BIO - RANDALL W. RICE

• Over 35 years experience in building and testing information systems in a variety of industries and technical environments

• ASTQB Certified Tester – Foundation level, Advanced level (Full)

• Director, American Software Testing Qualification Board (ASTQB)

• Chairperson, 1995 - 2000 QAI’’’’s annual software testing conference

• Co-author with William E. Perry, Surviving the Top Ten Challenges of Software Testing and Testing Dirty Systems

54

CONTACT INFORMATION

Randall W. Rice, CTAL

Rice Consulting Services, Inc.

P.O. Box 892003

Oklahoma City, OK 73170

Ph: 405-691-8075

Fax: 405-691-1441

Web site: www.riceconsulting.com

e-mail: rrice@riceconsulting.com