exploratory testing - seprof.sebern.com · unconsciously, perform exploratory testing at one time...

Copyright © 2012-2014 by Mark J. Sebern

Exploratory TestingDr. Mark Sebern SE2832-2014-18


What is it? (1)

“Exploratory software testing is a powerful and fun approach to testing. In some situations, it can be orders of magnitude more productive than scripted testing. I haven't found a tester yet who didn't, at least unconsciously, perform exploratory testing at one time or another. Yet few of us study this approach, and it doesn't get much respect in our field. It's high time we stop the denial, and publicly recognize the exploratory approach for what it is: scientific thinking in real-time. Friends, that's a good thing.” — James Bach, Satisfice Inc. www.satisfice.com/articles/what_is_et.shtml


What is it? (2)“A style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.” — Cem Kaner, kaner.com www.kaner.com/pdfs/QAIExploring.pdf

As opposed to what? Scripted testing!


Test This - How?• Functional tests

• Not security issues

Test script #1 • Launch SUT • Enter user ID “xyz” • Enter password “zyx” • Press Enter key • Expect result: login success

Test script #2 • Launch SUT • Enter user ID “xyz” • Enter password “zyx” • Click Login button • Expect result: login success

Test script #3 • Launch SUT • Enter user ID “abc” • Enter password “zyx” • Click Login button • Expect result: login rejected


Testing Patterns• Input fields

• Valid data

• Invalid data

• Length < maximum

• Length = maximum

• Length > maximum

• Actions

• Keyboard

• Buttons

• Operations

• Add, modify, inquire, delete


Exploratory Testing Goals

• Probe for weaknesses

• Learn about design and implementation of a software system

• Create mental model of SUT function

• Design tests to prove/disprove

• Execute tests, observe behavior

• Evaluate against conjecture(s)

• Repeat until conclusion reached


Timeboxing• Where have we heard

this idea before?

• 60-120 minutes at a time

• Long enough to accomplish something

• Not so long that attention is lost

• Easier to schedule


Test Session Charters

• Thoroughly investigate a specific systems functionality

• Understand the performance characteristics of the software

• Ensure that all fields are properly validated

• Force all error conditions to verify each error message

• Check the design against user interface standards


Advantages? (vs scripted)

• Helpful when it’s difficult to determine the test sequence in advance

• Can provide immediate feedback

• Can refine scope, impact, variations of existing defects

• Alternative when scripted tests aren’t finding anything

• What exactly can scripted tests find?


Disadvantages?

• May not detect defects immediately upon injection — SUT must be (nearly?) complete

• No exploration needed if all tests defined in advance

• No traceability — prove tests were done?

• Can lose focus — like web browsing distractions

• Difficult/costly to repeat when SUT changes — not automated for continuous integration


Bottom Line?

• Both scripted and exploratory approaches have a role to play

• Neither can completely replace the other

• Possible testing effort allocation

• 25% new test development

• 50% executing “old” tests (regression?)

• 25% exploratory


Scrum Connection

• How does exploratory testing fit in the Scrum process?

• In what sprint tasks?

• How is it planned & tracked?

• How do you know when it is complete?

• How is it like/unlike code & design review activities?

exploratory testing - seprof.sebern.com · unconsciously, perform exploratory testing at one time...

Documents