(exploratory) testing with scripted support
DESCRIPTION
Some consideration and reflections on applying good software test thinking in signalling protocol environments using "agile" and incremental development models.TRANSCRIPT
![Page 1: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/1.jpg)
(Exploratory) TestingWith Scripted Support
Simon Morley October 2014
Experience & Considerations
Saturday 25 October 14
![Page 2: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/2.jpg)
My Situation Context• Complex Environment
• Meaning overhead & set-up costs
• Meaning preparation, tooling or ordering
• Many signaling protocols
• Meaning script support for signal injection and check
• Agile & Incremental SW Development
Saturday 25 October 14
![Page 3: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/3.jpg)
Testing in Agile
• A common “agile” development model:• Backlog item / Opportunity Analysis
• Plan
• Design
• Test
• Build
• Iterate
Saturday 25 October 14
![Page 4: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/4.jpg)
Testing in Agile
• A common “agile” development model:• Backlog item / Opportunity Analysis
• Plan
• Design
• Test
• Build
• Iterate
Where testing prep & execution happens
Saturday 25 October 14
![Page 5: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/5.jpg)
Testing in Agile
• A common “agile” development model:• Backlog item / Opportunity Analysis
• Plan
• Design
• Test
• Build
• Iterate
Where testing prep & execution happens
Saturday 25 October 14
![Page 6: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/6.jpg)
Testing in Agile
• A common “agile” development model:• Backlog item / Opportunity Analysis
• Plan
• Design
• Test
• Build
• Iterate
Where testing prep & execution happens
Saturday 25 October 14
![Page 7: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/7.jpg)
Testing in Agile
• A common “agile” development model:• Backlog item / Opportunity Analysis
• Plan
• Design
• Test
• Build
• Iterate
Where testing prep & execution happens
Saturday 25 October 14
![Page 8: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/8.jpg)
Testing in Agile
• A common “agile” development model:• Backlog item / Opportunity Analysis
• Plan
• Design
• Test
• Build
• Iterate
Where testing prep & execution happens
Saturday 25 October 14
![Page 9: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/9.jpg)
Testing in Agile
• A common “agile” development model:• Backlog item / Opportunity Analysis
• Plan
• Design
• Test
• Build
• Iterate
Where testing prep & execution happens
Saturday 25 October 14
![Page 10: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/10.jpg)
Testing in Agile
• A common “agile” development model:• Backlog item / Opportunity Analysis
• Plan
• Design
• Test
• Build
• Iterate
Where testing prep & execution happens
Saturday 25 October 14
![Page 11: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/11.jpg)
Testing before “testing”• Testing in the “Opportunity Analysis”,
“Planning” & “Design” “phases” might be:
• Feature Investigation - from a testing perspective
• Testable?
• Creates issues with existing framework, tools and maintenance?
• Feature Walkthrough
Saturday 25 October 14
![Page 12: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/12.jpg)
Possible Testing Approach• Investigate the new feature
development
• Develop framework and tools, develop scripts, comms, test scripts & framework & feature, comms, debug scripts, comms, debug environment, comms, test feature, comms, report completion, archiving.
Saturday 25 October 14
![Page 13: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/13.jpg)
Feature Investigation
• Feature Walkthrough
• Test environment impacts & walkthrough
• Test Analysis summary
Saturday 25 October 14
![Page 14: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/14.jpg)
Test Analysis Summary• Describe* or reference the current
feature, test environment and framework situation
• Describe* impacts to test environment and framework
• Describe* approach
• Include coverage goals for product, feature and testing
Saturday 25 October 14
![Page 15: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/15.jpg)
Comms?• Communication
• Can be: Verbal & Written
• Can be: Formal & Informal
• Always: Two-way!
• Reporting - stand-ups, progress boards, meetings
• Reflect & decide about next steps
Saturday 25 October 14
![Page 16: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/16.jpg)
Reflect & Decide?• What has the testing so far revealed?
• Product coverage goals met?
• Testing coverage goals met?
• Perceived risk acceptable?
• Stakeholder view on these?
Saturday 25 October 14
![Page 17: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/17.jpg)
Reflect & Decide #2
• Next steps?
• Additional test steps?
• Enough already?
• Change direction or approach?
Saturday 25 October 14
![Page 18: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/18.jpg)
Reflect & Decide #3• Pause & Reflect
• Control Q: “What don’t we know?”
• Silent Evidence
• Note: This pause & reflection is important - see thinking about heuristic discovery.
Saturday 25 October 14
![Page 19: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/19.jpg)
Necessary support?• Can the tools & framework support
changes?
• Is the tool & framework architecture modular?
• Remember to test the tool architecture!!!
• Do the tools & framework allow quick prototyping?
• If not -> fix it!
Saturday 25 October 14
![Page 20: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/20.jpg)
Describe*• Describe: Meaning the level sufficient in
the team or project.
• If coordination is needed between teams this should be a format that is commonly understood (picture/symbols, dashboard or text document).
• You agree what’s appropriate!
Saturday 25 October 14
![Page 21: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/21.jpg)
Exploratory Testing Traps• Sometimes people think of ET means
less:-
• Documentation
• Structure
• Traceability
• And more
• Random banging on a keyboard
Saturday 25 October 14
![Page 22: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/22.jpg)
Exploratory Testing Traps• Sometimes people think of ET means
less:-
• Documentation
• Structure
• Traceability
• And more
• Random banging on a keyboard
Wrong !!!
Saturday 25 October 14
![Page 23: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/23.jpg)
ET Traps #2• Actually ET is about more
responsibility in skilled testing
• The skilled tester is responsible for communicating his/her approach, thinking, results and any mandated archiving
Saturday 25 October 14
![Page 24: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/24.jpg)
ET Traps #2• Actually ET is about more
responsibility in skilled testing
• The skilled tester is responsible for communicating his/her approach, thinking, results and any mandated archiving I.E. Plenty of structure,
traceability & documentation
where appropriate!
Saturday 25 October 14
![Page 25: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/25.jpg)
ET Traps #3• No support for skilled testers and
testing.
• Support for skilled testing is a pre-requisite (hand-in-hand) to realizing emphasized tester responsibility.
Saturday 25 October 14
![Page 26: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/26.jpg)
ET Traps #3• No support for skilled testers and
testing.
• Support for skilled testing is a pre-requisite (hand-in-hand) to realizing emphasized tester responsibility.
Fix these if you want good & valuable testing!
Saturday 25 October 14
![Page 27: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/27.jpg)
Discussion• Exploratory testing can be performed
with scripting support
• Good testing (in complex environments) requires elements of:
• Domain knowledge - built through early investigation
• Test environment knowledge & understanding
Saturday 25 October 14
![Page 28: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/28.jpg)
Discussion #2
• Plus
• Rapid learning
• System and problem analysis
• Decision making
• Framing, Consensus & Commumnication
Saturday 25 October 14
![Page 29: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/29.jpg)
References• Experience Report: Introducing Exploratory
Testing
• Deliberated Exploration - A Walk in the Woods
• Experiences with Semi-Scripted Exploratory Testing
• Framing: Some Decision Analysis and Frames in Testing
• Silent Evidence in Testing
• On Thinking about Heuristic Discovery
Saturday 25 October 14
![Page 30: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/30.jpg)
Thanks!• Simon Morley
• Twitter: @YorkyAbroad
• Blog: testers-headache.blogspot.com
• Questions?
• Get in touch!
Saturday 25 October 14
![Page 31: (Exploratory) Testing with Scripted Support](https://reader033.vdocuments.net/reader033/viewer/2022051513/54807e5c5906b5f9288b4719/html5/thumbnails/31.jpg)
Feature Walkthrough
Saturday 25 October 14