test automatisering and exploratory test - testtech.dk · test automatisering and exploratory test...

Fitting Solutions

Test automatisering and Exploratory Test

Christian Nørlyng

Software Test Developer

3 of April 2014

Who am I?

Fitting Solutions

•Software Test Developer at Oticon

•ISTQB Certified Test Analyst (adv. level)

•ISTQB Certified Technical Test Analyst (adv. level)

•Certified Agile Tester

•Certified Scrum Master

10+ years of test experience from

• Small & Large Teams

• Agile & Waterfall Projects

• Medical – Directory – Telecom – Defense – Maritime

Have been mostly Test Automation

The Testing Challenge at Oticon Fitting Solutions

20.000+ different HI’s since 2011, which must be tested at least once pr release

Different Windows platforms. Even ”XP”

Tons of un thought of hardware combinations

Regression testing the old platforms and HI’s

Multiple programming devices

Infinite amount of ways through the application

Safety Critical

High Volume Market

At least 4 Releases pr year

What bugs are you looking for??

What is your test mission?

What kind of bugs are you looking for?

What concerns are you addressing?

Who is your audience?

Make automation serve your mission.

Expect your mission to change.

Possible Missions for Test Automation

Find important bugs fast

Measure and document product quality

Verify key features

Keep up with development

Assess software stability, concurrency,

scalability…

Provide service

Efficiency

Reduce testing costs

Provide proof

Reduce time spent in the testing phase

Automate regression tests

Improve test coverage

Make testers look good

Reduce impact on the bottom line

Reduce time spent in the testing phase

Automate regression tests

Improve test coverage

Make testers look good

Reduce impact on the bottom line

Tighten build cycles

Enable “refactoring” and other risky

practices

Prevent destabilization

Make developers look good

Play to computer and human strengths

Increase management confidence in the

product

Most Important Mission

Increase management

confidence in the product.

Automation and bugs

Test Automation are only looking for defects found during ”regression testing”

• All automated tests, except model based test are ”second time arround” a feature

Feature Picking

”Features”

”Trade offs”

”Possibilities”

”Context”

Starting with the

inter section!

Feature Picking

”Wants”

”Seems Quick”

”Critical”

”Features”

Starting with the

inter section!

Provide the best value to the project!!!

Feature Picking

”Planned”

”Unsertain”/ ”Sort of Planned”

Pick the stable features which is not part of unsertain

”Features”

”Stable Early”

Choosing Best Automation Candidates

Good candidates

• Short or simple transactions

• Many data combinations

• Expected results are stable or easy to generate at runtime

• Tests that are executed regularly

• Tasks that are difficult to do manually

• Highest priority features

Poor candidates

• Long or complex transactions

• One-offs

• Unstable or difficult to predict results

• Tests that cross multiple applications

Designing Good Test Cases

”Structural Test Patterns”

Test cases should have a single objective

Test cases should result in one of two dispositions: pass or fail

Test cases are independent

No test case relies on the successful completion of another test

Test cases start and stop at a known ‘base’ state

Approaches and ROI

Automation MUST have ROI

ROI = (Savings + Total Investment)/ Total Investment

Quick and Dirty

• Implement, Run and Discharge

• Intinial High ROI. Decreases over time with change

Build up test method framework

• Design, Impliment, Improve, Maintain and Execute

• Intinial medium ROI. Increases over time with repeated runs

Model Based Testing

• Design, Model, Impliment, Improve, Maintain and Execute

• Initial low ROI. Increases over time, if managed correctly

13

Framework Design

Traditional

Sequential

TestCases

A pseudo test case broken down into ActionWords, each KeyWord is unique to the SUT (Software Under Test)

KW 6

KW 5

KW 4

KW 3

KW 2

KW 1

Example: Framework Design

KW = KeyWord

KW 7

KW 6

KW 3

KW 2.1

KW 1

KW 7

KW 5.1

KW 4

KW 3

KW 2.2

KW 1

KW 7

KW 5.2

KW 4

KW 3

KW 2.3

KW 1 1

2.1 2.2 2.3

3

6 4

5.1

7 Saving above 50% of work for just this example.

5.2

KW = KeyWord An Advantage

Module Based Test Design

The End for Test Automation

What I did not talk about

Selecting the right Test Tool

As of 2 days ago the 3 best tools are:

• TestComplete - frequently upgraded, the 3 platforms, Not expensive

• HP Functional Tester – not so frequently upgraded, the 3 platforms, expensive

• Selenium - frequently upgraded, web only, free. tons of add-ons.

Continueous Integration

Test Architecture

Daling With change

Automation Approach: ”Quick And Dirty”

Practical Implementaion: ”Capture Replay”, ”Objects Mapping”

End Times

Fitting Solutions Exploratory Testing

What ET is…

Simultaneous learning, test

design and test execution

Unscripted

Targeted

Repeatable

Relevant

A rapid cycle of

- Design a test

- Execute it

- Observe results

Until we've characterized the capabilities

and limitations of the system with respect

to a charter

Charter Pile

Do a Charter in Session

Report generarion

New Ideas for Charters

•Strategy •Modeling •Decision Making •Configuring •Operation •Observation •Evaluation

The Test Session

Uninterrupted

e-mail, meetings, telephone calls etc.

Reviewable

A report should be produced

Chartered

Mission associated with this session;

• What are we testing?

• What are we looking for?

Timeboxed

ET Execution 19

From Rapid Software Testing, copyright © 1996-2002 James Bach

20

Charter: A clear mission for the session

A charter may suggest what should be tested, how it should be tested, and what problems to look for.

A charter is not meant to be a detailed plan.

General charters may be necessary at first:

“Analyze the Insert Picture function”

Specific charters provide better focus, but take more effort to design:

“Test clip art insertion. Focus on stress and flow techniques, and make sure to insert into a variety of documents. We’re concerned about resource leaks or anything else that might degrade performance over time.”


Doing Exploratory Testing

Keep your mission clearly in mind.

Keep notes that help you report what you did, why you did it, and support your assessment of product quality.

Keep track of questions and issues raised in your exploration.

To supercharge your testing, pair up with another tester and test the same thing on the same computer at the same time.

21


During the Session….

Learning

/Notes

Design

Testing

22

When are you DONE ?

Testing does not reveal NEW useful information.

Experienced tester feels comfortable.

Critical risk features have had time to go through exhaustive

testing.

AND: Finalising Notes, learnings and ideas

Then: Debrief with the “Manager”


Session Based Test Management

Brief enough for accurate reporting.

Brief enough to allow flexible scheduling.

Brief enough to allow course correction.

Long enough to get solid testing done.

Long enough for efficient debriefings.

Beware of overly precise timing.

Time Box/Session 24

Short: 60 minutes (+-15)

Normal: 90 minutes (+-15)

Long: 120 minutes (+-15)


Measurement begins with observation

The manager reviews session sheet to assure that he understands it and that it follows the protocol.

The tester answers any questions.

Session metrics are checked.

Charter may be adjusted.

Session may be extended.

New sessions may be chartered.

Coaching happens.

Debriefing 25


ET Documentation

Planning:

Charter – overview of what to test (plan)

Migh be a flip chart on the wall

Mission – What are we looking for?

Execution

Notes – what happened during testing?

What did I do? Why did I do it?

Used to assess product quality after test.

Data files – input data used for testing

Bug reports – enough details to recreate the test / bug

Track questions and Issues

26


Why does it Work?

Testers use results of previous results to guide their future

testing on the fly

They do not have to complete a series of scripted tests

before focusing in or moving on to explore a more target rich

environment

This accelerates bug detection when used intelligently

27


The Theory

Discovery helps you choose relevant things to do among the

infinite number of possibilities

The variations possible are “learned” as other tests are run

You run tests you didn’t think of initially – can’t anticipate

everything!

“Sampling” vs complete coverage of conditions where complete

coverage is not practical

Tests drawn from using a testing technique the team normally

doesn’t apply

Tests are not written ahead of time – fast, flexible

You log as you go – requires practice, new habit

Does knowing more about the intended use help you do

exploratory testing better? More efficiently?

28


Thank you for your time.

29

test automatisering and exploratory test - testtech.dk · test automatisering and exploratory test...

Documents