CUCUMBERSPEC A TOOL TAKES YOUR BDD TO THE NEXT LEVEL

30-5-2015

PROJECT TEAM

2

• Adrian Yankov

– Graduation Intern

• Kyra Hameleers

– Test Engineer @ Nspyre

• Rachid Kherrazi

– MBT competence Leader @ Nspyre

– Project Leader of this project

GOALS

• Sharing knowledge on 3 topics

– Behaviour-Driven Testing

– Model-Based Testing

– Combined strengths

• Encourage the usage of Model-Based Testing

CONTENT

• Introduction

– Behaviour-Driven Testing(BDT)

• What is it and why?

– Model-Based Testing(MBT)

• What is it and why?

– Using Gherkin in MBT

• CucumberSpec Tool

• Testing a drone with this approach

4

BEHAVIOUR-DRIVEN TESTING

• An example

5

BDT LANGUAGE AND BENEFITS

Why?

• Main benefit – the team gets together and writes down a

common, shared knowledge in a well-known format.

• Generates test cases from a high-level formal human

language.

Tools

• There are several tools such as SpecFlow, Behave and

Cucumber etc.

6

EXAMPLE: SPECFLOW

7

SOME NICE FEATURES

8

• Data Parameterization

– Supports placeholders.

– Contained within < >

Example:

BDT WITH SPECFLOW

9

GUI

Application/

database layer

Test

inte

rface Physical test

cases

Logical test

cases (Gherk

in)

Requirements

Features

Use cases

Test Engineer Architect/

designer

SUT

create

MODEL-BASED TESTING

Manual Automatic

Testcase specification (design van logische test cases)

Testcase execution (includes reports)

Testcase generation

Model

base

d t

est

ing

Tra

dit

ional

manual

test

ing

Tra

dit

ional

auto

mati

c t

est

ing

Manual C# Scripting

Testing skills + Scripting skills + Modeling skills

Modeling

Lightweight

Modeling languages

Generate

Scripting

10

LOOK AND FEELING (SPEC EXPLORER)

11

MBT WITH SPEC EXPLORER

12

GUI

Application/

database layer

Test

inte

rface

Automatic

generation

of test

scripts

Test Model

Model includes

expected

behavior

Requirements

Features

Use cases

Gherkin

Test Engineer Architect/

designer

SUT

create create

BDT IN COMBINATION WITH MBT

Our tool CucumberSpec generates a SpecExplorer test

model from the separate scenarios.

13

CucumberSpec Isolated scenarios

USING GHERKIN IN MBT

Benefits

Visualization of interaction space, generate more and better (exhaustive) testcases

by using different smart exploration strategies to suit the tester’s needs.

14

GUI

Application/

database layer

Test

inte

rface

Automatic

generation

of test

scripts

Test Model

Model includes

expected

behavior

Requirements

Features

Use cases

Gherkin

Test Engineer Architect/

designer

SUT

CucumberSpec

check create

CUCUMBERSPEC SUMMARY

Idea of our CucumberSpec

• Generates a Test model from a feature file, Connecting the separate scenarios in a model

• Intention is to support most features (parameterization, tags….)supported by BDD tools like Behave and SpecFLow

• + A lot of “Model Based” Benefits,

• Generate more tests cases based on different coverage used in MBT

– Testing interaction between the scenarios

– More data permutations and combinations

– This allows exhaustive testing

• No refinement is needed.

• Fully automatic process.

16

AR.DRONE

17

Security

Guard

AR.DRONE

18

Mobile AR.Drone

SW Hardware

Router Camera 1 Motors

Camera 2 Hoogte

Sensor GPS

Application

UI

Controller

Communication

Business Layer

Data Acces Layer

Commun

ication

Layer

REQUIREMENTS

• Functional requirements:

1. Takeoff and Land via Android commands

2. Fly around via Android commands

3. Always be able to use the emergency command

• Non-Functional requirements:

1. Usability

2. Reliability

3. Security

19

To keep it simple in this PoC, we

focused only on a limited

number of functional

requirements.

TEST SCOPE

• Functional requirements

1. Takeoff and Land via Android commands

1. Takeoff

2. Land

2. Fly around via Android commands

1. Left/Right

2. Up/Down

3. Forward/Backward

4. Turn Lef/right

3. Always be able to use the emergency command

20

TESTING THE DRONE

21

Actions:

1. Take off

2. Emergency

3. Land

4. R: Right

5. L: Left

6. F: Forward

7. B: Backward

8. l: Turn left

9. r: Turn right

10. u: Up

11. d: Down

TEST MODEL States:

1. Take off

2. Flying

3. Landing

4. Landed

5. Emergency

22

Actions:

1. Take off

2. Emergency

3. Land

4. R: Right

5. L: Left

6. F: Forward

7. B: Backward

8. l: Turn left

9. r: Turn right

10. u: Up

11. d: Down

Take off Flying Landing Landed Emergency

Take off x Reached 30 cm land X Emergency

Flying x

Right/ Left/ Up/ Down/

Forward/ Backward/ Turn

left/ Turn right land x Emergency

Landing take off, height<0 take off, height>0 X Reached 0 cm Emergency

Landed take off X X x Emergency

Emergency x Emergency, height>0 X Emergency, height<0 x

Some issues with traditional test design:

Traditional state transition test technique does not scale

high coverage (>1-switch) is complex, costly and difficult to

maintain

CREATE GHERKIN (SMALL EXAMPLE FOR EXPLANATION PURPOSES)

23

0-switch state transition test

cases in a high-level formal

human language(Gherkin).

Landed Flying

Landed x Take off

Flying Land Forward/

Backward

GHERKIN SCENARIOS (SMALL EXAMPLE FOR EXPLANATION PURPOSES)

24

Isolated scenarios

GENERATED SPEC EXPLORER (SMALL EXAMPLE FOR EXPLANATION PURPOSES)

25

Visualization of

interaction space,

This allows generation

of more and better

(exhaustive) testcases

by using different smart

exploration strategies

(supported by the MBT

tool) to suit the tester’s

needs.

COMPARISON (SMALL EXAMPLE FOR EXPLANATION PURPOSES)

26

Isolated scenarios model

CucumberSpec

BACK TO TEST SCOPE

27

Take off Flying Landing Landed Emergency

Take off x Reached 30 cm land X Emergency

Flying x

Right/ Left/ Up/ Down/

Forward/ Backward/ Turn

left/ Turn right land x Emergency

Landing take off, height<0 take off, height>0 X Reached 0 cm Emergency

Landed take off X X x Emergency

Emergency x Emergency, height>0 X Emergency, height<0 x

CREATED GHERKIN FILE

28

GENERATED MODEL

29

TEST CASES

30

TEST EXECUTION

• Test code can be generated

• The Tool can generate test

scripts for different execution

tools e.g NUnit/JUnit.

• No code to test it too

31

CONCLUSION

33

Metrices Cucumber MBT CucumberSpec

Approach

using gherkin files and action

definition to generate and

execute test suite

using models to

generate and execute

test suite

using gherkin files to generate

models, and generate and

execute test suite

Technique Behavior Driven Testing Model Based Testing

Combining strength of BDD and

MBT

Modeling Create gherkin files Create test model Generate test model

Effort High Medium/High Medium

Coverage 0-switch Transition coverage Transition coverage

Number of testcases 20 7 7

NEXT STEP • Next step: Explore automatic test execution via

Xamarin/Calabash (benefit cross-platform testing)

• We are looking for a pilot project based on real industry case from industry

• If you are interested please contact us

• or send a mail to Rachid.Kherrazi@Nspyre.nl

34

SUMMARY

• Combining the best of 2 world

– Behavior-Driven Testing

– Model-Based Testing

35

CucumberSpec Isolated scenarios

36