Hands on Experiment Design

Foutse Khomh, Queen’s University, CanadaMassimiliano Di Penta, University of Sannio,

Italy

Use of acceptance test during change tasks

Usage of natural language to specify (change) requirements: current state of the practice highly inaccurate error prone

“Agile movement” advocates [Melnik et alt. 2004]: Acceptance test cases constitute an expressive

form of documentation Acceptance test cases are “talking”

representation of the requirements We focus on evaluating whether Fit

acceptance tests are useful in maintenance/evolution tasks

Framework for Integrated Test

The most well-known open source implementation of the table-based acceptance testing approach. A table represents a set of test cases.

Fit lets: customers/analysts write “executable”

acceptance tests using simple HTML tables (Fit tables).

developers write “Fixtures” to link the test cases with the actual system itself.

to compare (Test Runner) these test cases with actual values, returned by the system, and highlights the results with colors.

Red for failures Green for passed tests.

“The complete picture”

Text Reqs.

Fit Table

Fixture

Analyst(and customer) (i, o)

System

io’

Developer

o ≠ o’

Test Runner

Output Fit TableO = expected output

O’ = actual output

failure

Examples of Fit Tables

Core Fit tables:Column Fit tables for testing

calculationsAction Fit tables for testing

user interfaces or workflows;

Row Fit tables to check the result of a query

(some) other Fit tables:Summary Fit tablesHtml Fit tablesCommand line Fit tables…

fit_tests.DiscountStructure

small bags

beverage

discount

total price()

2 Coffee 0 1.24

4 Tea 0 2.48

5 Coffee 1 2.1

5 Tea 0 3.1

7 Coffee 1 3.34

7 Tea 0 4.34

fit.ActionFixture

start fit_tests.VerifySupply

enter type of product Coffee

check number of small bug remained 10

enter number of box 5

press buying boxes

check number of small bug remained 310

Column Fit table

Action Fit table 1 box = 60 small bags

small bag price = 0.62

FitNesse

A collaborative testing and documentation tool.

It supports Java (eclipse plug-in), .Net, C++

It Combines Fit with a Wiki Web for writing “natural language requirements” + Fit tables.

It provides a simple way to run tests (Fit tables) and suits.

It supports sub Wikis for managing multiple projects.

http://fitnesse.org

Text only vs. Text + Fit

Text only Text + Fit

Change Requirement: Change price of Boxes

The vendor of boxes of beverages changed his selling policy. Each five bought boxes (of the same type) one is added as a gift.

Change Requirement: Change price of Boxes

The vendor of boxes of beverages changed his selling policy. Each five bought boxes (of the same type) one is added as a gift.

fit.ActionFixturestart fit_tests.VerifySupplyenter type of product Coffeecheck

number of small bug remained 310

enter number of box 11

press buying boxescheck

number of small bug remained 960

check cash account 504

What to compare

Where do we start?

We have an idea/conjecture about a cause effect relation

We have a theory Thus we can formulate a hypothesis And to test it.. we run an experiment!

The planning steps follows...

Planning: steps

Contextselection

Definition

Hypothesisformulation

Variableselection

Selection ofsubjects

Experimentdesign

InstrumentationValidity

evaluationExperiment

design

Study planning

Context Selection

Select the type of study to perform: survey, case study, and experiment . In this training we focus on experiment (more details @page 10 of the lecture slides)

Identify the objects and subjects of the study (more details @ page 40-49 of the lecture slides)

Hypothesis formulation

Formulate the null hypothesis.

(more details @ page 50-54 of the lecture slides)

Variables Selection

Dependant variables Used to measure the effect of treatments Derived from hypotheses

(more details @ page 56-68 of the lecture slides)

Variables Selection

Independant variables Variables we can control and modify The choice depends on the domain knowledge One independent variable is the main factor of our

experiment Other independent variables are the co-factors

(more details @ page 70-72 of the lecture slides)

Experiment Design Is the set of treatment tests

Combinations of treatments, subjects and objects

Defines how tests are organized and executed

Based on the formulated hypotheses Experimental design is based on three

principles1. Randomization2. Blocking3. Balancing(more details @ page 74-98 of the

lecture slides)

Validity evaluation Answer the following crucial questions :

To what extent are our results valid? They should be at least valid for the population of interest Then, if we could generalize…

Mitigates the following threats to validity1. Conclusion validity (C)2. Internal validity (I)3. Construct validity (S)4. External validity (E)

Prioritize threats to validity: Many threats are conflictual

(more details @ page 101- 121 of the lecture slides)

SummaryWhat to produce: Experiment definition Context selection Hypotheses formulation Identification of dependent and

independent variables Experiment design Experiment procedure and identification

of the required material

Operation After having designed an experiment we

need to execute it We are in touch with subjects for the first

time Besides the pre-experiment briefing and

training

Although the design and plan are perfect, everything depends on the operation If something goes wrong in a couple of hours

we could waste months of work…

Experiment operation: steps

Preparation

Experimentdesign

Execution

Datavalidation

Experimentdata

Experiment operation

Preparation Obtain consent from participants Do not disseminate sensitive data Reward in some way participation Before the experiment, it is advisable to show

a short presentation explaining the experiment and its objectives

Introduce the objects and the instrumentation Describe in detail the steps of the experiment

(more details @ page 125- 130 of the lecture slides)

Execution Different ways to execute an experiment

Online You can actually monitor the experiment

Offline Distribute the task via email and wait for results

Run a post-experiment questionnaire to understand whether anything went wrong with

clarity of objectives material, time available, tasks and to collect some qualitative information.

(more details @ page 131- 136 of the lecture slides)

Data validation Once the experiment has been

completed, we need to do a consistency check on the collected data Were treatments correctly applied? Did subjects understand the provided forms? Did subjects correctly fill the forms? Remove subjects that

Did not participate to the experiment Exhibited a weird behavior (e.g. did not pay

attention to the task) Try to have a quick look at data as soon

as possible At least using descriptive statistics

Replication Replication is essential in experimentation

You should document your experiments so that others can

Repeat the experiment with different subjects Replicate your statistical analysis

Look at the non-replicable case of the cold fusion!

(more details @ page 138-139 of the lecture slides)