Upload File

model-based testing for the masses · –watir – graphviz • half-hearted, piecemeal adoption...

  • Home
  • Documents
  • Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the
29
Model-Based Testing for the Masses Harry Robinson

Upload: others

Post on 14-Jun-2020

2 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

Model-Based Testing for the Masses

Harry Robinson

Page 2: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

Is the Industry Ready for Model-Based Testing?

Page 3: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

The State of Testing Today

$ 59 500 000 000€ 49 400 000 000

“It's not going to work.”Jim Allchin, Senior MS executive,

Wall Street Journal 23 Sept 2005

Windows Vista Release Pushed Back to January 2007

Tuesday, March 21, 2006

Page 4: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

But Formality Intimidates Testers

Page 5: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

And Big Models Intimidate Testers

Model Based Testing for Real: The Inhouse Card Case StudyPretschner, Slotosch, Lötzbeyer, Aiglstorfer, Kriebel

Page 6: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

“Modelling is hard”

- AGEDIS Case Studies: Model-Based Testing in Industry

And …

Page 7: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

A Great Industrial Technology should be …

• Cheap

• Easy to apply

• Measurable

Page 8: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

CheapTest teams typically have

• Little budget to start

• Less after buying shelfware

• Very little budget for methodologies

Page 9: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

Easy to Apply (1)

• This will depend on the tester’s– Background

– Methodology training

Page 10: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

Industry Testers: Education

Some college 28%

PhD 1%

Master’s 20%

Bachelor’s 51%2005 Salary Survey

Better Software magazineMarch 2006

Page 11: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

Industry Testers: Area of Study

CS/IS 31%

Other 21%

Liberal Arts 10%

Business 14%Engineering 20%

Math 4%

2005 Salary Survey Better Software magazine

March 2006

Page 12: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

Industry Testers: Methodologies

vs.

Page 13: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

Easy to Apply (2)

• Integration with existing processes– Lightweight

– Incremental approach

pls

test

See “What Do You Mean ‘pls test?’ ”http://www.qualitytree.com/ruminate/041597.htm

Page 14: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

Measurable (1)

Number of test cases created

Number of test cases generated?

Page 15: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

Measurable (2)

Number of bugs found

Number of bugs prevented?

“Nobody Ever Gets Credit for Fixing Problems that Never Happened”- Repenning and Sterman

Page 16: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

Measurable (3)

“Do your testing for nuthin’, find your bugs for free!”- Noel Nyman, Microsoft

Page 17: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

Factors in Favor of MBT

• Time• Machines

Page 18: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

Time

PM Dev Test

PM Dev TestT

bugs

T

bugs

T

bugs

T

bugs

T

bugs

T

bugs

bugs

bugs

bugs

bugs

Page 19: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

Machines

• A typical test machine• Costs € 250 ($300 USD ) per year to buy and operate• Can work 100 hours per week

• A typical test engineer • Earns € 41 500 ($50 000 USD ) per year• Works 40 to 50 hours per week

Page 20: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

0100002000030000400005000060000700008000090000

1 tester, 1 cpu

2 testers, 2 cpus

1 tester, 10 cpus

cpu costtester cost

52000hrs/yr

2080hrs/yr

4160hrs/yr

€ Hours of Testing vs Cost

Page 21: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

0100002000030000400005000060000700008000090000

1 tester, 1 cpu

2 testers, 2 cpus

1 well-paid tester,10 cpus

cpu costtester cost

52000hrs/yr

2080hrs/yr

4160hrs/yr

€ Hours of Testing vs Cost

Page 22: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

Caught in the Middle

MBT

programmers

academics

"Because MBT … requires strong software development skills, when I recruit test engineers, I look for developers and get them excited about testing.”

Jeff Feldstein, Cisco Systems

testers

Page 23: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

What’s Happening Now?

• Home-Brew Models– C# / Python– Watir– Graphviz

• Half-hearted, piecemeal adoption– Whiteboard modeling– Hand-traced test sequences

Page 24: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the
Page 25: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

Where Could Models Take the Industry?

• Better specs

• Better metrics

• Integration of testers into the development process

• Continuous testing

• Testers becoming more technical

Page 26: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

MBT is Entering the Mainstream“These [High Volume Test Automation] techniques are not widely used in

industry, but we believe they have the potential to help us substantially increase the reliability of software.”

Experiments with High Volume Test AutomationCem Kaner, Pat McGee

“Modeling in general seems to be gaining favor; particularly in domains where quality is essential and less-than-adequate software is not an option.”

Model-Based Software TestingJames Whittaker, Ibrahim El-Far

Page 27: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

What Needs to Happen?

Responses:• Easier notation• Incremental approach• Pilots and examples

bool Running = false;bool Scientific = false;

[Action] void SetScientific(bool newScientific)requires Running;requires Scientific != newScientific; {Scientific = newScientific;

}

Spec#

ZIndustry’s issues with modeling:• Too academic• Too process-heavy• Too steep a learning curve

Page 28: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

How Can You Help?• Do case studies of well-known systems

• Publish in QA magazines

• Teach Advanced Software Testing

• Spend time with practitioners

• Encourage your students to study industrial testing

• Share your insights with the industrial community

Page 29: Model-Based Testing for the Masses · –Watir – Graphviz • Half-hearted, piecemeal adoption – Whiteboard modeling – Hand-traced test sequences. Where Could Models Take the

Thank You!

[email protected]


Enhancing Effectiveness of WATIR through Keyword … · Watir, pronounced as “Water” is an open source library for automating web browsers. Watir ... The advantages for automated

Using Graphviz as a Library (cgraph version) · Using Graphviz as a Library (cgraph version) Emden R. Gansner August 21, 2014 1. Graphviz Library Manual, August 21, 2014 2 Contents

Visualisation of Honeypot Data Using Graphviz and Afterglow

Keyword Driven Framework using WATIR

Let's Draw a Graph: An Introduction with Graphviz

Watir&web driver

Formal Language Theory. Homework Read documentation on Graphviz – –

SOLVED Problem With Protégé OWL-S Plugin and Graphviz

Diagram with Graphviz - Actifsource

HD-GraphViz: highly distributed graph visualization on tiled displaysmajumder/docs/icvgip12.pdf · 2014. 2. 12. · HD-GraphViz : Highly Distributed Graph Visualization on Tiled Displays

Watir - tutorialspoint.com · Watir has inbuilt libraries to test page-performance, alerts, iframes test, browser windows, take screenshots etc. Disadvantages of Watir Like any other

Watir Web Strategija

Watir web automated tests

WATIR Presentation - Sumanth Krishna. A

Graphviz and Dynagraph – Static and Dynamic Graph Drawing ... · The algorithms of Graphviz concentrate on static layouts. Dynagraph is a sibling of Graphviz, with algorithms and

Watir The Beginning

Benchmarking Web-testing - Selenium versus Watir and the ... · Benchmarking Web-testing - Selenium versus Watir and the Choice of Programming Language and Browser ... Introduction

Graphviz and TikZ

Test watir

Ruby & Watir

Clean & Dirty Acceptance Tests with Cucumber & Watir

WATIR part2

Drawing graphs with - Graphviz

Андрей Дзыня - "Watir - начало"

What you can do In WatiR

Automated Web Testing With WATIR

DOTTY Documentation - Graphviz | Graphviz - Graph

Graphen visualisieren mit Graphviz - Spline · Was ist das Thema und warum ist es interessant? Verschiedene Lösungsansätze Graphviz Grundlagen Attribute ... É erzeugt Bilder in

Capybara vs watir

Introduction to WATIR

A Technique for Drawing Directed Graphs - Graphviz | Graphviz

Domain Specific Watir Page Objects

Graphviz Tutorial

Bulk and Piecemeal Auction

Watir Recipes Sample