how to learn the history of software testing

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

How to Learn The History of Software Testing

Keizo Tatsumi

2010-12-19 Translated into English on July 2014

If it wasn't useful, it wouldn't be learned.

If it couldn't ever be interesting, it wouldn't

deserve to be learned.

WACATE 2010 Winter

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Agenda

Prehistory

The History of Computers and Software Engineering

The Growth of Software Testing

The History of Testing Techniques

The History of Testing in Japan

The Forefront of Software Testing Research

Concluding Remarks

(C) K. Tatsumi 2014 2

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Prehistory

The First Computer

Charles Babbage’s Analytical Engine

• Conceived in 1837 (unfinished)

• The input via punched cards

• The output via a printer, a curve plotter and a bell

The First Programmer

Ada Byron, Lady Lovelace

• Worked on the Analytical Engine (1843)

– Ada’s notes on the Analytical Engine were recognized as a description of a computer and software

(1/2)


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Prehistory

Who was The First Tester ?

(2/2)

The first programmer, Lady Lovelace, who coded for

Charles Babbage's wonderful but unfinished

computer in the nineteenth century, I'm sure often

said, "Just one more week, Mr. Babbage, and it'll be

done." Lucky for her the hardware was never finished,

so she never did have to go beyond desk checking.

(B. Beizer, Software system testing and quality assurance, 1984, p.277)

Boris Beizer wrote: • “The first discussion of testing and debugging predates

computers by almost a century, in the memoirs of lady Ada Lovelace who wrote software for Babbage’s (never completed) mechanical computer.”

(posted to comp.software.testing on Sep. 18 2007)


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Agenda

Prehistory






Concluding Remarks


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING


Timeline http://a-lifelong-tester.cocolog-nifty.com/Chronology/History_of_Software_TestingChronology_20120508_English.pdf


http://a-lifelong-tester.cocolog-nifty.com/Chronology/History_of_Software_TestingChronology_20120508_English.pdf








HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Victor Basili and John Musa, The Future Engineering of Software: A Management Perspective, 1991

1960s: The Functional Era

• IT penetrated institutions

• How to exploit IT to meet institutional needs

1970s: The Schedule Era • Software crisis NATO Software Engineering Conference, 1968

• How to develop software in a timely, planned, and controlled fashion

• Life-cycle models and schedule tracking ex. Royce waterfall model, Quality models

1950 1980 1990 1960 1970 2000 2010

Schedule Era Cost Era Quality Era Functional Era

Historical Division by Management Perspective (1/2)


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

1980s: The Cost Era

• Hardware costs decreased, PC created a mass market

• The importance of productivity in software development increased

• Various cost models ex. Function point, Putnam model, COCOMO

1990s: The Quality Era

• The increased dependence of institutions on information processing

• The consumer mass market increases the demands on quality

1950 1980 1990 1960 1970 2000 2010

Schedule Era Cost Era Quality Era Functional Era

Historical Division by Management Perspective (2/2)


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Software Engineering Evolution

Barry Boehm,

A Hegelian View of Software Engineering Evolution,

2006

Autonomy;

Bio-

Computing

1990’s 2010’s 2000’s 1970’s 1980’s 1960’s 1950’s

COTS

Software

as Craft

Theses

Syntheses

Antitheses

Formality,

Waterfall

Productivity;

Reuse;

Objects;

Peopleware

Plan-

Driven

Software

Maturity

Models

Agile

Methods

Engineer

Software

like

Hardware

Risk-Based

Agile/Plan-

Driven Hybrids;

Model-Driven

Development

Integrated

Sw-systems

Engineering

Value-Based Methods;

Collaboration; Global

Development; Enterprise

Architectures Software

Differences,

Engineer

Shortages

Many defects

Scalability,

Risk Mgmt.

Prototyping

Time to Market,

Rapid Change

Scalability

Domain Engr.

Risk Mgmt.

Compliance

Process Overhead

Software

Value-Add

Soft

SysE

Global

Systems

of

Systems

B. Boehm, A View of 20th and 21st Century Software Engineering, 2006


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Agenda

Prehistory






Concluding Remarks


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING


D. Gelperin and W. Hetzel,

The Growth of Software Testing, 1988

Five evolutionary periods divided by landmark literatures

1. The Debugging-Oriented Period ( -1956)

2. The Demonstration-Oriented Period (1957-1978)

3. The Destruction-Oriented Period (1979-1982)

4. The Evaluation-Oriented Period (1983-1987)

5. The Prevention-Oriented Period (1988- )


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

The Debugging-Oriented Period

Testing was not differentiated from debugging

Evaluation

Demonstration Destruction

Debugging Prevention

1950 1980 1990 1960 1970 2000 2010

S. Gill, The diagnosis of mistakes in programmes on the EDSAC, p.539, 1951

" Software problems were submerged in the concern for hardware reliability."

"The difficulty lies not in detecting the presence of a mistake, but in diagnosing it."

You "wrote" a program and then you "checked it out."

• Program checkout, debugging, and testing were not clearly differentiated.

McCracken, Digital Computer Programming, 1957 (The earliest programming text)

• Techniques referred to as today‘s debugging and testing are described in "Chapter 13 Program Checkout".


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

The Demonstration-Oriented Period

Testing to make sure that the software satisfies its requirements

C. Baker, Review of "Digital Computer Programming", 1957

• "McCracken fails to distinguish the two phases of program checkout." – Debugging: The process of making sure that the program does what the

coder meant it to do. – Testing: The process of making sure that the program solves the problem

it is intended to solve.

Big Projects (late 1950s-1960s)

• SAGE (US air defense control system), SABRE (Computer reservation system), NASA's Mercury, Gemini, and Apollo projects

• IBM OS/360 ref. F. Brooks, "Mythical Man-Month"

"Software Engineering"

• NATO Software Engineering Conferences (1968, 1969) – "Testing shows the presence, not the absence of bugs." (Dijkstra,1969)

(1/2)

Demonstration Debugging Evalua

tion Destruction

Prevention

1950 1980 1990 1960 1970 2000 2010


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

The Demonstration-Oriented Period

Growth of Professionalism

• Debugging techniques

– Debugging Techniques in Large Systems Symposium, 1970

• The first formal conference on software testing

– The Computer Program Test Methods Symposium, 1972

• The first book on software testing

– Hetzel (Ed.), "Program Test Methods," 1973

Growth of Software Testing Research

• The fundamental theorem of software testing

– Goodenough and Gerhart, "Toward a Theory of Test Data Selection," 1975

• The first workshop on software testing

– Software Testing and Test Documentation Workshop, 1978

The origin of ISSTA (International Symposium on Software Testing and Analysis)

cf. W. Wulf, Concise definitions of software quality attributes, 1973

B. Boehm, et al., Software quality characteristics, 1973


tion Destruction

Prevention

1950 1980 1990 1960 1970 2000 2010

(2/2)


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

The Destruction-Oriented Period

Testing to detect implementation faults

G. Myers, The Art of Software Testing, 1979 • "Testing is the process of executing a program with the

intent of finding errors"

• "Since exhaustive testing is out of the question, the objective should be to maximize the yield of the testing investment by maximizing the number of errors found by a finite number of test cases."

• "Test-case design is so important because complete testing is impossible; a test of any program must be necessarily incomplete. The obvious strategy, then, is to try to make tests as complete as possible."

1950 1980 1990 1960 1970 2000 2010


tion Destruction

Prevention


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

The Evaluation-Oriented Period

Testing to evaluate products during the software lifecycle FIPS 101, Guideline for Lifecycle Validation, Verification, and

Testing of Computer Software, 1983

1950 1980 1990 1960 1970 2000 2010


tion Destruction

Prevention

• "A VV&T methodology is a procedure of review, analysis, and testing employed throughout the software lifecycle from software planning through the end of software use to ensure the production and maintenance of quality software."

• "No single VV&T technique can guarantee correct, error-free software. However, a carefully chosen set of techniques for a specific project can help to ensure the development and maintenance of quality software for that project."


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

The Prevention-Oriented Period

Testing to prevent faults in requirements, design, and implementation

W. Hetzel, The Complete Guide to Software Testing (2nd Ed.), 1988

1950 1980 1990 1960 1970 2000 2010


tion Destruction

Prevention

• STEP (Systematic Test and Evaluation Process) A life cycle prevention model that sees testing parallel to

development with an activity sequence containing planning, analysis (setting test requirements or objectives), design (specifying an architecture for the set of tests and details

of individual cases and procedures), implementation (acquiring or developing test data, procedures,

and test support software), execution (running and rerunning tests and determining the

results), and maintenance (saving and updating the tests as the software

changes).

cf. W-Model (C) K. Tatsumi 2014 17

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Phases in a Tester‘s Mental Life

Phase 0 - There‘s no difference between testing and debugging. Other than in support of debugging, testing has no purpose.

Phase 1 - The purpose of testing is to show that the software works.

Phase 2 - The purpose of testing is to show that the software doesn't work.

Phase 3 - The purpose of testing is not to prove anything, but to reduce the perceived risk of not working to an acceptable value.

Phase 4 - Testing is not an act. It is a mental discipline that results in low-risk software without much testing effort.

B. Beizer, Software Testing Techniques, 2nd Ed., 1990


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Agenda

Prehistory






Concluding Remarks


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Testing Techniques : Beginnings

The state of the art in software testing before 1970s

< The early 1960s >

Published papers (cumulative)

• -1969: 57 papers (*1) cf. -1973: 200 papers (*2), -1977: 400 papers+ (*3)

• There are few papers on software testing techniques.

• Many papers only address testing in a peripheral manner such as test procedures or automation.

< The late 1960s >

IBM OS/360 Project (F. Brooks, "Mythical Man-Month")

• Released in 1966. MVT, multi tasking OS, was released in 1967.

• A lot of software testing work must have been carried out.

• Much know-how about test process, test techniques, and test management must have been accumulated.

(1/2)

(*1) W. Elmendorf, "Program Testing – A Bibliography of Published Literature, 1962–1968", 1969

(*2) W. Hetzel, Number of references in "Program Test Methods", 1973

(*3) E. Miller, Number of references in "Tutorial: Program Testing Techniques", COMPSAC '77, 1977


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Testing Techniques : Beginnings

The test control process, IBM W. Elmendorf, "Controlling the functional testing of

an operating system," 1969

(2/2)

• Elmendorf started to work on software testing for OS/360 in 1965.

• A disciplined test control process

- From the "laissez-faire approach" to a disciplined approach

- From the "testing-is-an-art" approach to a scientific approach

A pioneering work of a preventive test

approach like W-Model


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Equivalence Partitioning / Boundary-Value Analysis

<History>

1979: The names and concepts of EP and BVA were first introduced by G. Myers of IBM in his "The Art of Software Testing."

Similar concepts had already appeared in Elmendorf‘s paper in 1967.

• External Interaction Variations

"... If the actions taken by the control program are predictable, acceptable ranges of field values, field sizes, field repetitions, etc., will be identified as variations at, and just beyond, the extremes of each range. There are external limits which, if exceeded, cause unpredictable actions."

W. Elmendorf, Evaluation of the Functional Testing of Control Programs, 1967


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Cause-Effect Graphing

<History>

1970: W. Elmendorf developed Cause-Effect Graphing by adopting a hardware logic testing method.

• Extending the concepts to cover software specific issues such as constraints on the inputs

• Development of a test case generation tool

– TELDAP (TEst Library Design Automation Program) was developed for generating test cases from a cause-effect graph, based on the techniques for generating test patterns for the logic circuits in hardware.

1979: Cause-Effect Graphing became widely known after publication of Myers‘ book which introduced this technique.

W. Elmendorf, Automated Design of Program Test Libraries, IBM-TR-00.2089, 1970


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Decision Table Testing

<History>

Circa 1958: Decision tables were developed by General Electric and Sutherland corp. individually

• To express the logic of product design, operation planning, management decision rules, etc. .

1960: GE developed TABSOL (Tabular systems oriented language) which generated program codes from decision tables.

1965: The early paper to apply DT to software testing

• Test categories and test parameters are entered to condition fields. Consequences and test actions are entered to action fields.

1975: Condition table method by Goodenough & Gerhart

• A technique derived from decision table techniques for developing and describing test predicates

B. Grad, Tabular form in decision logic, DATAMATION, July 1961

B. Scheff, An application of decision tables as the source language for automatic testing, 1965

J. Goodenough and S. Gerhart, Toward a theory of test data selection, 1975

T. Kavanagh, TABSOL A fundamental concept for system-oriented languages, 1960


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Orthogonal Array / Pairwise Testing

<History>

1920s-30s: Design of Experiments (DoE), R. Fisher

1940s: Concepts of orthogonal arrays as DoE, C. Rao

1950s: Quality engineering (Taguchi Methods)

1980s: Applying DoE to software testing

• 1984: Operating system testing, Fujitsu Ltd., Japan

• 1985: Ada compiler testing, R. Mandl, US

1990s: Spreading in the US from the mid 1990s

• 1992: OATS, AT&T Bell Lab.

• 1994: CATS, AT&T

• 1994: AETG, commercial tool, Belcore

S. Sato and H. Shimokawa, Methods for setting software test parameters using the DoE, 1984

R. Mandl, Orthogonal Latin squares: an application of experiment design to compiler testing, 1985

R. Brownlie, J. Prowse, M. Phadke, Robust testing of AT&T PMX/Starmail using OATS, 1992

G. Sherwood, Effective testing of factor combinations, 1994

D. Cohen et al., The automatic efficient test generator (AETG) system, 1994


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Timeline of Combinatorial Testing

(1957: Decision table)

～1967: Equivalence partitioning, Boundary value analysis

1970: Cause Effect Graph [Elmendorf]

(1983～)1984: Orthogonal array / Combination Table

[Satoh, Shimokawa]

1987: Test Case Design Support System [Tatsumi]

(198x～)1992: OATS [Brownlie, Prowse, Phadke]

(1990～)1994: CATS [Sherwood]

(1992～)1994: AETG [Cohen, et. al]

1998: IPO [Lei, Tai]

2000: Covering arrays [Williams]

2000: CTE XL [Daimler Chrystler]

(2000～)2004: PICT [Microsoft]

2007: FireEye 2009:ACTS (IPOG) [Lei, Kuhn]

AT&T, Bellcore

1988: Category-partition method [Ostrand, Balcer]

1993: Classification-tree method [Grochtmann, Grimm]

(1976: Test Factor Analysis Method) [Fujitsu]

(199x～)2004: Orthogonal Array (HAYST method)

[Akiyama(Fuji Xerox)]

1980 1990 2000 2010 1950 1960 1970 ‘85 ‘95 ‘05

Combinatorial techniques

Input condition analysis techniques

(1983～)1985: Orthogonal Latin Squares [R. Mandl]

Fujitsu


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Control Flow Testing (1/2)

<History>

1960s: Applying graph theory to software development

• 1960: Compiler development, Karp

• 1963: Applying to software test design, Miller & Maloney

• 1976: Cyclomatic complexity and Basis path testing, McCabe

1960s: Measuring test coverage

• Code coverage tool development in IBM Poughkeepsie

– C. Warner Jr., Evaluation of program testing, 1964

Earliest known use of a hardware instruction coverage monitor: COBOL and FORTRAN source. (from Beizer)

– I. Hirsh, MEMMAP/360, 1967

Earliest known description of a software statement and branch coverage analyzer. (from Beizer)

R. Karp, A note on the application of graph theory to digital computer programming, 1960

J. Miller and C. Maloney, Systematic mistake analysis of digital computer programs, 1963

T. McCabe, A Complexity Measure, 1976


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Control Flow Testing (2/2)

Early 1970s: Dynamic analysis systems (Coverage measurement tools)

• 1972: PACE (Product Assurance Confidence Evaluator), TRW

• 1972: PET (Program Evaluator and Tester), McDonnell Douglas

• 1974: RXVP (Requirement Evaluation & Verification Package), General Research

J. Brown, et al., Automated Software Quality Assurance: A Case Study of Three Systems, 1972

L. Stucki, Automatic Generation of Self Metric Software, 1972

E. Miller, et al., Structurally based automatic program testing, 1974

Early 1970s: Coverage criteria

• 1972: TER (Test Effectiveness Ratio) [Brown]

• 1975: C0, C1, C2, … [Miller]

(1975-1977) C0 : Programmer‘s intuition, C1 : Every statement in a program exercised at least once, C2 : Every program predicate outcome exercised at least once, ...

(1977-) C0 : Every statement executed at least once, C1 : Every segment executed at least once, C1

p : Every predicate term executed for each

outcome, C2 : C1 + interior and boundary tests for each iteration, ...

J. Brown, Practical applications of automated software tools, 1972

E. Miller, The Art and the Theory of Program Testing, 1975

E. Miller, Coverage levels, in "Infotech State of the Art Report: Software Testing," Vol. 1, 1979


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Data Flow Testing

<History>

Late 1960s: Data flow analysis techniques were developed in the field of optimizing compilers. F. Allen( IBM)

1974: Applying data flow analysis techniques to software testing, Osterweil & Fosdick (Univ. of Colorado)

1982-: Data flow criteria, Rapps & Weyuker Coverage criteria subsumption including control flow criteria and data flow criteria

F. Allen and J. Cocke, A program data flow analysis procedure, 1976

L. Osterweil and L. Fosdick, Data Flow Analysis as an Aid in Documentation, Assertion Generation, Validation and Error Detection, 1974

S. Rapps and E. Weyuker, Data Flow Analysis Techniques for Test Data Selection, 1982


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

State Transition Testing

<History>

Originally, state transition diagrams and state transition tables were developed to represent finite state machines (FSM).

1956: Moore‘s "Gedanken-experiments on sequential machines"

• The origin of FSM model based testing

1978: Coverage criteria "n-switch cover" proposed by Chow

E. Moore, Gedanken-experiments on sequential machines, 1956

T. Chow, Testing software designs modeled by finite-state machines, 1978


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Agenda

Prehistory






Concluding Remarks


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

The History of Testing in Japan (1/3)

1958, The Earliest paper on software testing

Fujiwara, A report on program testing on the IBM 704 large scale electronic computer (in Japanese), 1958

• Preliminary program testing for a numerical weather prediction system of the Japan Meteorological Agency

1964, Discussion on the program inspection

Information Processing Society of Japan, Special features : Software ,IPSJ Magazine, 1964

• Necessity of independent section for software inspection • Role of software quality assurance section • Process improvement through the feedback from QA section

1969, Hitachi’s Software Works (Factory)

Establishing the function of software quality assurance in the software factory system

1971: Also, Fujitsu created a formal product inspection procedures for release to the customer.


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING


1972, An Approach for Software Inspection A. Kanno, An Approach for Software Inspection, 1972

• Concepts, activities and techniques in software quality assurance department in Hitachi software factory

1974, Development of QA techniques K. Sakata(Hitachi), Formulation for predictive methods in

software production control, 1974 • Static prediction and failure rate transition model

Quality prediction by reliability growth curve or Gompertz curve

• Dynamic prediction: quality probe

Quality prediction by "quality probe" that comprises a small percentage of the complete regular test


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING


1980-, R&D of testing techniques and tools

1980: AGENT(Automated GENeration system for Test cases), Hitachi • Test case generation tool based on cause-effect graphing

1984: AGENT Function Diagram, Hitachi • Representation of functional specification (dynamic part -> state transition

diagram, static part -> decision table or cause-effect graph)

1984: Application of DoE to software testing, Fujitsu • Test case generation technique and tool for application of DoE

1988: CFD(Case Flow Diagram or Cause Flow Diagram), NEC • External and internal specifications are organized into Cause Flow Diagram,

then decision table is generated from the diagram.

1991, Japan's Software Factories

M. Cusumano, Japan's Software Factories: A Challenge to U.S. Management, 1991

• A management research work about Japanese software development and a survey of the making of Japan‘s computer industry, Hitachi, Toshiba, NEC, and Fujitsu etc.


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Agenda

Prehistory






Concluding Remarks


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

The Forefront of Software Testing

A. Bertolino, "Software Testing Research: Achievements, Challenges, Dreams," 2007

• 29th ICSE, Future of Software Engineering track

• Bertolino is the KA co-leader for Software Testing in the IEEE Guide to the SWEBOK

The software testing research roadmap

• Achievements The most notable achievements from past research

• Dreams The desired destination

• Challenges The challenges faced by current and future testing research


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

The Many Faces of Software Testing

Software testing:

Observing a sample of executions, and giving a verdict over them

6 aspects of software testing

WHY: test objective looking for faults?, the product can be released?, evaluate the usability?

HOW: test selection ad hoc, at random, or systematic way

HOW MUCH: test adequacy coverage analysis, reliability measures

WHAT: levels of testing unit test, component/subsystem test, integration test

WHERE in house, simulated environment, the target final context

WHEN when is it in the product lifecycle that we perform the observations?


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Software Testing Research Roadmap

A. Bertolino, “Software Testing Research: Achievements, Challenges, Dreams,” FOSE’07, p.85-103, 2007, Figure 1

Testing

process

Reliability

testing

WHY How How much What Where When

Protocol

testing

Test

criteria

Comparison among

test criteria

Co

mp

on

en

t-bas

ed

testin

g

Ob

ject-o

rien

ted

testin

g

Education of software testers

Testing patterns

Controlling evolution

Leveraging user population

and resources

Understanding the

costs of testing

Test input

generation

On-line testing

Test

oracles Model-based testing

Anti-model-based testing

Explicit test

hypotheses

Test effectiveness

Empirical body of evidence

Compositional

Testing

Domain-specific

test approaches

Achievements Challenges Dreams

Efficacy-

maximized

test engineering

100% automatic

testing

Test-based

modeling

Universal

test theory


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Achievements

Testing process

Test criteria

Comparison among test criteria

Object-oriented testing

Component-based testing

Protocol testing

Reliability testing


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Dreams

(1) Universal test theory A comprehensive theory which is useful to backup and nourish

test technology

(2) Test-based modeling Test-based modeling is closely related to the old idea of

“Design-for-testability.”

(3) 100% automatic testing Advanced techniques for generating the test inputs, innovative

support procedures to automate the testing process

(4) Efficacy-maximized test engineering Practical testing methods, tools and processes for development

of high quality software


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Dream: Universal test theory

<Challenges>

Explicit test hypotheses • Making explicit for each technique which are its underlying

assumptions

Test effectiveness • Assessing the effectiveness of existing and novel test criteria

Compositional testing • Reusing the test results observed in the separate testing of the

individual levels

• A foundational theory for component-based software reliability

Empirical body of evidence • An empirical body of knowledge which is at the basis for

building and evolving the theory for testing


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Dreams: Test-based Modeling

<Challenges>

Model-based testing • Combining different styles of modeling (transition-based,

pre/post condition-based and scenario-based)

• Integrating model-based testing practice into current software processes

Anti-model-based testing • A model is derived a posteriori via testing for the cases in which

the models do not exist or are not accessible, such as for COTS or legacy components.

Test oracles


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Dream: 100% automatic testing

<Challenges>

Test input generation • Model-based test generation, random test generation, search-

based test generation

Domain-specific test approaches

On-line testing • Monitoring a system‘s behavior in real life operation using

dynamic analysis and self-test techniques


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Dream: Efficacy-maximized test engineering

<Challenges>

Controlling evolution

Leveraging user population and resources

Testing patterns

Understanding the costs of testing

Education of software testers


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Transversal challenges

<Challenges>

Testing within the emerging development paradigm • Testing of service-oriented applications in the Service-oriented

Computing

• A special importance for testing of services (monitoring the real-world execution is the only way to observe the application behavior)

Coherent testing of functional and extra-functional properties • Conventional functionality testing does not provide for any

notion of time nor tackle resource usage and workloads.

• For the model-based approach, we need effective ways to enhance models with desired extra-functional constraints.


HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Concluding Remarks

The Analects - Chinese Text Project http://ctext.org/analects

• The Master said, "If a man keeps cherishing his old knowledge, so as continually to be acquiring new, he may be a teacher of others."

• The Master said, "Is it not pleasant to learn with a constant perseverance

and application? Is it not delightful to have friends coming from distant quarters? Is he not a man of complete virtue, who feels no discomposure though men may take no note of him?"

• The Master said, "Learning without thought is labor lost; thought without

learning is perilous."

• The Master said, "You, shall I teach you what knowledge is? When you

know a thing, to hold that you know it; and when you do not know a thing, to allow that you do not know it - this is knowledge."


http://ctext.org/analects

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

HIS

TO

RY

OF

SO

FT

WA

RE

TE

ST

ING

Now is Your Turn to Make History!

Thank you !
