Testing AutomationCFICSE VV07 October 1999

Dr. Terry Shepard

Royal Military College of Canada

Electrical and Computer Engineering

613-541-6000 ext. 6031

Testing Automation Topics

• Motivation for Automation

• Design for Testability

• What can be automated?

• Context and Needs

• Intrusive vs. non-intrusive

• Software Test Technologies Report– Classification of testing tools

Motivation for Testing Automation

• reduce costs

• ensure repeatability of tests

• ensure that test scenarios are followed exactly

• increase number of tests that can be run by a deadline

• improve reliability estimation

Design for Testability

• Common for computer hardware

• Primarily means– Controllability– Observability– Well behaved interfaces between modules

• Determines what can easily be automated– e.g. instrumentation code built into source code

What can be automated?

• Test execution

• Comparison of actual and expected results

• Logging of state at the time of a failed test– recovery for debugging purposes

• Management of test logs

• Collection of metrics

• Version control (in part)

• Test case selection: hard!

Automated support can be provided for other tasks

• Test planning and budgeting

• Test setup procedures

• Test adequacy decisions

• Test scenarios with external dependencies

Context and Need

• In a given project, the context will determine candidate testing tools– constraints include language, platform, OS, …

• Given a set of constraints, a market search and evaluation can be undertaken

• Broad range of applicability and variety of tools makes needs assessment hard

• Cost justification should be applied

Intrusive vs. Non-Intrusive

• Intrusion is generally not possible in an operational setting

• Build black box recorder capabilities into the software?

• Continuing hardware performance improvements will make this increasingly feasible

Software Test Technologies Report

• August 1994

• Software Technology Support Center, Hill AFB, Air Logistics Command, Ogden, Utah– Online at www.stsc.hill.af.mil– STSC has an Annual Conference, publishes

Crosstalk, …

• No update in classification since 1994

1994 STSC Test Tools Classification

• Test Resource Management

• Requirements and Design Test Support

• Implementation and Maintenance Test Support– Compilers– Source Code Static Analyzers– Test Preparation– Test Execution (Dynamic Analyzers)– Test Evaluators

Test Resource Management Tools

• Configuration Managers– test version control– defect list management– change control management

• Project Managers– can help to elevate the visibility of testing early in a

project

Requirements and Design Test Support Tools• Analyzers for Software Plans,

Requirements, and Designs

• System/Prototype Simulators

• Requirements Tracers

• Requirements-Based Test Case Generators

• Test Planners

Implementation and Maintenance Test Support Tools

• Compilers (syntax checking)

• Source Code Static Analyzers

• Test Preparation Tools

• Test Execution Tools (Dynamic Analyzers)

• Test Evaluators

Implementation and Maintenance Test Support Tools

• Compilers (syntax checking)

• Source Code Static Analyzers

• Test Preparation Tools

• Test Execution Tools (Dynamic Analyzers)

• Test Evaluators

Source Code Static Analyzers

• Auditors (enforce style manuals)

• Complexity Measurers

• Cross Referencing Tools– e.g. to verify uses

• Size Measurers

• Structure Checkers

• Syntax and Semantics Analyzers

Implementation and Maintenance Test Support Tools

• Compilers (syntax checking)

• Source Code Static Analyzers

• Test Preparation Tools

• Test Execution Tools (Dynamic Analyzers)

• Test Evaluators

Test Preparation Tools

• Data Extractors

• Requirements-Based Test Case Generators

• Test Data Generators

• Test Planners

Implementation and Maintenance Test Support Tools

• Compilers (syntax checking)

• Source Code Static Analyzers

• Test Preparation Tools

• Test Execution Tools (Dynamic Analyzers)

• Test Evaluators

Test Execution Tools (Dynamic Analyzers) - Part 1

• Assertion Analyzers

• Capture-Replay Tools

• Coverage/Frequency Analyzers

• Debuggers

• Emulators

• Network Analyzers

• Performance/Timing Analyzers

Test Execution Tools (Dynamic Analyzers) - Part 2

• Run-Time Error Checkers

• Simulators

• Status Displayer/Session Documenters

• Test Execution Managers

• Validation Suites

Implementation and Maintenance Test Support Tools

• Compilers (syntax checking)

• Source Code Static Analyzers

• Test Preparation Tools

• Test Execution Tools (Dynamic Analyzers)

• Test Evaluators

Test Evaluators

• Comparators

• Data Reducers and Analyzers

• Defect/Change Trackers

Other sources for testing tool info

• comp.software.testing

• www.cs.queensu.ca/shepard

• www.soft.com

• www.sqe.com

• www.methods-tools.com

• article by Robert Poston: A Guided Tour of Software Testing Tools

Testing Tool Market (Ovum - www.ovum.com)

• Currently $450M, growing at 30% per year

• dominant players (with 60% of the total market) are:– Mercury Interactive– Rational– Compuware– Segue