software testing (introduction)enrico/mmis/testing_1_intro.pdf · testing (2) testing: is the...
TRANSCRIPT
Testing is .... Testing is important
A study conducted by NIST in 2002 reports that software bugs cost the U.S. economy $59.5 billion annually. More than a third of this cost could be avoided if better software testing was performed
In 1996, the European Space Agency's US$1 billion prototype Ariane 5 rocket was destroyed less than a minute after launch, due to a bug in the on-board guidance computer
Testing is difficult More difficult than coding ... Several aspects to consider Several approaches/techniques and tools
Industries need good testers ($$$$$$$) 2
3
RAMS (Relaibility, Availability, Maintanability and Safety)
Not too much terminology/definitions Too much differences/ambiguities Boring … A glossary (http://csqa.info/software_testing_glossary)
Not too much theory Ex. Test selection problem is undecidable
No managerial aspects Test planning, Test processes, Test documents …
It is a pragmatic (sub) course! Techniques, Tools Code and exercises …
Self-contained Prerequisites
a little of Java and C Eclipse?
4
Learning Objectives of RAMS Test Generation from Requirements (black box testing) Structural and Mutation Testing (white box testing) Unit testing (Junit)
Mock objects Test-first vs. test-last (Test driven development) Integration, acceptance testing Regression testing Testing of OO Systems (partially)
Tools/Frameworks: Junit (Unit testing) Fitnesse (Acceptance testing)? Clover/Emma (White box testing) Jumble (Mutation testing) Capture and replay tools?
5
Today agenda Terminology
Failures, faults, errors Testcase and testsuite Testing vs. Debugging Adequacy and Design for testability Test plan, Test Script, … Drivers and stubs
scaffolding
Types of testing White vs. Black box testing Unit, Integration, System, Regression Testing
Testing OO Software
Testing OO code more difficult than testing of procedural code There are some differences
Typical OO software characteristics that impact testing
State dependent behavior Encapsulation Inheritance Polymorphism and dynamic binding Abstract and generic classes Exception handling
7
Failures, Faults, Errors
Failure: it is an observable incorrect behavior or state of a given system It occurs at runtime
Fault: (commonly named “bug/defect”) it is a defect in a system Fault ≠> Failure
Error: it is the developer mistake that produce a fault Often caused by human activities such as the
typing or logic/conceptual errors
8
Code
1 program double ();
2 var x,y: integer;
3 begin
4 read(x);
5 y := x * x;
6 write(y)
7 end
Failure: executing x = 3 --> y =9 • correct output would be 6
Fault: The fault is in line 5 * is used instead of +
Error: The error may be: • a typing error
• * instead of + • a conceptual error
• meaning “double a number” not clear
Example “The following program should
double a number”
Fault
9
Testing (1)
10
Testing (2) Testing: is the process of executing a program
with the intent of finding errors Testing cannot guarantee the absence of faults but
improve the confidence of the SUT Formal methods (e.g., model checking) can be used to
statically verify software properties, this is not testing!
SUT The output is correct?
I1, I2, I3, …, In, …
Expected results = ? Obtained results “Inputs”
11
Excuses for not testing
It takes too much time to write the tests It decrease productivity It is difficult It is useless because testing can not prove the
correctness It takes to long to run the tests It is not my job to test my code I’m being paid to write code, not to write tests …
12
Testcase and Testsuite
Several definitions of Testcase … A set of inputs, execution preconditions, and
expected outcomes developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement [IEEE, do178b]
A Test case is a sequence of steps to test the correct behaviour of a functionality/feature of an application. A test case should have an input description and an expected behaviour
Also called test data
Testsuite = collection of Testcases
13
Examples of testcase Test Input Description:
1. Login to <Abc page> as administrator 2. Go to Reports page 3. Click on the ‘Schedule reports' button 4. Add reports 5. Update
Expected Results: The report schedule should get added to the report schedule Table
Test case for sort:
• Test data: <“A” 12 -29 32 > • Expected output: -29 12 32
ascending
14
Program Behaviour
Can be specified in several ways: Requirements
plain natural language Formal mathematical
specification Language Z
State diagram A state diagram specifies
program states and how the program changes its state on an input sequence
…
15
Oracle
• The entity that performs the task of checking the correctness of the observed behavior is known as an oracle
• Requirements, specifications, developer knowledge, user needs play the role of Oracle
17
Test Plan • A test cycle is often guided by a test plan
• Often a Test Plan is a list of steps to follow
• Example: • The sort program is to be tested to meet the
requirements as follows: • Execute sort on at least two input sequences, one with “A”
and the other with “D” • Execute the program on an empty input sequence • Test the program for robustness against erroneous inputs
such as “R” • All failures of the test program should be recorded in a
suitable file using the Company Failure Report Form
Test Plan: Describe scope, approach, resources, test
schedule, items to be tested, deliverables, responsibilities,
approvals needed
19
Debugging
Testing discover the faults Debugging: the process
used to determine the cause of an error and to remove it
Two phases: fault localization/location
ex. finding the file that contain the fault
fault removal
Testing +
Debugging
21
White vs. Black box testing A white box testing is based upon
explicit knowledge of the SUT and its structure Called also structural testing Statement coverage testing is an example
The aim is to create enough testcases to ensure every statement is executed at least once
A black box testing approach will device testcases without any knowledge of the SUT or any aspect of its structure Called also functional testing Usually testcases are generated starting from
requirements/specifications
Inputs Outputs
Inputs Outputs
22
Which is better?
They are complementary … Even if an empirical study shown that:
With expert testers functional testing detected more faults than did structural testing
V. Basili and R.W. Selby IEEE Transactions on Software Engineering Volume 13 , Issue 12 (December 1987) Pages: 1278-1296 Year of Publication: 1987
23
Structural testing example
Statement coverage Example
With two input data we obtain statement coverage
X=7 => a=8 and b=6 Statements [1, 2, 3, 4, 5, 6, 8]
X=22 => a=23 and b=21 Statements [1, 2, 3, 4, 6, 7, 8]
1 scanf("%d", &x); 2 a = x + 1; 3 b = x - 1; 4 if (a < 10) 5 x++; 6 if (b > 20) 7 x--; 8 printf("%d\n", x);
24
Limits in structural testing
Often the number of testcases is huge impractical
It fail to reveal the “missing feature errors” They can be discovered
only considering the specifications
Usually, It is used only for Unit testing by developers
Path coverage
25
Functional testing Example
Assuming that an item code must be in the range 99...999 and quantity in the range 1..100
Equivalence classes for code: E1: Values less than 99 E2: Values in the range E3: Values greater than 999
Equivalence classes for qty: E4: Values less than 1 E5: Values in the range E6: Values greater than 100
Code = 127 Quantity = 3
Price 87 euro
FindPrice Example
Input: (E1, E4), (E1, E5), ...
26
Adequacy Adequacy = level of confidence of a testsuite
applied to a System It answer to the question
How much is good that testsuite? Different criteria
Based on requirements Based on structure
Ex. Coverage (65% vs. 78%) Based on number of Killed mutants
Mutation Testing (we will see it …) …
27
Design for testability (1)
Testability = ease of performing testing Key factors: observability and controllability
Good development practices simplify testing GUI – Business logic – data logic must be separated Interface between the modules must be well-defined
Badly designed systems makes testing difficult and limits the testing options
Bad practice: before developing and after testing
28
Design for testability (2)
“Well architected application” “Badly designed system”
Test Tool
29
Test Script
A test script is a set of instructions that will be performed on the SUT to test that the system functions as expected
They can executed manually or automatically The major advantages of Automated testing is that tests:
may be executed continuously without the need for a human intervention
are easily repeatable Test scripts can be written using:
a special test tool Capture/reply tools
a well-known programming language Simple way: using main() to test
a testing framework Ex. Junit (we will see it …)
Calc.new(); Calc.add(2); Calc.add(3); Calc.subtract(1); Assert(Calc.total==4)
30
Using main to test (1) public class Stack { … public static void main (String[] args) {
Stack aStack = new Stack();� if (!aStack.isEmpty()) {� System.out.println (“Stack should be empty!”);
} aStack.push(10);�
aStack.push(-4);� System.out.println(“Last element:“ + aStack.pop());� System.out.println(“First element: “ + aStack.pop());
if (!aStack.isEmpty()) {� System.out.println (“Stack should be empty!”);
} } }
-4
10
isempty() pop() push(…) size()
Output
31
Using main to test (2)
Void testSize(int expected_value) { if (this.size()==expected_value) System.out.println(“OK”); else System.out.println(“Not Ok”); }
Test method
public class Stack { … public static void main (String[] args) {� Stack aStack = new Stack();� aStack.testSize(0),
aStack.push(10);� aStack.push(-4);� aStack.testSize(2);
aStack.pop(); aStack.testSize(1); aStack.pop();
aStack.testSize(0); } }
isempty() pop() push(…) size() testSize(…) A test method is added
32
V&V Verification checks that the software is correct
with respect to the requirements documentation
Validation checks that the software is what the customer requires/want, i.e., that it performs the functions that the customer needs
Usually Verification precede Validation Requirements Customer needs Code
Verification Validation
33
Types of testing
C1: Source of test generation • ex. Requirements vs. code
C2: Lifecycle phase in which testing takes place • ex. Coding vs. maintenance
C3: Goal of a specific testing activity • ex. Performance vs. security
C4: Characteristics of the artifact under test • ex. DB vs. Web services
One possible classification is based on the following four classifiers:
34
Test Generation
Any form of test generation uses a source document Both manual that
automatic
35
C1: Source of test generation
36
C2: Lifecycle phase in which testing takes place
37
Unit testing A unit is “a software piece which is separately
testable” Traditional (Procedural): units are module, function,
procedure, etc Object oriented (encapsulated data+operatons): units
are Classes Unit testing = each module is individually tested to
ensure that it meets its specification/requirements Recommendation: always isolate units from the
rest other modules system calls hardware dependencies
Scaffolding (OO code)
Classes to be tested
Tool example: JUnit
Tool example: MockMaker
Problem: Most classes are not complete programs - Additional code must be added to execute them
39
Integration testing (1)
A set of modules are tested together This requires careful testing of the interfaces
between the modules, as they communicate and exchange information It occurs after unit testing and before system testing
Usually “interfaces” are a common source of errors Especially if the modules are written by different
programmers Ex. Parameters may not be in the correct order Ex. Preconditions can be respected in a module and not in the
other
Often “additional” software must be developed by the test analyst Stubs and drivers
40
Stub
A Stub is a piece of code written to replace the modules or procedures that the program under test depends on and calls, so that it can be executed An imitation of a unit Used in place of the real unit to
facilitate testing ~mock objects in OO
programming
A B
SUT
A’
“my reply is always ok”
41
Testing methods within a class
using STUBs
public class WaterTankController { public static void main (String[] args) { Sensor sensorObject = new Sensor (); Alarm alarmObject = new Alarm();
int waterLevel = sensorObject.getReading (); checkReading (waterLevel); } public static void checkReading (int w) { if (w < 0 || w > 100) System.out.println (“ Invalid reading ”); else if ((w >= 0 && w <= 20) || (w >= 80 && w <= 100)){
System.out.println (“Water level exceeded the limit – EMERGENCY”); alarmObject.raiseAlarm (waterLevel); } else System.out.println (“ Normal range ”); } }
42
Testing Methods … STUBs (continued)
public class Sensor {
…
public int getReading () {return 10;}
}
public class Alarm {
…
public void raiseAlarm (int n) {
System.out.println (“ Alarm raises ”);
}
}
Stub
Stub
43
Driver
Driver is a “calling program”
It is used to establish the connection between the main module and sub modules when the Main module is under construction Used also in unit/integration
testing
A B
Driver
C
Main
44
Integration Testing (2)
Approaches (only for procedural code) bottom-up top-down mixed (up to big-bang)
top-down: whenever using non-tested components, stubs are required
bottom-up: when using non-tested components, drivers are required
45
Integration testing (3)
Program structure Modules to test
Top down Bottom up Mixed
46
Big Bang Integration
Non-incremental strategy Unit test each module in
isolation Integrate as a whole
test main
test A
test E
test C
test D
test B
test F
test main, A, B, C
D, E, F
main
A B C
D E F
47
System testing System testing of software or hardware is
conducted on a complete, integrated system to evaluate the system's compliance with its specified requirements
System testing falls within the scope of black box testing
Several aspects have to be considered: GUI software testing Usability testing Performance testing Load testing Volume testing Stress testing Security testing …
48
Regression testing Selective re-testing of a system or component to
verify that modifications have not caused unintended effects Ripple effects
Can be conducted at each of the test levels unit, integration, system
Version X Version X+1
Modifications • find affected testcases (red) • change affected testcases (red) • execute them • re-execute the others (ex. T2)
T1 T2 T3 Tn
…
T1 T2 T3 Tn
…
Testcases to
modify
Changed
System testing “interface beween components”
Phases
51
C3: Goal of specific testing activity
52
C4: Characteristics of the artifact under test
Software = software testing Client-server = client-server testing OO software = OO software testing Web service = Web service testing Real time software = Real time Soft. testing Database system = Transaction flow testing …
53
What we don’t see … Design and code Walkthroughs and Inspections
Also called “static testing”
Performance testing Load testing, Volume testing, Stress testing, …
Security testing
Model based testing Automatic generation of testcases Web application testing Testing of distributed systems GUI testing / System testing Quality metrics …
54
References (used to prepare these slides)
Slides of the book “Foundations of software testing” by Aditya P. Mathur (chapter 1) http://www.cs.purdue.edu/homes/apm/
foundationsBook/InstructorSlides.html Slides prepared by Egidio Astesiano
Software engineering course II Proceedings “scuola estiva: Software testing
Metodi e tecniche”, 7-11 Giugno 1993, Riva dei Tessali
Wikipedia …