illinois institute of technology
DESCRIPTION
Illinois Institute of Technology. CS487 Software Engineering Software Testing Techniques Mr. David A. Lash. Why Test? . Testing is the filter to catch bugs before the are “discovered” by customer Every time the program is run by a customer, it generates a “test-case”. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/1.jpg)
CS48715-1/51
Illinois Institute of Technology
CS487
Software Engineering
Software Testing Techniques
Mr. David A. Lash
![Page 2: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/2.jpg)
CS48715-2/51
Why Test?
Testing is the filter to catch bugs before the are “discovered” by customer
– Every time the program is run by a customer, it generates a “test-case”.
Software development is a human activity with huge potential for errors
Testing before release helps assure quality and save money
![Page 3: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/3.jpg)
CS48715-3/51
Testing Steps
Start at testing each individual new component and work way out
– Unit test – Integration test– High Order Test– Customer Acceptance testing
Different testing techniques are used at different times in the process
A test specification is often used as a testing road-map that is generally reviewed by a team.
![Page 4: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/4.jpg)
CS48715-4/51
Purpose of testing is ... – To find errors
A good test ...– Trys to discover undetected errors– Is successful when errors are found
To design good test cases must understand the system and the software development process.
Zero Defects is not possible. – Always a condition or usage that can lead to an
incorrect behavior.– Done testing when continued testing is no longer
economical.
Testing Objectives
![Page 5: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/5.jpg)
CS48715-5/51
Tester VS Developer
Developer Tester
Constructive Process Destructive Process
Paid to get code inproduction
Paid to find errors
Often focused on theirdevelopment piece
Often focused on theoverall sub-system/system
Personal involvementin development canbias viewpoint
Viewpoint is customeror overall systemhealth
It is critical that developers and testers work together as a team
![Page 6: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/6.jpg)
CS48715-6/51
Software Quality Assurance Modes
Verification– “Are we building the product right?”– Does product meet requirements during this particular
phase– Can and should be done as part of implementation.
Validation– “Are we building the right product?”– Evaluating software at end of software development
process VS requirements– 3rd party is generally most effective validators
(developer ego can interfere with judgment). Formal Reviews Quality and Configuration audits
![Page 7: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/7.jpg)
CS48715-7/51
Testing Goals
Show that the program is correct.– Zero Defect Software is not possible.
There is always some condition that you are not aware of that can cause a incorrect behavior.
– Microsoft does test. Their testing covers “main-line” systems. There are several million possible
hardware configurations. It is the integrators responsibility to ensure
that the system works with Microsoft Software.
![Page 8: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/8.jpg)
CS48715-8/51
Testing Goals - Reality
To Have “confidence” in the software system. To Find all major defects
– You must first define major.– Defect scale
To find all major defects with given resources– Number of testers– Amount of time.
Design a set of test cases that have a high probability of finding defects.
![Page 9: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/9.jpg)
CS48715-9/51
Testing Case Design
Test cases should be designed to have the highest likelihood of finding problems
Can test by either– Black-box or using the specifications of what the
software should do Tests are derived from the I/O specification. Used in most functional tests. A.K.A. data-driven, input/output-driven.
– White-Box - testing internal paths and working of the software
Examine internal program structure and derive tests from an examination of the program’s logic.
Used to develop test cases for unit and integration testing
A.K.A. Glass-box, Logic-driven, Structural.
![Page 10: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/10.jpg)
CS48715-10/51
White-Box Strategies
Statement– Requires each statement of the program to be
executed at least once. Branch
– Requires each branch to be traversed at least once.
Multi-condition– Requires each condition in a branch be
evaluated.
![Page 11: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/11.jpg)
CS48715-11/51
White-Box Strategies
Basis Path– Execute all control flow paths through the
code. Based on Graph Theory. Thomas McCabe’s Cycolmatic Complexity: V(g) : #edges - #nodes + 2
Data Flow– Selects test data based on the locations of
definition and the use of variables.
And a number of others
![Page 12: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/12.jpg)
CS48715-12/51
Statement Coverage
The criterion is to require every statement in
the program to be executed at least once
Weakest of the white-box tests.
Specified by the F.D.A. as the minimum level of
testing.
![Page 13: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/13.jpg)
CS48715-13/51
Statement Coverage
Example:void example(int a, int b, float *x){
1 if ((a>1) && (b==0))2 x /= a;3 if ((a==2) || (x > 1)4 x++;
} Test case(s)1. a=2, b=0, x=3
![Page 14: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/14.jpg)
CS48715-14/51
Statement Coverage
Test Case
a=2, b=0 & x=3
Coverage
– acbed
What happens with data
that takes:
– abed
– abd
a > 1&&
b==0
x /= a;
a
b
c
d
e
Yes
a==2||
x > 1
No
x++
Yes
No
![Page 15: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/15.jpg)
CS48715-15/51
Branch Coverage
This criterion states that one must write enough
test cases such that each decision has a true
and false outcome at least once.
A.K.A. Decision coverage
More comprehensive than statement coverage.
![Page 16: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/16.jpg)
CS48715-16/51
Branch Coverage
Example:
void example(int a, int b, float *x)
{
1 if ((a>1) && (b==0))
2 x /= a;
3 if ((a==2) || (x > 1)
4 x++;
} Test case(s)
1. a=2, b=0, x=3
2. a=3, b=1, x=1
![Page 17: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/17.jpg)
CS48715-17/51
Branch Coverage
Test Case
1. a=2, b=0 & x=3
2. a=3, b=1 & x=1
Coverage
1. ace
2. abd
What happens with data
that takes:
– abe, or
– acd
a > 1&&
b==0
x /= a;
a
b
c
d
e
Yes
a==2||
x > 1
No
x++
Yes
No
![Page 18: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/18.jpg)
CS48715-18/51
Multiple-condition Coverage
This criterion requires one to write sufficient test cases such that all possible combinations of condition outcomes in each decision, and all points of entry are invoked at least once.
More comprehensive than branch coverage First step is to identify the test conditions
– ifs, whiles, for– reduce to simple predicates
![Page 19: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/19.jpg)
CS48715-19/51
Multiple-condition Coverage
Example:void example(int a, int b, float *x){
1 if ((a>1) && (b==0))2 x /= a;3 if ((a==2) || (x > 1)4 x++;
} Test Conditions
a>1, b=0; a>1, b!=0; a<=1, b=0; a<=1, b!=0;a==2, x > 1; a!=2, x>1; a==2, x<=1; a!=2, x<=1.
![Page 20: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/20.jpg)
CS48715-20/51
Multiple-condition Coverage
Test Conditions
1. a>1, b=0 5. a=2, x >1
2. a>1, b!=0 6. a=2, x<=1
3. a<=1, b=0 7. a!=2, x>1
4. a<=1,b!=0 8. a!=2, x<=1
Test Cases
1. a=2, b=0 & x=4 (1,5)
2. a=2, b=1 & x=1 (2,6)
3. a=1, b=0 & x=2 (3,7)
4. a=1, b=1 & x=1 (4,8)
Coverage
– all
a > 1
x /= a ;
h
j
i
l
Yes
a==2
No
x++
Yes
No
b==0 Yes
x > 1 Yes
No
No
k
m
n
![Page 21: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/21.jpg)
CS48715-21/51
Basis Path
Execute all independent flow paths through the code. Based on a flow graph.
– An independent flow path is on that introduces at least 1 new set of statements or conditions
– Must move along at least 1 new edge on flow graph
– Flow graph shows the logical control flow using following notation:
Sequence Ifwhile
until
![Page 22: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/22.jpg)
CS48715-22/51
Control Flow Example
1
2
3 4
5
6
7 8
9 10
![Page 23: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/23.jpg)
CS48715-23/51
Corresponding Flow Graph
1
6
2, 3
6
6
6
6
6
6
R2
R1
R4
R3
Edges (11)Nodes (9)
Regions (4)
V(G) = E - N +2
11 - 9 + 2 = 4
V(g) = RegionsV(g) = 4
V(g) concerned with Uniquepaths
1. Path 1-112. Path 1-2-3-4-5-10-1-11
3. Path: 1-2-3-6-8-9-10-1-114. Path: 1-2-3-6-7-9-10-1-11
![Page 24: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/24.jpg)
CS48715-24/51
Number of Independent Paths
1
6
2, 3
6
6
6
6
6
6
R2
R1
R4
R3
Edges (11)
Nodes (9)
Regions (4)
V(G) = E - N + 211 - 9 + 2 = 4
V(g) = RegionsV(g) = 4
V(g) concerned with Unique paths1. Path 1-11
2. Path 1-2-3-4-5-10-1-113. Path: 1-2-3-6-8-9-10-1-114. Path: 1-2-3-6-7-9-10-1-11
![Page 25: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/25.jpg)
CS48715-25/51
Another Examplei=1;
total.input =total.valid = 0;
sum = 0;
value[i] <> -999
total.input <100
total.input ++;
value[i] >=min &&
value[i] <=max
sum=sum+value[i];
i++;
Enddo
total.valid >0
aver = sum/total.valid;
aver=-999
no
No
Yes
Ye
s
No -
Done
1.
4.
3.
2.
5.
6.
7.
8.
9.
10.
11. 12.
13.
![Page 26: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/26.jpg)
CS48715-26/51
Corresponding Flow Graph
1
2
3
4
6
5
78
9
10
1112
13
![Page 27: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/27.jpg)
CS48715-27/51
Corresponding Flow Graph1
2
3
4
6
5
78
9
10
1112
13
i=1;total.input =
total.valid = 0;sum = 0;
value[i] <> -999total.input < 100
total.input ++;
value[i] >= min&&
value[i] <= max
sum=sum+value[i];
i++;
Enddo
total.valid > 0
aver = sum/total.valid;
aver=-999
no
No
Yes
Yes
No -
Done
1.
4.
3.
2.
5.
6.
7.
8.
9.
10.
11. 12.
13.
![Page 28: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/28.jpg)
CS48715-28/51
Number of Paths
1
2
3
4
6
5
78
9
10
1112
13
V(g) = E - N + 217-13 + 2 = 6
R = 6
![Page 29: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/29.jpg)
CS48715-29/51
Black-Box Testing
Focuses on functional requirements of the
software without regard to the internal structure.
A.K.A. data-driven, input/output-driven or
behavior testing
Used in most system level testing– Functional,
– Performance, etc.
Tests set up to exercise full functional
requirements of system
![Page 30: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/30.jpg)
CS48715-30/51
Black Box Testing Find Errors in ...
Incorrect or missing functions (compare to white box)
Interface errors Errors in External Data structures Behavior performance problems (Combinations
of input make it behave poorly). Initialization and Termination errors (Sensitive
to certain inputs (e.g., performance) Blackbox done much later in process than
white box.
![Page 31: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/31.jpg)
CS48715-31/51
Black-box Strategies
Exhaustive input testing– A test strategy that uses every possible input
condition as a test case.– Ideal
Random– Test cases are created from a pseudo random
generator.– Broad spectrum. Not focused.– Hard to determine the result of the test.
![Page 32: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/32.jpg)
CS48715-32/51
Black-box Strategies
Equivalence Partitioning– A black-box testing method that divides the
input domain of a program into classes of data which test cases can be derived.
Boundary Value Analysis– A test case design technique that
complements equivalence partitioning, by selecting test cases at the “edges” of the class.
And others.
![Page 33: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/33.jpg)
CS48715-33/51
Equivalence Partitioning
Divides the input domain of a program into classes of data which test cases can be derived.
– 1 test case uncovers classes of errors Helps reduce the number of inputs What are the properties of a well-selected test
cases:– It reduces, by more than one, the number of
test case that must be developed.– It covers a large set of other possible test
cases.
![Page 34: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/34.jpg)
CS48715-34/51
Identifying Equivalence Classes
Take each input condition (a sentence or phrase in the specification) partition or divide it into 2 or more classes.
Class– Valid equivalence classes– Invalid equivalence classes
![Page 35: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/35.jpg)
CS48715-35/51
Rules for Equivalence Classes
Range - If an input condition specifies a range
(i.e. n is an integer from 1 to 1000).– 1 valid (1< n < 1000)
– 2 invalid (n < 1 and > 1000)
Specified Value - A black-box testing method
that If an input condition specifies a specific
value ( i.e. 6 character string) identify:– 1 valid (6 character string)
– 2 invalid (5 character string, 7 char string)
![Page 36: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/36.jpg)
CS48715-36/51
Rules for Equivalence Classes
Value Set - If the input specifies a set of valid values, define:
– 1 valid condition within the set.– 1 invalid condition outside the set.
![Page 37: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/37.jpg)
CS48715-37/51
Rules for Equivalence Classes
Boolean - If an input condition specifies a “must be” situation (e.g. “first character alpha”) then identify:
– 1 valid (first character alpha).– 1 invalid (first character not alpha).
If there is any reason to believe that elements in an equivalence class are not handled in an identical manner by the program, split the equivalence class into smaller equivalence classes.
![Page 38: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/38.jpg)
CS48715-38/51
Equivalence Partition Example
Area Code Blank or 3 Character Number
Prefix 3 Digit not begin 0 or 1
Suffix 4 Digit #
Password 6 digit alpha numeric string (notrequired)
Command Things like check, deposit, pay, …
![Page 39: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/39.jpg)
CS48715-39/51
Equivalence Partition ExampleArea Code 1. Area (Boolean- there or not)
2. Range Values between 200-9991. 1 valid, 1 not2. 1 valid, 2 not
Prefix 1. Range > 200 < 999 1. 1 Valid 2 not
Password 1. Boolean (There or not)2. Value (6 characters)
1. 1 valid 1 not2. 1 valid 1 not
Command TSet of valid commands 1. 1 Valid 1 not
![Page 40: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/40.jpg)
CS48715-40/51
Boundary Value Analysis
Experience shows that test cases exploring boundary conditions have a high payoff.
– E.g., Most program errors occur in loop control.
Different from equivalence partitioning:– Rather than any element in class, BVA selects
tests at edge of the class.– In addition to input condition, test cases can be
derived for output conditions.
But similar to Equivalence partitioning ...
![Page 41: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/41.jpg)
CS48715-41/51
Guideline for Boundary-Value Analysis
If an input condition specifies a range of values, write test cases for the ends of the range, and invalid-input test cases for situations just beyond the ends.
– If a domain of an input is -1.0 to 1.0 write test cases for the situation -1.01 to 1.01.
– Or in general, if bounded by a and b write test cases just above and below
![Page 42: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/42.jpg)
CS48715-42/51
Guideline for Boundary-Value Analysis
If an input condition specifies a number of values, write test cases for the minimum and maximum number of values and one beneath and beyond these values.
– For example an input file can contain 1-255 records, write test cases for 0, 1, 255 and 256
![Page 43: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/43.jpg)
CS48715-43/51
Guideline for Boundary-Value Analysis
Apply the preceding rules to the output.– For example, if output is a output report, then
create an output report with maximum and minimum allowable table entries.
Apply rules to internal data structures ...– If use an array that has 1-100 elements max
then set up test cases for 0, 1, 100, 101 elements.
Look for other applications for BVA
![Page 44: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/44.jpg)
CS48715-44/51
Test Case Grid
ID
Condition TC 1 TC 2 TC 3
1 1< item count < 999 X
2 item count < 1 X
3 item count > 999 X
4 .5 < item weight < 1 X
![Page 45: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/45.jpg)
CS48715-45/51
Test Case Grid
Equivalence Class case and Boundary-Value analysis cases can be shown on the same table.
– Separate sections for Equivalence Class cases and Boundary-Value analysis.
– Equivalence Class cases first
![Page 46: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/46.jpg)
CS48715-46/51
Test Case Documentation
Minimum information for a test case– Identifier– Input data– Expected output data
Recommended to add the condition being tested (hypothesis).
Format of test case document changes depending on what is being tested.
Always include design worksheets.
![Page 47: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/47.jpg)
CS48715-47/51
Simple Test Case Format
Id Condition Input Data Expected
![Page 48: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/48.jpg)
CS48715-48/51
Test Case Formats
Testing worksheet– Test Case
Identifier (serial number) Condition (narrative or predicate) Input (Stimuli data or action) Expected Output (Results)
– Test Results Actual Output (Results) Status (Pass/Fail)
![Page 49: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/49.jpg)
CS48715-49/51
PSP Test Case Format
Test Name/NumberTest ObjectiveTest Description
Test Conditions
Expected Results
Actual Results
![Page 50: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/50.jpg)
CS48715-50/51
ANSI/IEEE Test Case Outline
Test-case-specification Identifier– A unique identifier
Test Items– Identify and briefly describe the items and
features to be exercised by this case Input Specifications
– Specify each input required to execute the test case.
Output Specifications– Specify all of the outputs and features required
of the test items.
![Page 51: Illinois Institute of Technology](https://reader035.vdocuments.net/reader035/viewer/2022062719/56813280550346895d991be8/html5/thumbnails/51.jpg)
CS48715-51/51
ANSI/IEEE Test Case Outline
Environmental needs– Hardware– Software– Other
Special procedural requirements– Describe any special constraints on the test
procedures which execute this test case. Interfaces dependencies
– List the id’s of test cases which must be executed prior to this test case