the erik jonsson school of engineering and computer science software testing process control a novel...
TRANSCRIPT
The Erik Jonsson School of Engineering and Computer Science
Software Testing Process Control
A Novel ApproachJoão W. Cangussu
University of Texas at DallasDepartment of Computer Science
[email protected]/~cangussu
Association for Software Engineering Excellence
September 13, 2005
The Erik Jonsson School of Engineering and Computer Science
Motivation
➋ Can we achieve a level o control for the STP that is close to what is observed in other engineering disciplines?
❶ Current techniques for the Software Testing Process (STP) are mostly based on guidelines and semi-formal methods.
The Erik Jonsson School of Engineering and Computer Science
Goal
Increase the controllability and prediction capabilities of the Software Testing Process by
developing a control mechanism to correct for deviations in the
Testing Process
The Erik Jonsson School of Engineering and Computer Science
Testing Process Control Status
✔ Several recommended procedures are available. None is based on formal models. There is a lack of control algorithms.
✔ COCOMO: For cost and effort estimation; no feedback loop.
✔ Reliability models: For the estimation of software reliability; once again, no feedback loop.
So what do Level 5 companies/groups do?
➊ Often chaotic.
The Erik Jonsson School of Engineering and Computer Science
The Problem
Software TestProcess
inputparameters
observedquality (Qo)
expectedquality (Qe)
TestManager
error = Qe - Qo
ControlMechanism
set ofsolutions
The Erik Jonsson School of Engineering and Computer Science
System Testing
t- timecp1 cp2 cp3 cp4 cp5 cp6 cp7 cp8 cp9
where cpi = check point ir0
approximation
deadlinet0
num
ber
of r
emai
ning
def
ects
rf
schedule set bythe test manager
observed
Estimated
The Erik Jonsson School of Engineering and Computer Science
Control Approach
Actual STPwf+wf
+
wf+wf
+ STP State Model
robserved(t)
rexpected(t)
Initial Settings(wf,)
rerror(t)Controller
’
w’f
Test Manager
+
+
wf
The Erik Jonsson School of Engineering and Computer Science
Why a State Model?
➊ State Variable approach has been used to successfully model different types of system
✔ Predator/Prey Model (Voltera)
✔ Economic Model (Samuelson)
✔ Warfare Model (Lanchester)
➋ Enables the use of Control Theory techniques
✔ Feedback Control - Friedland, Luenberger
✔ Optimization - Chong, Zak
✔ System Identification - Ljung
The Erik Jonsson School of Engineering and Computer Science
Modeling
Deming states:
All models are wrong, some models are
useful.
The Erik Jonsson School of Engineering and Computer Science
Assumption I
The rate at which the velocity of the number of remaining errors changes is directly proportional to the net applied effort (en) and inversely proportional to the complexity of the program under test.
rseser cnc
n
The Erik Jonsson School of Engineering and Computer Science
Assumption II
rwse fcf )(
for an appropriate bc
c ss
)(
The effective test effort (ef) is proportional to the product of the applied work force and the number of remaining errors.
The Erik Jonsson School of Engineering and Computer Science
Assumption III
rer
1
for an appropriate constant .
The error reduction resistance (er) is proportional to the error reduction velocity ( r ) and is inversely proportional to the overall quality of the test phase (). er opposes r.
The Erik Jonsson School of Engineering and Computer Science
State Model
d
ccb
c
fF
sr
r
ss
wr
r
1
010
)1(
The Erik Jonsson School of Engineering and Computer Science
Analogy
Block
Dashpot
Rigid surface
External force
Xcurrent
Xequilibrium
X: Position
Number of remainingerrors
Spring Force
Effective Test Effort
Software
Mass of the blockSoftware
complexity
Quality of thetest process
Viscosity
Spring
To err isHuman.
The Erik Jonsson School of Engineering and Computer Science
Parameter Calibration
➋Good historic data help avoid initial over/under estimation
➊Models are approximations and need to be calibrated
➌Calibration algorithm considerably decreases the effect of initial over/under estimation
The Erik Jonsson School of Engineering and Computer Science
Model Parameters
➊ Parameters to Estimate
✔ sc (software complexity)✔ γ (quality of the testing process)
✔ R0
✔ ζ✔ ξ
➋ Automatic Parameter Calibration
The Erik Jonsson School of Engineering and Computer Science
Estimating sc
sc
1 2
3n
M1: KLOC
M2: CyclomaticComplexity
M3: Halstead Metric
Mn: InformationFlow Metric
n
ii
n
iiic Ms
1
1
1
The Erik Jonsson School of Engineering and Computer Science
Estimating γ
1 2
3n
Q1Q2
Q3Qn
n
ii
n
iii Q
1
1
1
Where Qi is a quality factor
The Erik Jonsson School of Engineering and Computer Science
Estimating the α weights
Quality Factor Qi Q1 Q2 … Qn
Q1: test plan adequacy
1 2 … 1/4
Q2: test team experience
1/2 1 … 3
… … … 1 …
Qn: testing tool adequacy/automation
4 1/3 … 1
The eigenvector associated with the largest eigenvalue of this matrix give us the appropriated weights
The Erik Jonsson School of Engineering and Computer Science
Estimating R0
λ1 λ2
λ1 λ2
Approximator
R0
EstimatorData from ongoing testing
R0
CorrectionR0
R0c
Turn key solution:
does not depend on historical or “unavailable” data from previous projects
does not depend on initial values ofInternal parameters
The Erik Jonsson School of Engineering and Computer Science
Estimating ξ and ζ
)ln(1
)ln(1
Theorem
Mapping
Spectral
:M of seigenvalue
:A of seigenvalue
M2
12
M2
M1
11
M1
2M1
21
12
11
Te
Te
AT
AT
M
AA
A
A
The Erik Jonsson School of Engineering and Computer Science
Parametric Control
),(
)()(
maxmax
max
qualityeffort
eTrtTr t
r -
num
ber
ofre
mai
ning
err
ors
t- time
R0
Rf
r(T)
r(T+t)
tt0
The Erik Jonsson School of Engineering and Computer Science
Computing Δwf and Δγ
teTrtTr max)()(
bc
f
c
cb
c
f
s
w
s
ss
wAI
12
1
ˆ
ˆ
ˆ
ˆ1
detdet
fwff wwandwhere ˆˆ
The Erik Jonsson School of Engineering and Computer Science
Results – Control and Time Prediction
The Erik Jonsson School of Engineering and Computer Science
Results – Control and Time Prediction
The Erik Jonsson School of Engineering and Computer Science
Results – Control and Time Prediction
The Erik Jonsson School of Engineering and Computer Science
Results – TEX78
The Erik Jonsson School of Engineering and Computer Science
Results – Transformer Project
The Erik Jonsson School of Engineering and Computer Science
Results – Feedback Application
The Erik Jonsson School of Engineering and Computer Science
Results – More Projects
The Erik Jonsson School of Engineering and Computer Science
Results – More Projects
21.2 weeks21 weeks
13.6 weeks
13.4 weeks76463518
21.4 weeks
21 weeks13.6
weeks13.4
weeks75859016
21.6 weeks21 weeks
13.6 weeks
13.4 weeks75853514
18 weeks21 weeks10 weeks13.4 weeks55745812
16 weeks21 weeks10 weeks13.4 weeks
5572819
EstimatedActualEstimate
d Actual
90% Defect Reduction
70% Defect ReductionEstimated
R0
Observ-ed
Defect #Week
The Erik Jonsson School of Engineering and Computer Science
Convergence of Completion Time Estimates
The Erik Jonsson School of Engineering and Computer Science
Convergence of Completion Time Estimates
68.5% required at most 11% of total time to converge
90% required at most 21% of total time to converge
The Erik Jonsson School of Engineering and Computer Science
Estimates for R0
The Erik Jonsson School of Engineering and Computer Science
Estimates for R0
The Erik Jonsson School of Engineering and Computer Science
Comparing R0 Estimates
The Erik Jonsson School of Engineering and Computer Science
Comparing R0 Estimates
The Erik Jonsson School of Engineering and Computer Science
Convergence of R0 Estimates
The Erik Jonsson School of Engineering and Computer Science
Convergence of R0 Estimates
The Erik Jonsson School of Engineering and Computer Science
Sensitivity Analysis
➊ Changes in the process’s parameters are more effective at early stages of the process
➋ Under certain conditions, late changes can make the process slow down instead of speed up (Brooks’s law).
➌ Improvements in the quality of the process is better than increases the size of the test team.
The Erik Jonsson School of Engineering and Computer Science
Conclusions
❶ The state model is reasonably accurate in modeling the behavior of the STP
➋ The parameter identification technique provides accurate results for model calibration and defect estimation
➌ Static and dynamic analysis has shown consistency of the model with the real world
The Erik Jonsson School of Engineering and Computer Science
Future Work
❶ Develop a modular hierarchy of models for the other phases of the SDP and interconnect them
➌ Develop a secure web-based tool to host the techniques
➋ Investigate alternative control techniques
✔ Stochastic control✔ Fuzzy control