integration testing of components guided by incremental state machine learning
DESCRIPTION
Keqin Li, CNRS LSR-IMAG Roland Groz, INPG LSR-IMAG Muzammil Shahbaz, France Telecom August 2006. Integration Testing of Components Guided by Incremental State Machine Learning. Content. Motivation Background Framework Scenario Testing Unit Testing Integration Testing Conclusion. - PowerPoint PPT PresentationTRANSCRIPT
Integration Testing of Components Guided by Incremental State Machine Learning
Keqin Li, CNRS LSR-IMAGRoland Groz, INPG LSR-IMAGMuzammil Shahbaz, France
TelecomAugust 2006
2
Content Motivation Background Framework Scenario Testing Unit Testing Integration Testing Conclusion
3
Motivation
Software development is more and more based on component integration
Unknown Internal structures and insufficient documents
Only partial and informal information is available
Provide a required system assembling COTS of which he has a limited knowledge
Unit testing and integration testing
Have to rely on intuitions and little support is available
Component: Integrator:
4
Our Approach Infer formal models of components
from behaviours observed during tests
Perform integration testing based on the formal models
Iteratively improve the formal models
5
Background - Angluin’s Algorithm [1987] Infer formal model (conjecture)
from observed behavior (observation table)
For deterministic finite-state acceptor
Teacher answers Membership query Equivalence query
6
Process
EmptyObservation
TableMembership
QueryClosed& Cons.
No
EquivalenceQuery
YesYes? End
Add strings toObservation Table
Yes
Teacher
String AcceptedOr Not
Conjecture Yes OrCounterexample
No
7
Assumptions Black boxes with known interfaces Only external interfaces are
controllable Slow environment Test scenarios
8
Framework
UnitTesting
Models ofComponents
IntegrationTesting
ScenarioTesting
InitialObservation
Tables
ConsultDomainExpert
9
I. T.C.E.
Models
M1(1)
M2(1)
…
Mn(1)
U. T.
U. T.
U. T.
I. T.
M1(2)
M2(2)
…
Mn(2)
U. T.
U. T.
U. T.
C.E.I. T.
M1(3)
M2(3)
…
Mn(3)
U. T.
U. T.
U. T.
U. T.:UnitTesting
I. T.:IntegrationTesting
C.E.:Counter-example
10
Scenario Testing A test scenario is an input/output
sequence containing external inputs and outputs
A test case is constructed for each test scenario
Replace problematic component Record observed behaviour of each
component in its observation table
11
Unit Testing Each Component is tested
individually For each component
Complete the initial observation table Using extended learning algorithm to
construct closed and consistent observation table
Make conjectured model
12
Integration Testing
TestGeneration
Discre-pency?
Yes
CoverageCriterionSatisfied?
No Yes
No
UnitTesting
TestExecution
13
Test Generation based on Composition of FSMs M N [Petrenko 98] An algorithm to compute M N
[Lima 97] Assign colour to “global”
transitions An algorithm to generate tests
covering colours [Kim 02]
14
References [Petrenko 98] A. Petrenko, N. Yevtushenko,
“Solving Asynchronous Equations,” Proceedings of FORTE’98, 1998
[Lima 97] L. Lima Jr., A. Cavalli, “A Pragmatic Approach to Generating Test Sequences for Embedded Systems,” Proceedings of IWTCS’97, 1997
[Kim 02] M. Kim, C. Besse, A. Cavalli, and F. Zaïdi, “Two Methods for Interoperability Tests Generation: An Application to the TCP/IP Protocol,” Proceedings of TestCom 2002, 2002.
15
Dealing with Discrepancy Observe the behaviour of each
component, and compare with its model
Discrepancy Unexpected output sequence New input symbol
16
Refine (relearn) model Counterexample or new input
symbol has been found Add corresponding strings into
observation table Perform learning algorithm to
make the observation table closed and consistent again
Make conjectured model
17
Consulting Domain Expert Models of components and test
results Domain expert
Identify unexpected behaviours Propose additional test scenarios
Iteration
18
Results of Integration Testing For each component, we have a
model The joint behaviour of these
components has been systematically tested
Faults might be revealed during testing
19
Conclusion Combines machine learning and
model-based testing for the integration of components
Reuses Angluin’s algorithm Iteratively refines models
20
The Models Just a “by-product” Our goal is not to build a complete
model of the system Just used as basis for the testing
process to: Generate integration tests of
components Assess the coverage, hence the
quality of the integration testing process
21
The Next Steps Move to Parameterized FSM
presentation by Muzammil Shahbaz Identify the type of approximation
needed Experiment and case study Revisit model-based test
generation strategies
Thank you
Backup
24
Observation Table
00bbb10bba00ba00ab01aa01bb00b10a01εaε
S
S.A
E
25
Closed
tSA, sS, s.t., row(t)=row(s)
00bbb10bba00ba00ab01aa01bb00b10a01εaε
S
SA
E
26
Consistent
s1S, s2S, row(s1)=row(s2),
Then, aA,
row(s1a)=row(s2a)
00bbb10bba00ba00ab01aa01bb00b10a01εaε
S
SA
E
27
Conjecture
00bbb10bba00ba00ab01aa01bb00b10a01εaε
S
SA
E
q0
q1 q2
a b
ab
a
b
28
Observation Table
xybbyxbaxyabyxaaxybyxaxxεba
row(ε)
row(b)row(a)
a/x b/x
b/y
a/ya/x b/x
29
Example – Component Hotel Reservation
q0
q2
q4
q3
q1
q5
q6
q7
q9
q8 q10
q11
N/HLH1/TL
Dbl/F
Sgl/F
T/PD1
T/PS1No/HL
No/TL
No/TL
No/F
No/F
OK/CD1
OK/CS1
H2/TL
Dbl/F
Sgl/F
No/TL
No/TL
T/PD2
T/PS2
No/F
No/FOK/CD2
OK/CS2
30
Initial Observation Table
ΩΩΩΩΩΩΩHLNoΩΩΩΩΩΩΩHLOKΩΩΩΩΩΩΩHLTΩΩΩΩΩΩΩHLSglΩΩΩΩΩΩΩHLDblΩΩΩΩΩΩΩHLH2ΩΩΩΩΩΩΩHLH1ΩΩΩΩΩTLTLΩNΩΩΩΩΩΩΩHLεNoOKTSglDblH2H1N
31
Add N to S
………………………ΩΩΩΩΩTLTLΩN/No
ΩΩΩΩΩΩΩHLNo
ΩΩΩΩΩTLTLΩN/DblHLΩΩFFΩΩΩN/H2HLΩΩFFΩΩΩN/H1ΩΩΩΩΩTLTLΩN/N
………………………ΩΩΩΩΩΩΩHLH1ΩΩΩΩΩTLTLΩNΩΩΩΩΩΩΩHLεNoOKTSglDblH2H1N
32
Conjecture
row(ε) row(N-H1)
row(N-H1-S)
row(N-H1-D)
row(N)
row(N-H1-D-T)
row(N-H1-S-T)
N/HL H1/TLD/F
S/F
T/PD1
T/PS1
No/HL
No/TL
No/TL
No/F
No/F
OK/CD1
OK/CS1
H2/TL
33
Example – Component User Interface
q0
UI_Name/N
q1HL/UO_HL
UI_H1/H1UI_H2/H2
q2TL/UO_TL
UI_Dbl/DblUI_Sgl/SglUI_No/No
q3F/UO_F
UI_T/TUI_No/No
q4
PD1/UO_PD1PS1/UO_PS1PD2/UO_PD2PS2/UO_PS2
UI_OK/OKUI_No/No
F/UO_FTL/UO_TLHL/UO_HL
CD1/UO_CD1CS1/UO_CS1CD2/UO_CD2CS2/UO_CS2
34
Integration Testing Environment
User Interface(M)
Hotel Reservation(N)
oMN
oNM
cM
oM
35
Counterexample UI_Name/UI_H2/UI_Dbl/UI_T The real input sequence of Hotel
Reservation is: N/H2/Dbl/T
The corresponding output sequence is: HL/TL/F/PD2
According to the model, the output should be: HL/TL/F/PD1
36
Comparing with Angluin’s Algorithm
InitialObservation
TableUnit Testing Closed
& Cons.
No
IntegrationTesting
Yes Counter-example? End
Add strings toObservation Table
Yes
No
ScenarioTesting Component Component
AComponent
B
Test Output Test Output