augmenting field data for testing systems subject to incremental requirements changes
TRANSCRIPT
.lusoftware verification & validationVVS
Augmenting Field Datafor Testing Systems Subject to
Incremental Requirements ChangesDaniel Di Nardo, Fabrizio Pastore, Lionel Briand
SnT Centre, University of Luxembourg
Journal First - ACM Transactions on Software Engineering and Methodology
Volume 26 Issue 1, July 2017
Data Requirements Change
Changes affect Data Fields, Data Types, Constraints
Old Test Cases Cannot be ReusedField Data is Not Available
OriginalModel Instance Incomplete Updated
Model Instance
Data LoadingSlice-basedConstraint
Solving
AutomaticModel
TransformationData Writing
Valid UpdatedModel Instance
UpdatedDataField
Data
Data Model
OriginalModel Instance Incomplete Updated
Model Instance
Data LoadingSlice-basedConstraint
Solving
AutomaticModel
TransformationData Writing
Valid UpdatedModel Instance
UpdatedDataField
Data
Data Model
1
Transmission Vcdu
VcduHeader
Packet
PacketZone
ActiveZoneIdleZone
1 11..*1
1
versionNumber : IntspaceCraftId : IntvirtualChanId : Int
apidValue : ApidsequenceCount : IntdataLength : Int
PacketHeader
SarPacketHeadercoarseTime : IntfineTime : Int
GpsrPacketHeadercoarseTime : IntfineTime : Int
ConfigurationspaceCraftId : IntcheckCrc : BoolidleVcid : VcididleApid : Apid
VcduConfigvcid : VcidminPacketSize : IntmaxPacketSize : Intapids : Apid[]
11
1..*
1
prev 1..*
1
1
1
«DataType» Apid
value : Int
1
Transmission Vcdu
VcduHeader
Packet
PacketZone
ActiveZoneIdleZone
1 11..*1
1
versionNumber : IntspaceCraftId : IntvirtualChanId : Int
apidValue : ApidsequenceCount : IntdataLength : Int
PacketHeader
SarPacketHeadercoarseTime : IntfineTime : Int
GpsrPacketHeadercoarseTime : IntfineTime : Int
ConfigurationspaceCraftId : IntcheckCrc : BoolidleVcid : VcididleApid : Apid
VcduConfigvcid : VcidminPacketSize : IntmaxPacketSize : Intapids : Apid[]
11
1..*
1
prev
context VcduHeader inv: self.vcdu.trans.config.vcduConfigs.vcid
→exists(x | x = self.virtualChanId)
OCL constraints
1..*
1
1
1
«DataType» Apid
value : Int
1
Transmission Vcdu
VcduHeader
Packet
PacketZone
ActiveZoneIdleZone
1 11..*1
1..*
1
versionNumber : IntspaceCraftId : IntvirtualChanId : Int
apidValue : ApidsequenceCount : IntdataLength : Int
PacketHeader
SarPacketHeadercoarseTime : IntfineTime : Int
GpsrPacketHeadercoarseTime : IntfineTime : Int
ConfigurationspaceCraftId : IntcheckCrc : BoolidleVcid : VcididleApid : Apid
VcduConfigvcid : VcidminPacketSize : IntmaxPacketSize : Intapids : Apid[]
11
1..*
1
prev
context Packet inv: ( self.apidValue.value=1 and self.pHeader.oclIsTypeOf(SarPacketHeader) ) or …
1
1
1
context VcduHeader inv: self.vcdu.trans.config.vcduConfigs.vcid
→exists(x | x = self.virtualChanId)
OCL constraints
«DataType» Apid
value : Int
1
Transmission Vcdu
VcduHeader
Packet
PacketZone
ActiveZoneIdleZone
1 11..*1
1..*
1
versionNumber : IntspaceCraftId : IntvirtualChanId : Int
apidValue : ApidsequenceCount : IntdataLength : Int
PacketHeader
«Replacement» MsiPacketHeadercoarseTime : IntfineTime : IntopMode : Intstatus : Int
GpsrPacketHeadercoarseTime : IntfineTime : Int
prev
context Packet inv: ( self.apidValue.value=1 and self.pHeader.oclIsTypeOf(SarPacketHeader) ) or …
ConfigurationspaceCraftId : IntcheckCrc : BoolidleVcid : VcididleApid : Apid
VcduConfigvcid : VcidminPacketSize : IntmaxPacketSize : Intapids : Apid[]
11
1..*
1
SarPacketHeadercoarseTime : IntfineTime : Int
context VcduHeader inv: self.vcdu.trans.config.vcduConfigs.vcid
→exists(x | x = self.virtualChanId)
OCL constraints
«DataType» Apid
value : Int
1
Transmission Vcdu
VcduHeader
Packet
PacketZone
ActiveZoneIdleZone
1 11..*1
1..*
1
versionNumber : IntspaceCraftId : IntvirtualChanId : Int
apidValue : ApidsequenceCount : IntdataLength : Int
PacketHeader
prev
context Packet inv: ( self.apidValue.value=3 or self.apidValue.value=4 and self.pHeader.oclIsTypeOf(MsiPacketHeader) ) or …
ConfigurationspaceCraftId : IntcheckCrc : BoolidleVcid : VcididleApid : Apid
VcduConfigvcid : VcidminPacketSize : IntmaxPacketSize : Intapids : Apid[]
11
1..*
1
«Replacement» MsiPacketHeadercoarseTime : IntfineTime : IntopMode : Intstatus : Int
GpsrPacketHeadercoarseTime : IntfineTime : Int
context VcduHeader inv: self.vcdu.trans.config.vcduConfigs.vcid
→exists(x | x = self.virtualChanId)
OCL constraints
«DataType» Apid
value : Int
OriginalModel Instance Incomplete Updated
Model Instance
Data LoadingSlice-basedConstraint
Solving
AutomaticModel
TransformationData Writing
Valid UpdatedModel Instance
UpdatedDataField
Data
Data Model
t1:Transmissionp1: Packet
apidValue=apid1sequenceCount=1dataLength=200
ph1: SarHeadercoarseTime=10fineTime=30
pd1: PacketDatadata=..
c1: ConfigurationspaceCraftId=1checkCrc=trueidleVcid=vcid0idleApid=apid0
vc1: VcduConfigvcid=vcid1minPacketSize=200maxPacketSize=300apids=<apid1,apid2>
v1:Vcdu a1:ActiveZone
vh1: VcduHeaderversionNumber=1spaceCraftId=1virtualChanId=vcid1…
p2: PacketapidValue=apid1sequenceCount=2dataLength=200
ph2: SarHeadercoarseTime=10fineTime=30
pd2: PacketDatadata=..
p3: PacketapidValue=apid1sequenceCount=1dataLength=200
ph3: SarHeadercoarseTime=10fineTime=30
pd3: PacketDatadata=..
Original Model Instance
OriginalModel Instance Incomplete Updated
Model Instance
Data LoadingSlicing-based
ConstraintSolving
AutomaticModel
TransformationData Writing
Valid UpdatedModel Instance
UpdatedDataField
Data
Data Model
t1:Transmissionp1: Packet
apidValue=apid1sequenceCount=1dataLength=200
ph1: SarHeadercoarseTime=10fineTime=30
pd1: PacketDatadata=..
c1: ConfigurationspaceCraftId=1checkCrc=trueidleVcid=vcid0idleApid=apid0
vc1: VcduConfigvcid=vcid1minPacketSize=200maxPacketSize=300apids=<apid1,apid2>
v1:Vcdu a1:ActiveZone
vh1: VcduHeaderversionNumber=1spaceCraftId=1virtualChanId=vcid1…
p2: PacketapidValue=apid1sequenceCount=2dataLength=200
ph2: SarHeadercoarseTime=10fineTime=30
pd2: PacketDatadata=..
p3: PacketapidValue=apid1sequenceCount=1dataLength=200
ph3: SarHeadercoarseTime=10fineTime=30
pd3: PacketDatadata=..
Replace obsolete classes
t1:Transmissionp1: Packet
apidValue=apid1sequenceCount=1dataLength=200
ph1: MsiHeadercoarseTime=10fineTime=30opMode=status=
pd1: PacketDatadata=..
c1: ConfigurationspaceCraftId=1checkCrc=trueidleVcid=vcid0idleApid=apid0
vc1: VcduConfigvcid=vcid1minPacketSize=200maxPacketSize=300apids=<apid1,apid2>
v1:Vcdu a1:ActiveZone
vh1: VcduHeaderversionNumber=1spaceCraftId=1virtualChanId=vcid1…
p2: PacketapidValue=apid1sequenceCount=2dataLength=200
pd2: PacketDatadata=..
p3: PacketapidValue=apid1sequenceCount=1dataLength=200
pd3: PacketDatadata=..
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
Replace obsolete classes
t1:Transmissionp1: Packet
apidValue=apid1sequenceCount=1dataLength=200
ph1: MsiHeadercoarseTime=10fineTime=30opMode=status=
pd1: PacketDatadata=..
c2: ConfigurationspaceCraftId=1checkCrc=trueidleVcid=vcid0idleApid=apid0
vc2: VcduConfigvcid=vcid2minPacketSize=200maxPacketSize=300apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu a1:ActiveZone
vh1: VcduHeaderversionNumber=1spaceCraftId=1virtualChanId=vcid1…
p2: PacketapidValue=apid1sequenceCount=2dataLength=200
pd2: PacketDatadata=..
p3: PacketapidValue=apid1sequenceCount=1dataLength=200
pd3: PacketDatadata=..
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
Load new configuration
t1:Transmissionp1: Packet
apidValue=apid1sequenceCount=1dataLength=200
ph1: MsiHeadercoarseTime=10fineTime=30opMode=status=
pd1: PacketDatadata=..
c2: ConfigurationspaceCraftId=1checkCrc=trueidleVcid=vcid0idleApid=apid0
vc2: VcduConfigvcid=vcid2minPacketSize=200maxPacketSize=300apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu a1:ActiveZone
vh1: VcduHeaderversionNumber=1spaceCraftId=1virtualChanId=vcid1…
p2: PacketapidValue=apid1sequenceCount=2dataLength=200
pd2: PacketDatadata=..
p3: PacketapidValue=apid1sequenceCount=1dataLength=200
pd3: PacketDatadata=..
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
context VcduHeader inv: self.vcdu.trans.con g.vcduCon gs.vcid→exists(x | x = self.virtualChanId)
Updated data may invalidate constraints
t1:Transmissionp1: Packet
apidValue=apid1sequenceCount=1dataLength=200
ph1: MsiHeadercoarseTime=10fineTime=30opMode=status=
pd1: PacketDatadata=..
c2: ConfigurationspaceCraftId=1checkCrc=trueidleVcid=vcid0idleApid=apid0
vc2: VcduConfigvcid=vcid2minPacketSize=200maxPacketSize=300apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu a1:ActiveZone
vh1: VcduHeaderversionNumber=1spaceCraftId=1virtualChanId=vcid1…
p2: PacketapidValue=apid1sequenceCount=2dataLength=200
pd2: PacketDatadata=..
p3: PacketapidValue=apid1sequenceCount=1dataLength=200
pd3: PacketDatadata=..
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
context Packet inv: …( (self.apidValue.value=3 or self.apidValue.value=4) and self.pHeader.oclIsTypeOf(MsiPacketHeader) ) or …
Updated constraints may not hold
t1:Transmissionp1: Packet
apidValue=apid1sequenceCount=1dataLength=200
ph1: MsiHeadercoarseTime=10fineTime=30opMode=status=
pd1: PacketDatadata=..
c2: ConfigurationspaceCraftId=1checkCrc=trueidleVcid=vcid0idleApid=apid0
vc2: VcduConfigvcid=vcid2minPacketSize=200maxPacketSize=300apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu a1:ActiveZone
vh1: VcduHeaderversionNumber=1spaceCraftId=1virtualChanId=vcid1…
p2: PacketapidValue=apid1sequenceCount=2dataLength=200
pd2: PacketDatadata=..
p3: PacketapidValue=apid1sequenceCount=1dataLength=200
pd3: PacketDatadata=..
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
Incomplete Updated Model Instance
OriginalModel Instance Incomplete Updated
Model Instance
Data LoadingSlicing-based
ConstraintSolving
AutomaticModel
TransformationData Writing
Valid UpdatedModel Instance
UpdatedDataField
Data
Data Model
t1:Transmissionp1: Packet
apidValue=apid1sequenceCount=1dataLength=200
ph1: MsiHeadercoarseTime=10fineTime=30opMode=status=
pd1: PacketDatadata=..
c2: ConfigurationspaceCraftId=1checkCrc=trueidleVcid=vcid0idleApid=apid0
vc1: VcduConfigvcid=vcid2minPacketSize=200maxPacketSize=300apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu a1:ActiveZone
vh1: VcduHeaderversionNumber=1spaceCraftId=1virtualChanId=vcid1…
p2: PacketapidValue=apid1sequenceCount=2dataLength=200
pd2: PacketDatadata=..
p3: PacketapidValue=apid1sequenceCount=1dataLength=200
pd3: PacketDatadata=..
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
Identify all the root-to-leaf paths
t1:Transmissionp1: Packet
apidValue=apid1sequenceCount=1dataLength=200
ph1: MsiHeadercoarseTime=10fineTime=30opMode=status=
pd1: PacketDatadata=..
c2: ConfigurationspaceCraftId=1checkCrc=trueidleVcid=vcid0idleApid=apid0
vc1: VcduConfigvcid=vcid2minPacketSize=200maxPacketSize=300apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu a1:ActiveZone
vh1: VcduHeaderversionNumber=1spaceCraftId=1virtualChanId=vcid1…
p2: PacketapidValue=apid1sequenceCount=2dataLength=200
pd2: PacketDatadata=..
p3: PacketapidValue=apid1sequenceCount=1dataLength=200
pd3: PacketDatadata=..
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
Generate a slice for each
a1:ActiveZonet1:Transmissionp1: Packet
apidValue=apid1sequenceCount=1dataLength=200
ph1: MsiHeadercoarseTime=10fineTime=30opMode=status=
pd1: PacketDatadata=..
c2: ConfigurationspaceCraftId=1checkCrc=trueidleVcid=vcid0idleApid=apid0
vc1: VcduConfigvcid=vcid2minPacketSize=200maxPacketSize=300apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu
vh1: VcduHeaderversionNumber=1spaceCraftId=1virtualChanId=vcid1…
p2: PacketapidValue=apid1sequenceCount=2dataLength=200
pd2: PacketDatadata=..
p3: PacketapidValue=apid1sequenceCount=1dataLength=200
pd3: PacketDatadata=..
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
Generate a slice for each
All the instances “linked by an OCL”to an element of the path must belong to the slice
context PacketZone inv: self.packets→forAll(x : Packet | self.vcdu.trans.config.vcduConfigs→select(y : VcduConfig |
y.vcid = self.vcdu.vcduHeader.virtualChannelId).apids→exists(z : Apid | z = x.apidValue) …
t1:Transmissionp1: Packet
apidValue=apid1sequenceCount=1dataLength=200
ph1: MsiHeadercoarseTime=10fineTime=30opMode=status=
pd1: PacketDatadata=..
c2: ConfigurationspaceCraftId=1checkCrc=trueidleVcid=vcid0idleApid=apid0
vc1: VcduConfigvcid=vcid2minPacketSize=200maxPacketSize=300apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu a1:ActiveZone
vh1: VcduHeaderversionNumber=1spaceCraftId=1virtualChanId=vcid1…
p2: PacketapidValue=apid1sequenceCount=2dataLength=200
pd2: PacketDatadata=..
p3: PacketapidValue=apid1sequenceCount=1dataLength=200
pd3: PacketDatadata=..
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
Generate a slice for each
Elements of collections are linked to the previous element only
t1:Transmission
p1: PacketapidValue=apid1sequenceCount=1dataLength=200
ph1: MsiHeadercoarseTime=10fineTime=30opMode=status=
c2: ConfigurationspaceCraftId=1checkCrc=trueidleVcid=vcid0idleApid=apid0
vc1: VcduConfigvcid=vcid2minPacketSize=200maxPacketSize=300apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu
a1:ActiveZone
vh1: VcduHeaderversionNumber=1spaceCraftId=1virtualChanId=vcid1…
“Solve” a slice//Declare classesabstract sig Vcdu {trans:one Transmission, packetZone:onePacketZone }..
//Declare class instancesone sig vc2 extends VcduConfig{}..
//Facts for associationsfact t1.config=c2..//Facts for config variablesfact { vc2.apids = apid3+apid4..
//Facts for data variablesfact { p1.apidValue = apid1 }fact { p1.sequenceCount = 1 }
//Alloy predicate for OCLspred PacketHeaderApid []{ all e: PacketHeader | …
Alloy Model
UMLt
oAllo
y
OCL constraints…
t1:Transmission
p1: PacketapidValue=apid1sequenceCount=1dataLength=200
ph1: MsiHeadercoarseTime=10fineTime=30opMode=status=
c2: ConfigurationspaceCraftId=1checkCrc=trueidleVcid=vcid0idleApid=apid0
vc1: VcduConfigvcid=vcid2minPacketSize=200maxPacketSize=300apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu
a1:ActiveZone
vh1: VcduHeaderversionNumber=1spaceCraftId=1virtualChanId=vcid1…
“Solve” a slice
UMLt
oAllo
y
OCL constraints…
Alloy Analyzer No Solution
Relax Model
//Facts for data variablesfact { p1.apidValue = apid1 }fact { p1.sequenceCount = 1 }
//Alloy predicate for OCLspred PacketHeaderApid []{ all e: PacketHeader | …
Remove facts that prevent
the generation of a solution
Alloy Instance
t1:Transmission
p1: PacketapidValue=apid3sequenceCount=1dataLength=200
ph1: MsiHeadercoarseTime=10fineTime=30opMode=1status=1
c2: ConfigurationspaceCraftId=1checkCrc=trueidleVcid=vcid0idleApid=apid0
vc1: VcduConfigvcid=vcid2minPacketSize=200maxPacketSize=300apids=<apid3,apid4,
apid5,apid6>
v1:Vcdu
a1:ActiveZone
vh1: VcduHeaderversionNumber=1spaceCraftId=1virtualChanId=vcid1…
“Solve” a slice
UMLt
oAllo
y
OCL constraints…
Alloy Analyzer No Solution
Relax Model
//Facts for data variablesfact { p1.apidValue = apid1 }fact { p1.sequenceCount = 1 }
//Alloy predicate for OCLspred PacketHeaderApid []{ all e: PacketHeader | …
Remove facts that prevent
the generation of a solution
Alloy Instance
Solve slice
Slice withupdated values
Copy toIncomplete
Model Instance
Generateslices
Incomplete Updated Model Instance
p2: PacketapidValue=apid1sequenceCount=2dataLength=200
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
t1:Transmission
v1:Vcdu
a1:ActiveZone
vh1
c2:..
vc1…
p1: PacketapidValue=apid3sequenceCount=1dataLength=200
ph1: MsiHeadercoarseTime=10fineTime=30opMode=1status=1
“Solve” next slice
p2: PacketapidValue=apid3sequenceCount=2dataLength=200
ph2: MsiHeadercoarseTime=10fineTime=30opMode=status=
t1:Transmission
v1:Vcdu
a1:ActiveZone
vh1
c2:..
vc1…
p1: PacketapidValue=apid3sequenceCount=1dataLength=200
ph1: MsiHeadercoarseTime=10fineTime=30opMode=1status=1
“Solve” next slice
Solving algorithm does not rewrite
variables generatedin previous steps
Solve slice
Slice withupdated values
Copy toIncomplete
Model Instance
Generateslices
Incomplete Updated Model Instance
If a value “used” in a previous slicehas been modified, restart from first
Solve slice
Slice withupdated values
Copy toIncomplete
Model Instance
Generateslices
Valid Updated Model Instance
Incomplete Updated Model Instance
OriginalModel Instance Incomplete Updated
Model Instance
Data LoadingSlice-basedConstraint
Solving
AutomaticModel
TransformationData Writing
Valid UpdatedModel Instance
UpdatedDataField
Data
Data Model
Empirical Evaluation
• RQ1: How does the proposed approach compare to a non-slicing approach?
• RQ2: Does the proposed approach scale to a practical extent?
• RQ3: Does the proposed approach allow for the effective testing of new data requirements?
• RQ4: Does the proposed approach allow for efficient and effective system testing compared to manual approach?
Case Study• Data Processing System for Satellite Data developed by SES
• Data Model
• 82 classes, 322 attributes, 56 associations, 52 OCL constraints
• Manual testing is expensive
• 130 class instances, 261 attributes, 1152 constrained attribute values
RQ1: How does the proposed approach compare to a non-slicing approach?
• To be justified the proposed approach should provide an advantage over one that does not use slicing
• Created a simplified technique that does not apply slicing
• All attribute values are generated from scratch from a single execution of the Alloy Analyzer
• Generated test inputs from 1 to 40 VCDUs
• 10 inputs for each size
RQ2: Does the proposed approach scale to a practical extent?
• Generation of new data should be fast enough and scale effectively as file size increases
• Generated test inputs from 50 to 500 VCDUs
• 10 inputs for each size
Execution time required to generate inputs vs number of VCDUs
Non-trivial inputsgenerated in reasonable time
RQ3: Does the proposed approach allow for the effective testing of new data requirements?
• Automatically generate new data intended for use in robustness test cases
• Integrate a mutation approach (ICST’15) to generate robustness test cases
• Measure code coverage of the source code that implements the new data requirements
• Compare with test cases written by engineers
Field Data (Version 0)
Updated Data
(Version 1)
RobustnessTest Inputs(Version 1)
Slicing BasedGeneration
Mutation BasedGeneration
RQ3: Does the proposed approach allow for the effective testing of new data requirements?
Test suite
S2 Instructions Covered S2 Branches Covered # of Tests
Avg Min Max Avg Min Max
Auto 74 (77.9%)
74 (77.9%)
74 (77.9%)
9 (81.8%)
9 (81.8%)
9 (81.8%)
103.1
Test suite
S2 Instructions Covered S2 Branches Covered # of Tests
Avg Min Max Avg Min Max
Auto 74 (77.9%)
74 (77.9%)
74 (77.9%)
9 (81.8%)
9 (81.8%)
9 (81.8%)
103.1
Manual 68 (71.6%)
- - 8 (72.7%)
- - 32
More effective than manual testing
RQ4: Does the proposed approach allow for effective system testing?
• In the presence of a new software version engineers are interested in testing the whole system
• Necessary to test unmodified functions when they process a new a data type (e.g. packet data extraction in the presence of a new packet type)
• Same approach of RQ3: robustness testing
• Measure code coverage of the whole system
• Compare with 32 expensive manually crafted test inputs
RQ4: Does the proposed approach allow for effective system testing?
More effective than manual testing
Test suite
Instructions Covered Branches Covered # of Tests
Avg Min Max Avg Min Max
Auto 23,432.1(72.2%)
23,273(71.7%)
23,529(72.5%)
978.7 (51.3%)
957 (50.2%)
987 (51.8%)
103.1
Manual 23,046(71.0%)
- - 950 (49.8%)
- - 32
RQ4: Does the proposed approach allow for efficient and effective system testing?
• Both data modelling and manual testing are expensive
• Data Model: 82 classes, 322 attributes, 56 associations, 52 OCL constraints
• Manual test: 130 class instances, 261 attributes, 1152 constrained attribute values
• Data model useful for other purposes (e.g., discuss with system engineers)
• Manual and automatically generate tests have negligible execution time:
• Manual, 10 mins – Automatically generated, 30 mins
• Automatically generated tests are more effective