Model Based Test Validation and Oracles for Data Acquisition Systems

University of Luxembourg Interdisciplinary Centre for Security, Reliability and Trust Software Verification and Validation Lab November 14, 2013

Daniel Di Nardo, Nadia Alshahwan, Lionel Briand, Elizabeta Fourneret University of Luxembourg Tomislav Nakic, Vincent Masquelier SES S.A., Luxembourg

2

DAQ System

Configurations

Log Files

Data

Context: Data Acquisition Systems

Structured/Complex

Defines how input transmissions are processed

Captures what happened while processing

Context and Problem Definition

3

•  Complexity of DAQ systems lies in the structure of the input transmission data and output log files and the mappings between these two elements.

•  Problem: –  Testing of such systems is difficult. –  Test cases can be composed of large and highly complex input and

output files.

Why is DAQ Testing so Hard?

4

•  Manually constructing test inputs is too much work

•  Input data has a complex specific structure –  Dependencies between different fields –  Dependencies between input fields and configuration fields

•  System output logs are also complex and their size can grow very large –  Manual inspection time consuming

•  Test oracle problem –  Manual validation of test outputs is challenging and error-prone

•  Automation specifically designed for these systems is required

State of the Art

5

•  No directly related work •  Model-based testing (MBT)

–  Large body of work –  Focused on behavioral models –  Not appropriate for modeling complex input/output file mappings

•  Modelling DAQ Systems –  Focused on simulation and design –  Test Oracle Automation not addressed by current approaches –  Test Case Validation has not yet been addressed

6

DAQ System

Configurations

Log Files

Data

How to Represent the System?

A Model Based Approach

7

DAQ System

Input-Input

Input-Config

Input-Output

Constraints (OCL): Input Transmission File

Config Files Output log Files

OCL

OCL

OCL

Modelling Methodology

8

•  Model system input and output data to automate test validation and oracle checking

•  Requirements and domain knowledge is used without source code

analysis èApproach is black box

•  Modelling notation is not sufficient

•  Precise methodology to support the modelling objectives is necessary

Transmission File Structure

9

•  A satellite transmission consists of multiple Channel Data Units

Channel  Data  Unit  Sync  

Marker  Frame  Data   Reed  Solomon  check  symbols  

Frame  Data  Header   PacketZone  

Header  Spacecra=  ID   Virtual  Channel  ID   Frame  Count   Header  Error  Control  

Modelling the File Structure

10

Example Constraint on Input and Configuration

11

•  Constraint to ensure that the virtual channelId is valid (can be one of many active ones or can be the idle channel).

!!context Header inv:!!let config : configurationData =! self.frameData.channelData.transmissionData.configuration!in!! config.cId->exists(x | x = self.channelId)! or! self.channelId = config.idleCId

Example Oracle Constraint

12

context ChannelData inv:!let! frameCount : Integer = self.frameData.header.frameCount,! prevFrameCount : Integer = self.prevOnChannel.frameData.header.frameCount !in !!not self.prevOnChannel->isEmpty() and !

if prevFrameCount < 16777215 ! then frameCount <> prevFrameCount + 1 !else prevFrameCount = 16777215 and frameCount <> 0 !endif !

implies !

self.transmissionData.outputData.frameDataReport.event! .eventType->exists(i | i = Events::COUNTER_JUMP))

•  Constraint to ensure that the COUNTER_JUMP event is captured in the output log.

Modelling Methodology – Applications

13

Four applications of the model in practice:

1.  Test Design and Generation

2.  Test Oracle

3.  Specifications Refinement

4.  Run-Time Verification

Modelling Methodology – Applications

14

Four applications of the model in practice:

1.  Test Design and generation

2.  Test Oracle

3.  Specifications Refinement

4.  Run-Time Verification

Automation

15

•  Tool architecture for the automation of test validation and oracles.

Case Study – Subject Selection

16

•  Selected a representative, complex DAQ system

•  System accepts a binary satellite transmission data file

•  Configuration files

•  Log files report on results of processing input file

•  32 test cases are approved for system validation (manually written)

•  Also use real transmission files (2 GB) to validate our approach and assess its scalability

Case Study – Execution and Data Collection

17

Study DAQ System

Create the Model

Validate the Test Inputs

Check Oracle on Test Inputs/Outputs

Record Execution Times

Case Study – RQ1

18

RQ1: How much effort is needed to produce the model for a real system?

Case Study Results – Modelling Effort

19

Modelling Effort

•  One man-month

•  Time needed is largely dependent on the person’s domain knowledge and expertise in modelling and in OCL

•  Size of the model

•  A surrogate measure to estimate the effort needed to follow our modelling methodology in a specific context

Case Study Results – Modelling Effort

20

•  Size of the Input, Configuration and Output Models that were created for the Case Study System

File Classes Attributes Associations Generalisations Input 36 156 17 4 Configuration 9 30 6 1 Output 23 132 15 0 Total 68 318 38 5

Case Study Results – Modelling Effort

21

•  Information about Constraints for the Case Study System Classified by the Files to which they apply

File # of Constraints

# of Clauses

# of Opers. on Collections

# of Iterative

Opers. Input 15 30 6 1 Input/Configuration

12 54 14 6

Input/Output 10 38 2 10 Input/Configuration/Output

12 87 15 19

Total 49 209 37 36

Case Study Results – Modelling Effort

22

RQ1: How much effort is needed to produce the model for a real system? Answer: •  Results show that the size of the model is much less than what is typically

observed when modelling system designs

•  Cost of modelling was considered acceptable by the system’s engineers

Case Study – RQ2

23

RQ2: How long does it take to validate test cases and check the oracle?

Acceptance Test Cases – Execution Time

24

•  Average, minimum, maximum execution times for all acceptance test cases

Operation Execution Time (ms) Min Max Avg.

Model Instantiation

684 845 762

Test Input Validation

1 56 41

Oracle Checking < 1 39 31 Total 685 940 834

Large Transmissions - Model Instantiation Time

25

•  Model instantiation time by input file size

Large Transmissions – Input Validation Time

26

•  Input validation time by input file size

Large Transmissions - Oracle Validation Time

27

•  Oracle checking time by input file size

Case Study Results – RQ2

28

RQ2: How long does it take to validate test cases and check the oracle? Answer: •  Results show that our approach is scalable in terms of execution time

•  Test validation and oracle checking execution time on real transmission files is manageable in practice, with less than 3 and 50 minutes for input and oracle constraints, respectively

•  Linear relationship between size of input file and execution time makes it possible to potentially process much larger files

Case Study – RQ3

29

RQ3: Is the model effective in practice in validating test cases and checking the oracle? Is it effective in uncovering issues, if any, in the input files, the DAQ system, or the specifications of the system?

Case Study Results – RQ3

30

•  Validated the actual 32 test cases using our tool •  Our test validation approach could help identify specifications changes

without the need to execute the transmission file on the system

•  No violations of the oracle constraints were reported •  When validating the real transmission files, we found that in some files

many input constraints were violated

Case Study Results – RQ3

31

RQ3: Is the model effective in practice in validating test cases and checking the oracle? Is it effective in uncovering issues, if any, in the input files, the DAQ system, or the specifications of the system? Answer: •  Results show that our approach is effective in validating test cases and

checking the oracle

•  Our approach is also able to identify implicit changes in specifications of the input file and the DAQ system

•  SES system integration •  Deploy the oracle checker into the SES build process for the

current system under evaluation. •  Training / Knowledge Transfer to Industry partner.

•  Apply our methodology to other DAQ systems.

Success Story

32

33

Overall Approach

34

Current Progress on Approach Oracle and input checker is fully

working

35

Next Steps: Automated Input Generation

•  Currently defining method of automatically generating erroneous inputs based on realistic faults

•  Example faults: •  Flipped bits •  Missing packets

•  Created an automated test validation and oracle checking approach for systems with complex inputs/outputs and mappings between inputs and outputs, e.g., Data Acquisition Systems.

•  Approach driven by models of the input/output structure and content.

•  We defined a specific modelling methodology using UML class diagrams and OCL constraints.

•  Case study shows that the modelling approach is scalable.

•  Input and oracle validation executed within reasonable times.

•  And people are using it!

Conclusion

36 Supported by the Fonds National de la Recherche, Luxembourg (FNR/P10/03 and FNR 4082113)

37

•  Automated test case generation •  Simple approach:

•  Start with a valid input file •  mutate values corresponding to model leaf node (e.g., flip

bits)

•  Possible selection criteria: •  break all constraints •  break combos of constraints

•  Mutate thousands of times -> run through input validator ->

select most diverse test cases for test suite

Next Steps

38

Progress: Tool Implementation

•  Implemented a testing framework: –  Loads existing input file into the model –  Checks constraints on input and configuration –  Checks constraints on output

•  Used the tool to validate the model against existing SES test cases –  Helps in reviewing the model and constraints –  Initial test of the approach where we know the expected result

39

Modelling Methodology – File Structure

40

File Item Model Element Example Field Class ChannelData Leaf Field Class or

Attribute Sync spacecraftId

Property Attribute dataLength in Packet class Containment Composition ChannelData is composed of Sync

and FrameData Alternative Sub-components

Generalisation PacketZone can either be an IdlePacketZone or a PacketSet

Optional/Multi Sub-components

Multiplicity One PacketSet can have one or many instances of Packet

Dependency Association The association between TransmissionData and Configuration

Computation Operation calculateRsCrc

Case Study Results – Threats to Validity

41

•  Internal Threats –  we used all the test cases and transmission files provided by the

system testers to avoid experimenter bias •  External Threats

–  might only be relevant in the DAQ application domain; nevertheless, this domain is important and widely used

•  Construct Threats –  to study scalability, used the size of the model and constraints and

the execution time of the validation and oracle checking processes –  for execution time, might depend on content of file the used;

transmission files are not only real but representative