V1.0 | 2018-11-06

Concepts for Unit and Functional System Tests

It's All About Software Testing

2

Situation: Perspective of Unit Testing and Functional System Testing

3

Goal: Joined Perspective of Testing

4

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Reuse in the test

design

Flexibility in the

chosen test design

method

Consideration of

both DUT and test

variants

Adequate test

coverage

Analysis and

reporting during

and after test run

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

5

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Flexibility in the

chosen test design

method

Consideration of

both DUT and test

variants

Adequate test

coverage

Analysis and

reporting during

and after test run

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

6

SUT to be stimulated during test execution by different kind of “sources”:

Sequential test input to stimulate and check the SUT

System environment, e.g. remaining bus simulation with other ECUs

The “real world”, e.g. a road course

Mapping the asynchronous behavior of the system environment and “real world” to a sequential test script is hardly possible

Challenges with Testing

SUT

Test Script

7

Challenges with Testing

Test Script

8

Model

• plant model

Challenges with Testing

Model

• system environment

Test Script

9

Challenges with Testing

SUT

Data Flow

Control Flow

Model

• plant model

Model

• system environment

Test Script

10

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Flexibility in the

chosen test design

method

Consideration of

both DUT and test

variants

Adequate test

coverage

Analysis and

reporting during

and after test run

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

11

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Flexibility in the

chosen test design

method

Consideration of

both DUT and test

variants

Adequate test

coverage

Analysis and

reporting during

and after test run

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

12

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Consideration of

both DUT and test

variants

Adequate test

coverage

Analysis and

reporting during

and after test run

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

13

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Consideration of

both DUT and test

variants

Adequate test

coverage

Analysis and

reporting during

and after test run

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

int a

int a

float b

int a

float b

int64 x

int a

int a

float b

int64 x

14

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Consideration of

both DUT and test

variants

Adequate test

coverage

Analysis and

reporting during

and after test run

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

15

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Adequate test

coverage

Analysis and

reporting during

and after test run

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

16

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Adequate test

coverage

Analysis and

reporting during

and after test run

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

int a

int a

float b

int a

int a

float b

int64 x

int a

float b

float b

int64 x

int64 x

17

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Adequate test

coverage

Analysis and

reporting during

and after test run

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

18

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Variant handling

for test structure,

test logic and

parameters

Adequate test

coverage

Analysis and

reporting during

and after test run

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

19

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Variant handling

for test structure,

test logic and

parameters

Adequate test

coverage

Analysis and

reporting during

and after test run

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

#ifdef

20

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Variant handling

for test structure,

test logic and

parameters

Adequate test

coverage

Analysis and

reporting during

and after test run

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

21

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Variant handling

for test structure,

test logic and

parameters

Adequate test

coverage

Design features

for definition of

relevant test case

data and

generation of test

cases

Analysis and

reporting during

and after test run

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

22

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Variant handling

for test structure,

test logic and

parameters

Adequate test

coverage

Design features

for definition of

relevant test case

data and

generation of test

cases

Analysis and

reporting during

and after test run

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

int a float b int64 x

23

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Variant handling

for test structure,

test logic and

parameters

Adequate test

coverage

Design features

for definition of

relevant test case

data and

generation of test

cases

Analysis and

reporting during

and after test run

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

24

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Variant handling

for test structure,

test logic and

parameters

Adequate test

coverage

Design features

for definition of

relevant test case

data and

generation of test

cases

Analysis and

reporting during

and after test run

Analysis windows,

e.g. for

visualization of

value charts

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

25

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Variant handling

for test structure,

test logic and

parameters

Adequate test

coverage

Design features

for definition of

relevant test case

data and

generation of test

cases

Analysis and

reporting during

and after test run

Analysis windows,

e.g. for

visualization of

value charts

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

float b

int a

int64 x

int a

26

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Variant handling

for test structure,

test logic and

parameters

Adequate test

coverage

Design features

for definition of

relevant test case

data and

generation of test

cases

Analysis and

reporting during

and after test run

Analysis windows,

e.g. for

visualization of

value charts

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Traceability and

Continuous

Integration

27

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Variant handling

for test structure,

test logic and

parameters

Adequate test

coverage

Design features

for definition of

relevant test case

data and

generation of test

cases

Analysis and

reporting during

and after test run

Analysis windows,

e.g. for

visualization of

value charts

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Measure code

coverage both

with unit and

system tests

Traceability and

Continuous

Integration

28

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Variant handling

for test structure,

test logic and

parameters

Adequate test

coverage

Design features

for definition of

relevant test case

data and

generation of test

cases

Analysis and

reporting during

and after test run

Analysis windows,

e.g. for

visualization of

value charts

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Measure code

coverage both

with unit and

system tests

Traceability and

Continuous

Integration

Set LightIntensity to 30

Wait 500ms

Check LightState is on

Call x = GetLightState(intensity = 30)

Check x == 1

29

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Variant handling

for test structure,

test logic and

parameters

Adequate test

coverage

Design features

for definition of

relevant test case

data and

generation of test

cases

Analysis and

reporting during

and after test run

Analysis windows,

e.g. for

visualization of

value charts

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Measure code

coverage both

with unit and

system tests

Traceability and

Continuous

Integration

30

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Variant handling

for test structure,

test logic and

parameters

Adequate test

coverage

Design features

for definition of

relevant test case

data and

generation of test

cases

Analysis and

reporting during

and after test run

Analysis windows,

e.g. for

visualization of

value charts

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Measure code

coverage both

with unit and

system tests

Traceability and

Continuous

Integration

Connection to test

management tools

and Jenkins,

changed-based

testing

31

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Variant handling

for test structure,

test logic and

parameters

Adequate test

coverage

Design features

for definition of

relevant test case

data and

generation of test

cases

Analysis and

reporting during

and after test run

Analysis windows,

e.g. for

visualization of

value charts

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Measure code

coverage both

with unit and

system tests

Traceability and

Continuous

Integration

Connection to test

management tools

and Jenkins,

changed-based

testing

Req1:

Pressing the button switches the light on.

Req1:

Passing function parameter value ‚1‘ leads to return value ‚true‘.

32

Challenges with Testing

Early test of

components,

independent of

availability of

other components

Usage of models

Reuse in the test

design

Separation of test

logic,

implementation

and test data

Flexibility in the

chosen test design

method

Integration of

different test

notations in one

tool (graphical,

tabular, coding)

Consideration of

both DUT and test

variants

Variant handling

for test structure,

test logic and

parameters

Adequate test

coverage

Design features

for definition of

relevant test case

data and

generation of test

cases

Analysis and

reporting during

and after test run

Analysis windows,

e.g. for

visualization of

value charts

Code coverage to

meet standards as

ISO26262,

IEC62304,

DO-178B

Measure code

coverage both

with unit and

system tests

Traceability and

Continuous

Integration

Connection to test

management tools

and Jenkins,

changed-based

testing

33

Conclusion

The requirements and challenges on test environments of HIL system tests, SW system tests, integration tests and unit tests are widely the same

Corresponding test techniques are applicable to all kind of tests

Vector strategy: provide a comprehensive test solution for all kind of tests based on well established tools

Combine the strength of vTESTstudio and CANoe for system tests with the strength of VectorCAST for code based tests

34 © 2018. Vector Informatik GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.0 | 2018-11-06

Tariq Javaid Vector UK

For more information about Vectorand our products please visit

www.vector.com