M Gallas CERN EP-SFT

LCG-SPI: SW-Testing 1

LCG-SPI: SW-Testing

LCG AppArea internal review (20/10/03) Manuel.Gallas@cern.ch

LCG/SPILCG Software Process & Infrastructure

M Gallas CERN EP-SFT

LCG-SPI: SW-Testing 2

Outline

SPI SW-Testing

• SW-Testing Overview• LCG-SPI SW-Testing• Test Frameworks• Test Documents• User support

M Gallas CERN EP-SFT

LCG-SPI: SW-Testing 3

•Software testing should be an integral part of the software development in the LCG App Area.•All level of software testing should be run as part of an automatic process.

SW-Testing Overview

GOAL (from RTAG2)

Software testing

Integration test

System test

Acceptance test

Unit test

Sw-testing team

IntegrationTests

System Tests

ExamplesLHC experiments

Use in Exp.Test FrameWorks

SW-testing doc

SW- testing support

SPI SW-Testing

SW-testing polices

Code Documentation

Software developer

CODE

Work PackageTest

Automated testing

Nightly Building

System

M Gallas CERN EP-SFT

LCG-SPI: SW-Testing 4

LCG-SPI SW-Testing

Aim: to help developers: to help projects:

• to produce test code suitable to be run in automatic way • to document their tests

• to plan their test phase• to run all their test in automatic way • to browse and scan easily the test results and failures

Our inputs: • Contacts within HEP-community.• What is available as free open source code.

Our constraints: • Avoid commercial software and licensing problems. • Avoid “do it yourself solutions”•Try to adopt commonly used open-source software.

Requirements from the users : • Work with different languages (C++, Python, …) and in different environments (platforms & compilers)• Allow two ways of testing: (1) “check the output of the test-code” (2) “check the test results inside the test-code”.• Provide a easy way to integrate existing tests.• Allow the creation of dependencies among tests.• Easily establish a criteria if a test should pass or fail in a given release• Organize the tests by components or packages / integration / system /acceptance .• Regression testing and check respect to the previous sw-release.• Easy integration with the Nightly Building System.• Provide also a graphical interface to run the tests and examine the test results.

M Gallas CERN EP-SFT

LCG-SPI: SW-Testing 5

Test Frameworks: Global picture

X-Unit familyJUnit QtUnit

CppUnit PyUnit

Oval

Test Scripts

Old tests

SW Product Examples

Unit testing

Acceptanc

etesting

QA activity

End User Installation

checks

Nightly Building

Project Release

Sw-Testing

QMTest• Uses a GUI for creating and running tests (also in batch).• Can run tests in parallel, supports execution of a single test or many at once (test-cases & test-suites) • Organizes tests hierarchically• Records dependencies among tests

Top layer

Bottom layer

- Adaptable to theprogramming language and developer- Prepared to be run in automatic way

- Integrates different ways to test.

- Common environment to run the tests and to access the test results.

M Gallas CERN EP-SFT

LCG-SPI: SW-Testing 6

•Main Scope: Unit–testing

•The same “assertion style” in different languages.

•Provides:- The name of the test case that failed.

-The name of the source file that contains the test.

-The line number where the failure occurred

• Different platforms/compilers: (Linux/Solaris/Windows)

Test Frameworks: X-Unit family

X-Unit family

JUnit

QtUnit

CppUnit

PyUnit

A simple test:

1. Subclass the TestCase CppUnit or PyUnit class

2. Override the method runTest().

3. When you want to check a value, call the (CppUnit or PyUnit) ASSERT(bool) and pass in an expresion that is true if the test succeeds

M Gallas CERN EP-SFT

LCG-SPI: SW-Testing 7

Test Frameworks: Oval

Oval:• validation and regression • developed & used within CMS experiment

•Can be use for Unit-test to Validation tests.

•Compare the output log file with a given reference file (Smart comparison of those lines which start with “[Oval]”)

• It is possible to set different run environments.

• Can run external scripts and external binaries.

Ovalfile(configuration)

ExecutableTest

Source Code

Referencefile

Log file

Oval run

Ovaldiff

Oval build

M Gallas CERN EP-SFT

LCG-SPI: SW-Testing 8

Test Frameworks: QMTest

QMTest

•Uses a graphical interface for creating and running tests

•The configuration files are in XML and can be created from the GUI. We provide also script to do it

• Runs tests in parallel

• Organizes tests hierarchically

• Supports execution of a single test or many at once

• Records dependencies among tests

• Can be run in batch mode -> easy integration with the Nightly-Building systems

M Gallas CERN EP-SFT

LCG-SPI: SW-Testing 9

Test Documents

• For all project is needed a Test Plan in order to have a clear idea of the type of tests, schedule, environment and responsibilities.• We use a template•Test Case template collects information about the existing tests and how they cover the expected functionality of the software.

M Gallas CERN EP-SFT

LCG-SPI: SW-Testing 10

User support: Web information

http://spi.cern.ch/

Test Frameworks

HowTo

Test doc

Policies

http://spi.cern.ch/testing

M Gallas CERN EP-SFT

LCG-SPI: SW-Testing 11

User support: How-To

3 Follow sw-testing policies

Automatic testing requires some standardization

1

2

M Gallas CERN EP-SFT

LCG-SPI: SW-Testing 12

• SW-Testing Overview• LCG-SPI SW-Testing• Test Frameworks• Test Documents• User support

SPI SW-Testing

Thanks to:• LCG-PI team • LCG-POOL team• LCG-SEAL team• LCH-experiments

• Software testing is a needed step in the software development process.

• SPI provides test frameworks, templates and help to have a coherent automatic testing procedure with a common interface to run and browse the test results.

•The proposed testing tools were chosen following the criteria of a simple approach, tools in use in the experiments and freely available.

Final Remarks

• Following standards makes life easier to SPI and the users. •Automatic testing is always needed and is a way to consolidatethe software.• Bug reports and typical use-cases from the users should be converted into tests

• SPI SW-testing has to evolve and consolidate followingthe needs of the LCG projects. •All SPI sw-testing tools, templates, documentation can be used by any project.