Automated Regression Testing for Embedded

Systems in Action

by Steve Peterson

AAND Tech, Inc

Situation

Client manufactures a Class II* medical device Produces reliable software. Productivity is OK All the T s are crossed and I s are dotted in terms of

process and documentation.

However …

* - See this site for an explanation of the FDA’s medical classifications: http://www.fda.gov/CDRH/devadvice/3132.html

Inefficiencies

Efficiency-wise we were getting killed by:

Requirements (traceability, maintaining) Test Plans (writing, maintaining, executing)

Old Way

Requirements written in Word documents. Manual Updates. Brute Force Traceability.

Test Plans written in Word documents. Execution by personnel.

Let’s Step Back in order to Solve This

What are the “requirements” for the process? We started with defining “project complete”

All Requirements are written. All Requirements are (successfully) tested. All Requirements are documented.

Next, Define the Deliverables*

Software/Firmware Requirements Documents Test Plans Test Results

*- This is far from a complete list, and doesn’t include items such as reviews and project plans.

Define Some Characteristics

Software and its process are Testable Changeable Traceable

Now something we would really like

For the 4 deliverables from a couple slides ago: Software/Firmware Requirements Documents Test Plans Test Results

We would like this to be in sync for the entire duration of the project!

Our Plan

Previously we did all the basics with revision controlling the documents, source code, binary images, etc.

We added: A method of automatically linking

requirements to tests/implementation A framework that will build the code (from

revision control) deploy it to the target, and perform automated regression testing.

Our Implementation*

These items are covered in detail later. Requirements remain in Word documents. However

now we have automated requirements traceability (to tests/implementation)

Automated Regression Tests Continuous Integration**

* - Many 3rd party solutions for requirements tracking and testing exist. Due to cost and limits of customization, we chose to implement our own.

** - True Continuous Integration runs the entire regression test suite after each check in. In our case we stretch the term “continuous” to mean nightly. This is due to the abbreviated test suite requiring 3 hours to run and several days for the formal V&V test suite.

Requirements Tracking Tool Dynamic and Static Real Time

generation of requirements traceability*.

Requirements based Testing. Finds untested requirements,

orphaned tests. PDF report generation Coming soon, changed

requirements, triggers for changed code.

PDF report generated by the tool tracing the requirements to test/implementation.

Static coverage occurs from parsing the requirements documents and using Ruby’s introspection on the classes to indentify coverage

Dynamic coverage occurs from running the regression test. Each assert statement lists the requirements that it’s testing. We generate coverage matrices both for assert successes and failures. For you UML fans, the relationship between asserts and requirements is many-many (an assert can test more than one requirement and a requirement can be tested by more than one assert).

Tri-State Coverage

Green Light – Test item passesRed Light – Test item failsYellow Light – No test coverage for this item. Red Light items get immediate attention. Yellow Light items are given some slack (but not too

much!) The idea is to continuously add features and keep

the code, tests, and documents in sync. No “big bang” testing or documentation effort at the

end!

Automated Testing Framework

Components Revision Control System (we used

Subversion) The Scheduler The Build engine The “Deployer” – installs the correct software

under test on the device under test, whether a web or embedded device.

Peripheral Measurement Devices.

Embedded Device

Want as much of the testing to be as non-invasive as possible.

Next revision will have robot button presses and a camera to read the display

Until then, the firmware allows simulated button presses and reading the LCD using commands via the serial port. Also a couple generate commands to read values.

Software safeguards exist to mitigate against these commands interfering with device operation.

Chassis of the DUT (Device Under Test)

with the serial port clip used to program the DUT,

Issue commands and read status

Build Engine

Performs a complete source code check out from the SVN* (Subversion) repository. The default is the trunk, but any branch, tag or SVN ID may be used. Launched from a cron job** or manual initiation.

The build may be production code base, a special developer code base, or for the embedded code, unit test code.

Applicable to both the embedded and web based technologies.

* - Subversion (SVN) is an open source revision control system: http://subversion.tigris.org/

** - A Cron job executes commands or scripts (groups of commands) automatically at a specified time/date.

The Scheduler (Page 1)

DUT (Device Under Test)

Peripheral Test Module<Programmable Power Supply,

Breathing Simulator,Manometer,

X10, power switch>

Test Run

Job

Software Version< Release Candidate,

Increment,Nightly Build,

Developer Produced>

1..*

1

1

1 1

1..* 1

UML diagram showing the relationship between a Job, Test Runs, a DUT, Peripheral Test Modules and installed Software version.

The Scheduler (Page 2)

A DUT (Device Under Test) may be a single embedded device, a web component or a system DUT consisting of both.

A test module consists of peripheral devices under script control such as a programmable power supply, breathing simulator, manometer.

A run consists of a DUT (see above), and 0 or more peripheral test modules running a single script

Web Interface for monitoring and controlling the testing “jobs”

The Scheduler (Page 3)

A job consists of 1 or more runs. May span numerous DUTs Noteworthy jobs are the formal V&V job (may require up to 3

days to run), the nightly (the full V&V run trimmed down to about 6 hours), and developer submitted jobs.

The job “manifest” specifies either a specific version of code to test (as produced by the build engine or a developer produced version).

The scheduler also acts as a traffic cop. A database tracks the attributes of various DUT units and queues up job/run requests based on run requirements.

Flash Memory Programmer (The Deployer)

Under script control, this programs the embedded device’s flash memory with the specified binary image.

Peripheral Test Modules (Page 1)

MacGyver Manometer*

* The term MacGyver Manometer is a tongue in cheek reference to the 80/s TV show, where MacGyver could create anything using a paper clip and rubber band (see photo above). In this case the DUT, which contains a pressure sensor more accurate than most handheld manometers, is outfitted with special firmware that displays the current pressure, and reports the pressure up to 16 samples/second over the serial connector.

Programmable Power Supply

Peripheral Test Modules (Page 2)

Hans Rudolph Breathing Simulator

A couple more pictures

More testing power per sq foot of space anywhere!

X10 Power Control

More Pictures

Robotic Arm Pressure Control Valve

Benefits Able to give embedded devices the same

automated regression testing benefits available to web and PC applications

Process accommodates change. It is still not free, but much less painful.

Sanity check. No more being 90% done for 50% of the project.

Much less Stress! Bugs are fixed immediately. Formal V&V testing took just a couple days, instead

of weeks. Over 97% of testing was automated!

Additional Benefits

Because we used the real devices (day after day, night after night), this doubled as life testing. Non-software items were validated as well.

Due to the easy command set for controlling the DUT and capturing information from the measuring equipment, this was used for research applications as well.

Ruby has a rich set of Microsoft Office plugins so automated formatting of data in Excel and Word is straight forward.

ChallengesAutomated Testing is not the same as a user

We found that in actuality it is (or very close). We tested the Behavior (read button presses tangible user responses). In addition we constantly had real people performing unstructured testing. If this located a problem, additional items were added to the automated testing.

How can you be sure that the tests work (how do you test the tests?)The test suites incorporate counter test cases – inject out of bounds conditions and verify that the test correctly catches it.

Contact

Steve Peterson

AAND Tech, Inc.

http://www.aandtech.com

http://www.jruby.net

Steve.peterson@aandtech.com