automated reliability testing via hardware interfaces' by bryan bakker
DESCRIPTION
The case study described in this presentation has taken place at a medical equipment manufacturer. The product developed was a medical x-ray device used during surgery operations. The system generates x-rays (called exposure) and a detector creates images of the patient based on the detected x-ray beams (called image acquisition). The image pipeline is real-time with several images per second, so the surgeon can e.g. see exactly where he is cutting the patient. The presentation describes the approach that has been taken to develop an automatic testing framework in order to execute reliability test cases and identify reliability issues. To achieve the control of the system under test, the existing hardware interfaces (physical buttons of the different keyboards, handswitches and footswitches) were used to inject the system with actions (with the use of LabVIEW). This has been done to minimize the so-called probe effect. The expected results of the test cases have been automatically retrieved from the log files generated by the system. This way the test framework could react on system failures immediately, without wasting valuable test time on scarce test systems. The log files were used to extract information about the performed actions and failures in order to measure the MTBF (Mean Time Between Failures) of different critical system functions (like start-up of the system, and image acquisition). The Crow-AMSAA model for reliability measurements has been chosen to report reliability metrics to the organization. A Return-On-Investment calculation has been performed to get buy-in from senior management who provided additional funding to further develop the testing framework, and to apply the same ideas to different products and projects. The presentation explains the points which were crucial for the success of this approach to automated reliability testing and briefly explains future plans and extensions (e.g. operational profiles).TRANSCRIPT
Automated Reliability Testing via hardware interfacesEuroSTAR 2011
Bryan Bakker
November, 2011
© Sioux Embedded Systems | Confidential | 2
Contents
Sioux Intro – The need for action Increment 1 – First success Buy-In from management Increment 2 – A language for the testers Increment 3 – Logfile interpretation ROI and Crow-AMSAA Results – key success factors
2
© Sioux Embedded Systems | Confidential | 3
About Bryan Bakker
Test Expert Certifications: ISTQB, TMap, Prince2 Member of ISTQB Expert Level on Test Automation Accredited tutor of ISTQB Foundation Domains: medical systems, professional security
systems, semi-industry, electron microscopy Specialties: test automation, integration testing, design
for testability, reliability testing
3
About Sioux
© Sioux Embedded Systems | Confidential | 4
HERENTALS
MOSCOW
NEDERWEERT
EINDHOVEN
UTRECHT
© Sioux Embedded Systems | Confidential | 5
Intro – The need for action
Medical Surgery Device: X-ray exposure + acquisition during surgery activities Real-time image chain Mobile device (frequently off/on) Quality and testing considered
important in organization
Reliability was an issue: “Frequent” startup failures Aborted acquisitions Always safe… but not reliable!
5
© Sioux Embedded Systems | Confidential | 6
The need for action
Reliability issues are nasty to analyse, solve and test Fixing defects in field (remember Boehm) Impact on other projects (development + test resources) High service costs Troublesome system test (up to 15 cycles!!)
6
Requirements Design Implementation Test Operation
Cost of defect fix (Barry Boehm)
© Sioux Embedded Systems | Confidential | 7
The need for action
Start with automated reliability tests (simple, short) Quick and dirty at first No SW resources available
To help with automation Implementing test interfaces
Goal: show (quick) that reliability issues can be reproduced
Expectation: … then attention and funding would increase
7
Hardware interfaces used to invoke actions on SUT Buttons on different keyboards Handswitches Footswitches Different power-switches
LabVIEW generates hardware signals Test cases defined in LabVIEW Only logfiles stored, no other verification performed No software changes needed for this approach
Increment 1 – First success
© Sioux Embedded Systems | Confidential | 8
Increment 1 – First success
© Sioux Embedded Systems | Confidential | 9
Increment 1 – First success
Simple, but quick first results Multiple reliability issues found Work to do for the developers
© Sioux Embedded Systems | Confidential | 10
System Under Test
Hardware Abstraction Layer (LabVIEW)
Input (Hardware)
Test cases (LabVIEW)
Control
Output Log file
Test Framework 1st Increment
Several defects found were already known: Customer issues Not reproducible -> no solution Now: developers could work on them And fix could be tested as well
Several presentations given explaining the approach
And… get clear what we are looking for!
Primary functions should work reliable
Management buy-in
© Sioux Embedded Systems | Confidential | 11
Definition of Reliability Hit
© Sioux Embedded Systems | Confidential | 12
PF = Primary FunctionPrimary function: e.g. startup, acquisitionNon-primary function: e.g. printing, post-viewing
LabVIEW is not that easy Provide general scripting language (Ruby) Ruby interfacing with LabVIEW via abstraction layer
Development of test libraries was started Still only control, no verification Log file analysis after test (tools were used)
Increment 2A language for the testers
© Sioux Embedded Systems | Confidential | 13
Increment 2A language for the testers
© Sioux Embedded Systems | Confidential | 14
System Under Test
Hardware Abstraction Layer (LabVIEW)
Input (Hardware)
Test cases + libraries (Ruby)
Control
Output Log file
Test Framework 2nd Increment
Logfile scanned during test case execution Determine pass/fail criteria Detect system states and act upon:
Hot generator extensive acquisition not possible Execute other test cases (e.g. power-cycle), until Generator has cooled down
Log file analysis after test was still performed Still no software changes in the SUT, but existing
interfaces were available now
Increment 3Logfile interpretation
© Sioux Embedded Systems | Confidential | 15
Increment 3Logfile interpretation
© Sioux Embedded Systems | Confidential | 16
System Under Test
Hardware Abstraction Layer (LabVIEW)
Input (hardware & software)
Test Execution Environment incl. test cases and library
(Ruby)
Control
Rep
osito
ry
(Tes
t cas
es +
Res
ults
)
Test Framework 3rd Increment
Output
Result
Test Scheduler (Ruby)
Best practise: Test actions by external interfaces Test verification by log file and internal state
information System statistics extracted from logfile:
Number of startups (succeeded and failed) Number of acquisitions (succeeded and failed)
Increment 3Logfile interpretation
© Sioux Embedded Systems | Confidential | 17
Reliability hits could be identified from logfile (semi-automatic)
Pareto charts
Performance measurements(timing info in logfile)
Crow-AMSAA
Statistics
© Sioux Embedded Systems | Confidential | 18
Crow-AMSAA Failure PlotExample
© Sioux Embedded Systems | Confidential | 19
Cum. Number of startups
Cum. Number of failed startups
Individual testruns
LogLog scale
Crow-AMSAA Failure PlotExample
© Sioux Embedded Systems | Confidential | 20
100 extra startups
10 failures
Best fit
Crow-AMSAA MTBF PlotExample
© Sioux Embedded Systems | Confidential | 21
Monitor trendMake predictions
MeanTime
BetweenFailures
>100 reliability hits identified Which ones would have slipped through other tests? Which ones would the customer complain about?
“Independent” analysis of hits: 8 would have been in system test, but not earlier 7 would not have been found, but customer would
complain (and fix would be necessary)
ROI
© Sioux Embedded Systems | Confidential | 22
ROI:(8 x X1) + (7 x X2) – costs > 0
Costs (man-hours + material) = 200K Euro X1: costs of defect found in system test: 10K Euro X2: costs of field defect: 200K Euro
80K + 1.4M – 200K 1.2M Euro saved
More money and time became available…Implementing/executing more testsMore projects/products
ROI
© Sioux Embedded Systems | Confidential | 23
© Sioux Embedded Systems | Confidential | 24
Results
Numerous reliability hits identified + solved MTBF measured and predicted Startup MTBF increased by factor 7.6 Acquisition MTBF incr. by factor 18 More testing hours on systems Customer satisfaction More projects wanted this approach Only 5 system test cycles remaining (was 15)
24
© Sioux Embedded Systems | Confidential | 25
Key success factors
Choose right project at the right time Incremental development (early visible
benefit) Communication / ROI Clear and simple reporting (Crow-AMSAA) Hardware interfaces
Low probe effect (not a single false alarm)Easy ported to different products
25
© Sioux Embedded Systems | Confidential | 26
Questions
26
This case study is described in detail:Dorothy Graham & Mark FewsterExperiences of Test Automation
Case studies of software test automation
ISBN 0321754069
[email protected]+31 (0)40 26 77 100