400 Commonwealth Drive, Warrendale, PA 15096-0001 U.S.A. Tel: (724) 776-4841 Fax: (724) 776-5760 Web: www.sae.org

SAE TECHNICALPAPER SERIES 2006-01-1600

Application of Robust Engineering Methodsto Improve ECU Software Testing

Eldon G. Leaphart, Steve E. Muldoon and Jill N. IrlbeckDelphi Corporation

2006 SAE World CongressDetroit, Michigan

April 3-6, 2006

The Engineering Meetings Board has approved this paper for publication. It has successfully completed SAE's peer review process under the supervision of the session organizer. This process requires a minimum of three (3) reviews by industry experts.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior written permission of SAE.

For permission and licensing requests contact:

SAE Permissions400 Commonwealth DriveWarrendale, PA 15096-0001-USAEmail: permissions@sae.orgTel: 724-772-4028Fax: 724-776-3036

For multiple print copies contact:

SAE Customer ServiceTel: 877-606-7323 (inside USA and Canada)Tel: 724-776-4970 (outside USA)Fax: 724-776-0790Email: CustomerService@sae.org

ISSN 0148-7191Copyright 2006 SAE International

Positions and opinions advanced in this paper are those of the author(s) and not necessarily those of SAE. The author is solely responsible for the content of the paper. A process is available by which discussions will be printed with the paper if it is published in SAE Transactions.

Persons wishing to submit papers to be considered for presentation or publication by SAE should send the manuscript or a 300 word abstract to Secretary, Engineering Meetings Board, SAE.

Printed in USA

1

ABSTRACT

Robust Engineering techniques developed by Taguchi have traditionally applied to the optimization of engineering designs. Robust Engineering methods also may be applied to software testing of ECU algorithms. The net result is an approach capable of improving the software algorithm in one of two ways. First the approach can identify the range of areas which prove problematic to the software such that a robust solution may be developed. Conversely, the approach can be used as a general strategy to verify that the software is robust over the range of inputs tested. The robust engineering methods applied to software testing utilize orthogonal array experiments to test software over a range of inputs. The actual software trials are best performed in the simulation environment and also via automated test hardware in the loop configurations in realtime. This paper outlines a process for applying Robust Engineering methods to software testing. Experiments derived from the robust methodologies are then implemented using both software simulation and hardware in the loop environments. A case study is presented where the process is applied to improve the robustness of an ECU software algorithm. Results are obtained from both using simulation and performing automated hardware in the loop tests.

INTRODUCTION

Robust Engineering techniques are typically applied to optimize of engineering designs. Concepts including parameter/control factor optimization experiments, two step optimization analysis, prediction and confirmation tests, etc. have widely been used to improve robustness of hardware/mechanical, software, and process designs. In addition to using robust techniques throughout the design phase, the test phase of a product can also benefit from the application of robust methods. Using robust design principles specifically for software test requires tailoring some of these methods due to unique constraints of software. Consequently these differences between traditional robust design processes vs.

software test methods using robust engineering are described. The orthogonal array test approach identifies noise factors for a specific algorithm anomaly condition as opposed to the typical usage with assigned control factors. The noise space or range of noise factor values is identified through a set of iterative tests. Thus, an algorithm solution may be formulated which is robust over the range of noise factors that may be present as opposed to a single anomaly condition. The overall goal of using robust methods during the design phase is to develop a design robust to external conditions early in the design phase. Given that testing always lags completion of a given design, the goal of using the robust methods for ECU software testing is to more thoroughly test the design, thus preventing anomalies from surfacing during actual field operation.

SOFTWARE DEVELOPMENT CYCLE

As a background it is useful to review the traditional “V” software development cycle to indicate how it relates to robust engineering methodologies. An example of the “V” diagram is shown in Figure 1. The left side of the “V” diagram describes the process of requirements capture along with the various design phases of a project. The step of implementing the desired design is shown at the bottom of the diagram, and testing phases are shown on the right side of the diagram. Once the design requirements are defined, implementation is performed followed by testing phases to show that the completed implementation meets the original requirements. Iteration from the testing phase back to design and subsequent implementation may be necessary when test results yield problems or show that requirements are not met.

During the design phase of a project, robust engineering principles may be applied to develop the design and show that performance is robust across chosen noise factors. This implies that software control factors including logic constructs, performance gains, calibrations, etc. are identified and varied to achieve the proper output response. Output measurements are performed such that signal / noise strategies are developed, calculated, and optimized.

2006-01-1600

Application of Robust Engineering Methods to Improve ECU Software Testing

Eldon G. Leaphart, Steve E. Muldoon and Jill N. Irlbeck Delphi Corporation

Copyright © 2006 SAE International

2

The software test phase presents some unique characteristics that must be considered prior to applying robust engineering principles. Three characteristics that are discussed here are the following:

• Software content is constant • Software performance is quantified as Pass/Fail • Software test effectiveness is a function of test input The first characteristic to consider is that software content either during test or over the lifetime of the product is constant. Ideally, software is not subject to noise factors such as environment, aging, and manufacturing. As such, during the testing phase, parameters of the software are not viewed as factors that may be modified.

The second characteristic to consider is that of measuring software performance. Although there may be several outputs which can be observed and measured, the software test phase only considers that the function under test passed or failed; e.g. requirements are met or not met. This implies that for testing, signal to noise robust strategies are computed as a function of number of failures observed rather than measurement of an output variable from the process.

Finally, the third characteristic to consider for this discussion is that software testing effectiveness is a function of test inputs. Software is typically designed to perform a specific task or exhibit a specific response for a given a set of input conditions. Software is often tested to insure that the desired functionality performs as intended. A more difficult task is to show that the software under test does not perform certain tasks incorrectly for any possible set of inputs or operating conditions. Software can be tested exhaustively with a narrow set of inputs and shown to pass, only to fail or exhibit some undesirable behavior once in the field. This is because the field usage produces some sequence or input characteristic which was not expected. Thus, some prior knowledge may be required to identify conditions that the software is vulnerable to and thus can be tested against. Knowledge of how the software will be used in the field is also extremely useful for identifying test inputs. Once these conditions are known, subsequent tests may be developed. In summary some analysis of the desired functionality, knowledge of possible problems and typical usage are recommended to serve as starting point for robust software test development.

Recognizing that software testing involves these unique characteristics, minor modifications to the traditional robust engineering design methods are proposed. The next section outlines this modified process.

DESIGNPHASE

TESTPHASE

RobustDesignMethods

RobustTest

MethodsIMPLEMENTATION

PHASE

Figure 1: Software Development “V” Cycle

ROBUST ENGINEERING SOFTWARE TEST PROCESS

The traditional Robust Engineering design process may be outlined in nine general steps. These steps are the following:

1. Define scope of the project 2. Define boundary of subsystem 3. Define ideal function with input M, output y

identified 4. Develop signal and noise strategies 5. Define control factors and levels 6. Formulate experiment 7. Conduct experiment / data collection 8. Data Analysis (S/N ratios, sensitivity, response

tables, 2 step optimization, predictions) 9. Perform confirmation experiments and

document For software test, some of the traditional design steps are slightly modified to consider the software characteristics described in the previous section. The modified steps include definition of the basic or “ideal” function, a modified P-Diagram, formulation of experiment using the Orthogonal Test Array, iteration step and test evaluation.

BASIC FUNCTION

The first item to note differences between design and test is the ideal function. The ideal function is defined as the ideal relationship between input signal M and output response y based on the physics of the system under study. The output response is described as linear with the gain beta (β) assigned. With an expression defined as y=βM, the output response is measurable in objective

3

quantities, thus signal / noise ratios may be obtained. A traditional ideal model is shown in Figure 2 below.

y

M

β

Figure 2: Traditional Ideal Function

Software testing requires an ideal function representation which takes into consideration an output response y which is either pass or fail. The resulting relationship between y and M is nonlinear and a function of software logic designed to meet a requirement. Figure 3 shows an example of the software test ideal function.

y

M

pass

fail

Figure 3: Software Test Ideal Function

MODIFIED P – DIAGRAM

A Parameter or P Diagram approach is used to describe the relationship between input signal, output response, control factors and noise factors for an engineering system. A general example of a P-Diagram is shown below in Figure 4.

Figure 4: Parameter (P) - Diagram

For software test purposes, Noise and Control factors are viewed differently and result in a modified P – Diagram. Reiterating, during the test phase and field operation software content is constant. Thus control factors are not considered as variables other than possibly operating modes of the software. Also, in the ideal sense, software is not subject to the typical noise factors of environment, aging, and manufacturing. It should be that noise factors still apply, however, they may be modeled as disturbances which add directly to the input signal M. These noise disturbances contribute to undesirable behavior in some cases. The goal of robust engineering applied to the software test phase is to identify areas where the software is not robust to a set of noise factors affecting the input, or conversely, show that for a given range of input signal noise disturbances, that the software exhibits no undesirable output. Figure 5 illustrates a Modified P-Diagram for the software testing phase.

Figure 5: Modified P-Diagram for Software Testing

Part,Process,Function

Minput

youtput

Noise Factorsenvironment

agingmanufacturing

Control FactorsPart, Process,

Functionparameters

SoftwareFunction

Minput

youtputΣ

Noise Factorsfrequencymagnitude

timingsequences

etc

Control Factorsprogram modes

4

ORTHOGONAL ARRAY EXPERIMENT SETUP

The experiment setup for both robust engineering design phases and software test phases utilize the orthogonal array. An orthogonal array provides for a balanced set of experiment trials. This allows for conclusions to be reached in a balanced manner. Typically, control factors identified from the P-diagram are assigned to the orthogonal array for experimentation. Software testing requires that the noise factors identified from the Modified P-Diagram be assigned to the orthogonal array.

The more formidable task with software test experimental setup is determining what levels of noise factors to begin with to perform the experiment. The process of identifying the noise factor levels is paramount to exploring the effects of the noise space against software performance. Two approaches are suggested as part of this discussion. Iteration around conditions producing a known or prior software anomaly is a more direct approach. A second approach is to identify a range of noise factors based on general observation of characteristics of the software objective under test. Both approaches require iteration to either hone in on the range of noise factors that is problematic, or adequately confirm that the software performs adequately across all noise factors.

The iteration process for the first method where a known problem has been observed and analyzed is detailed with the following steps.

1. Identify noise factor parameters based on analysis of known problematic input.

2. Define for the first row of the Orthogonal Array with noise factor levels which will reproduce the software anomaly. The measured response should indicate a fail condition for this trial.

3. Perform remaining trials of Orthogonal Array covering all combinations as defined by the size and type of the array.

4. Based on results from Orthogonal Array Test change range of noise factor parameters to bound the problem area. This could either require expanding the noise factor range around the initial levels or concentrating or a tighter range.

5. Repeat steps 3 & 4 and tabulate experiment results. Successive iterations should indicate where noise factor levels affect the intended software performance.

Based on the orthogonal array test results, the software may be declared robust across the range of noise factors identified or conversely be subject to redesign to address areas which produced failures. In the case of a

software redesign, all orthogonal array iterations may be re-run to serve as a confirmation that software changes improve performance in the areas of concern. Most of the steps outlined in the iteration process also apply for the second approach where noise factor levels are selected from general observations of the software scope. The exception is that iteration step 2 is not required since this method assumes that a specific issue has not been identified from which to start from.

TEST EVALUATION

The final step with the robust engineering software test process is to quantify the experimental test results for the orthogonal array trials. For traditional robust design projects, signal to noise ratio and sensitivity calculations are computed from the measured output responses of each experimental trial. For the robust engineering software testing process, the measured results are in the discrete form of pass / fail or 0 / 1.

Two approaches are available to quantify experimental results in the discrete form, two way interaction response tables and signal to noise ratio computed as a function of number of failures. The response table approach requires that the number of failures for each 2 factor interaction of the orthogonal array test i.e. (Ai Bj) be summed. The sum total for a two way interaction representing 100% failure is defined by the expression:

Size of Orthogonal Array / # of combinations of (Ai Bj)

Metrics associated with the effectiveness of the software testing may be obtained by totaling the number of 100% two way interaction failures, or simply tabulating total numbers of failures for the orthogonal array. An example template of a response table for an L8(27) orthogonal array test is shown below in Figure 6.

Figure 6: Response Table for 2 Way Interactions

A1 A2B1B2 Record data in

B1 B2 terms of % failureC1C2

C1 C2D1D2

D1 D2E1E2

E1 E2F1F2

F1 F2G1G2

5

An alternate method for quantifying discrete orthogonal array tests using signal to noise ratio calculations is described in [3]. In this approach, the total number of errors vs. array size is computed. A signal to noise gain is computed from the expression shown as:

S/N = -20 Log10[(Total Defects)/n + 1]

This expression will yield a S/N ratio of 0 db for a test without error. Tests which produce errors will result in a negative S/N ration gain. This type of response is typical of a “maximum the better” or “fewer errors the better” experiment design. Either signal to noise ratio or response table calculations can be used to quantify software test results during iteration steps or to show improvement between redesign confirmation trials.

PROCESS SUMMARY

In summary, a process for Robust Engineering Software Test is defined. The process is based on Robust Engineering design steps, but differ slightly in their application due to the unique characteristics of software. Careful analysis of the software function under test is recommended to select a set of noise parameters which will be most beneficial for providing desired test coverage. This process may require iteration if prior knowledge of potential issues with the software / design are less known. The advantage of using this method for software test is to show software compliance or robustness over a wide range of inputs as opposed to testing at a single operating point. The Robust Engineering Software Test process is summarized by the following steps:

1. Define scope of the project 2. Define boundary of subsystem 3. Define Pass / Fail function and Modified P -

Diagram 4. Analyze input signal and input timing sequence

characteristics for project and define noise factor

5. Determine initial noise factor levels 6. Formulate Orthogonal Array experiment 7. Conduct experiment / data collection via

simulation or Hardware In Loop configurations 8. Data Analysis (2 way interaction response

tables, S/N ratios, predictions) 9. Iterate steps 7 & 8 with modified noise factors 10. Perform re-design phase if necessary; address

issues identified across noise space 11. Perform confirmation run; Repeat experiments

with same iterations of noise factors to compute improvements.

The next section of this paper further details methods to implement and automate the orthogonal array

experiment phase for software testing. Two methods used are pure software simulation and hardware in the loop tests.

SOFTWARE SIMULATION TEST ENVIRONMENT

Simulation of software can be defined as the capability to model and execute logic constructs representative of realtime algorithms or processes. The simulation is performed in processing environments that may differ from the intended target application ie. microprocessor, programming language, execution speed, etc. however, the calculation results should be equivalent. A simulation environment is ideal for developing test stimuli and evaluation criteria per robust engineering methods for software test. Matlab™ / SIMULINK™ simulation tools (from The MathWorks Inc.) are one such tool used to simulate and test software.

SIMULINK SIMULATION

Matlab / SIMULINK is a flexible computational and simulation tool capable of modeling a variety of physical or software processes. Software can be modeled by constructing graphical building blocks to implement an intended function or by executing compiled C language source code within a mechanism referred to as an s-Function. The method of using SIMULINK s-Functions to implement compiled production C source code is thoroughly discussed in [6].

The SIMULINK environment allows for software modules or programs under test to be embedded into a single S-function block. Additional blocks may be added to produce input signals and monitor output for the software under test. Figure 7 is shows a model layout which has been constructed for test development

Figure 7 : SIMULINK Test Model Layout

ORTHOGONAL ARRAY SOFTWARE TEST DEVELOPMENT

Three elements required for simulation of an orthogonal array software test include input data sequences,

6

software under test, and the evaluation criteria. The process for robust engineering software testing requires that noise factor combinations be constructed as input to the software under test. These input data characteristics such as frequency, magnitude, timing, sequence, etc can be easily modeled with the available SIMULINK constructs. Data to be used as test stimuli may be created given a description of desired characteristics or requirements. Alternatively, actual field data may be imported and applied to the software. The set of data inputs can be easily arranged or scheduled per the sequence as specified by the orthogonal array.

The next element required for the simulation of the orthogonal array software test is the actual software subject to test. SIMULINK offers flexibility in this area relative to the representation of this software. The software may be modeled using the graphical blocks or conversely the s-Functions using compiled code may be used. The s-Function approach has advantages in that the source code for the target application is equivalent for a large extent to that executed in the model. This eliminates possible error which may be introduced as a function of reproducing the code with the graphical block approach and may be more time efficient to develop a completed model.

Lastly, data which propagates from the input blocks through the software s-Function to produce a desired result must be evaluated per a pass / fail criteria. Evaluation of this data output could be performed by visual inspection or analysis, however, it is much more efficient to develop a routine to evaluate the pass fail criteria automatically. Orthogonal array testing requires several successive trials to be conducted depending on the matrix size. A devised routine to evaluate pass / fail results, may be scripted to run as a part of each trial simulation.

With the three elements of data input, software, and evaluation criteria modeled, each orthogonal array trial may be run individually, or a script sequence may be developed which performs all orthogonal array trials in successively for a given matrix in simulation. The simulation environment serves as a test bed to develop test requirements and confirm results. The components developed in the simulation environment may be ported to the HIL environment to complete testing of the software with the target hardware.

HARDWARE IN LOOP TEST SIMULATION

PI AUTOSIM CAPABILITY

The Pi AutoSim is a bench-top simulator developed by Pi Technology Corp designed specifically for testing automotive electronic control units (ECUs) in a hardware-in-the-loop (HIL) environment. The simulator can be configured to be used in an open or closed loop

modeling mode. Simulation models can be created using the C/C++ programming language or imported from standard math tools like Matlab / SIMULINK. The simulation model is typically run in real time, executing the defined parameters and characteristics required for the targeted ECU environment, while the AutoSim input and output hardware (designed for common automotive type signals (analog, digital, PWM, and serial communications), complete the simulator to ECU input / output interface. A powerful and useful feature of the AutoSim in a testing or development environment is the ability to run user defined test scripts. These test scripts are typically created using C language and compiled for direct execution on the AutoSim platform. The test scripts allow complex combinations of input and output signals, as well as serial communications messages (i.e. CAN, J1850, etc.) to be sequenced into the simulation in a controlled and repeatable manner.

HIL SETUP DESCRIPTION

The HIL simulation set-up consists of the Pi AutoSim simulator chassis unit with I/O card complement, a “host” or user interface PC connected to the AutoSim, an automated signal line break-out box (AutoBOB), and a target ECU with wire harness. The AutoSim I/O and ECU are powered from an external DC power supply. The DC supply provides internal power for the AutoSim I/O cards, as well as a source for controlled DC power for the ECU. The host PC is connected to the internal AutoSim single board PC computer via ethernet. The host PC provides the user with a graphical user interface (GUI) PC software application from which the simulation modeling, testing script, and I/O configurations of the AutoSim are controlled. The ECU must be connected to the AutoSim through a specifically designed wiring harness, which matches the ECU signal lines to the required I/O card connections on the AutoSim chassis. The signal line wiring harness is routed through an automated break-out box or “AutoBOB” which is placed between the ECU and the AutoSim I/O. The AutoBoB is a model or script controlled device capable of providing DC power bus shorting and open circuit conditions for the signal lines. Serial communications interface devices and ECU specific instrumentation are also accommodated as needed or required. Data results from executed test scripts are generally stored in a text file format. Figure 8 shows the HIL and Pi AutoSim hardware setup.

7

Figure 8: Pi AutoSim Hardware in the Loop Setup

REAL-TIME AUTOMATION OF ORTHOGONAL ARRAY TEST

The Pi AutoSim is ideally suited for performing the multiple, controlled iterations of the orthogonal array test process. In this application of the orthogonal array testing, the focus is the performance of a specific algorithm function performed by the ECU application software. By using the test scripting capability, a series of varying input factors can be introduced to the ECU software in a systematic manner. The output response of the software may be monitored, and evaluated in real time. A general approach to automating the software orthogonal array test using the AutoSim capability is described in the following steps and illustrated in Figure 9:

1. Perform Robust Engineering Software Test process to formulate orthogonal array test parameters.

2. Identify the pass / fail criteria for the sub-system function being tested in terms of measurable simulation output signals.

3. Implement the orthogonal array test within the software simulation environment and iterate as necessary to obtain initial confirmation results.

4. Port input test scripts, data, requirements, etc and pass / fail criteria routines from software simulation environment to HIL environment

5. Design the AutoSim test script logic flow in a manner that automatically cycles through the input trials in defined by the orthogonal array, covering all array combinations.

PROBLEMANALYSIS ROBUST

ENGINEERING

ORTHOGONAL ARRAY TEST DESIGN

L16 (45) Orthogonal Array Test

A B C D EDetermine Pass Fail Criteria

No.

steering signal freq / amplitude

initial glitch voltage

glitch voltage magnitude

glitch duration cal

(loops)Slope

Actual Result 1 = Fail

1 1Hz +/- 100 seg 3.2 0.6 2 0 (neg)

2 1Hz +/- 100 seg 2.8 0.65 3 1 (pos)

3 1Hz +/- 100 seg 2.4 0.8 4 0 (neg)

4 1Hz +/- 100 seg 1.8 1 1 1 (pos)

5 1.5Hz +/- 100 seg 3.2 0.65 4 1

6 1.5Hz +/- 100 seg 2.8 0.6 1 0

7 1.5Hz +/- 100 seg 2.4 1 2 1

8 1.5Hz +/- 100 seg 1.8 0.8 3 0

9 0.5Hz +/- 100 seg 3.2 0.8 1 1

10 0.5Hz +/- 100 seg 2.8 1 4 0

11 0.5Hz +/- 100 seg 2.4 0.6 3 1

12 0.5Hz +/- 100 seg 1.8 0.65 2 0

13 2.0Hz +/- 100 seg 3.2 1 3 0

14 2.0Hz +/- 100 seg 2.8 0.8 2 1

15 2.0Hz +/- 100 seg 2.4 0.65 1 0

16 2.0Hz +/- 100 seg 1.8 0.6 4 1

SIMULINK ENVIRONMENT

CREATE DATA INPUTS

VERIFY EXPECTED RESPONSE Mdl.ico

AUTOSIM ENVIRONMENT

TEST SCRIPT DESIGN

AUTOMATED ITERITATIONS Net04.ico

HIL ENVIRONMENT

ORTHOGONAL ARRAY TEST IMPLEMENTATIONNet04.ico

L16 (45) Orthogonal Array TestA B C D E

Determine Pass Fail Criteria

No.

steering signal freq / amplitude

initial glitch voltage

glitch voltage magnitude

glitch duration cal

(loops)Slope

Actual Result 1 = Fail

1 1Hz +/- 100 seg 3.2 0.6 2 0 (neg)

2 1Hz +/- 100 seg 2.8 0.65 3 1 (pos)

3 1Hz +/- 100 seg 2.4 0.8 4 0 (neg)

4 1Hz +/- 100 seg 1.8 1 1 1 (pos)

5 1.5Hz +/- 100 seg 3.2 0.65 4 1

6 1.5Hz +/- 100 seg 2.8 0.6 1 0

7 1.5Hz +/- 100 seg 2.4 1 2 1

8 1.5Hz +/- 100 seg 1.8 0.8 3 0

9 0.5Hz +/- 100 seg 3.2 0.8 1 1

10 0.5Hz +/- 100 seg 2.8 1 4 0

11 0.5Hz +/- 100 seg 2.4 0.6 3 1

12 0.5Hz +/- 100 seg 1.8 0.65 2 0

13 2.0Hz +/- 100 seg 3.2 1 3 0

14 2.0Hz +/- 100 seg 2.8 0.8 2 1

15 2.0Hz +/- 100 seg 2.4 0.65 1 0

16 2.0Hz +/- 100 seg 1.8 0.6 4 1

PASS / FAILVERIFICATION

REPORT

Figure 9 : Robust Engineering S/W Test Process

CASE STUDY: AUTOMOTIVE SENSOR PROCESSING

The robust engineering software test process was applied to testing software for an automotive sensor processing application. The technique proved effective in highlighting the range of conditions which would prove problematic to the algorithm. The benefit is that a robust solution can be developed to address the entire range of disturbance type inputs as opposed to only implementing a solution for one specific operation condition.

GENERAL OVERVIEW

An automotive angular position sensor is comprised of multiple outputs which must be processed by an Electronic Control Unit (ECU) controller to determine a single output signal proportional to angular position. Due to the fact that the single output is a function of the

8

multiple inputs, any disturbance on the inputs has the capability of perturbing the output signal. Attempts have been made to minimize the effect of the input disturbances through filtering, rate limiting, etc., however, this has proven to only be partially effective for certain inputs. The robust engineering software test process was applied to test the algorithm over a wide range of inputs. The results indicate where improvements in the algorithm could be made to yield more robust performance.

BASIC FUNCTION / MODIFIED P - DIAGRAM

The scope of the investigation is limited to the ECU software responsible for processing and converting the sensor voltage outputs into a software variable proportional to angular position. An undesirable response was defined as a disturbance to the input of the ECU which results in a persistent offset in the angular position output.

A modified P – Diagram of the problem is shown below in Figure 10. Disturbance characteristics for the input signal (output from sensor), signal frequency, signal magnitude, disturbance voltage, disturbance magnitude and duration are identified with the P – Diagram.

yangularposition

Msensor

voltage inputs(n)

Sensor ProcessingSoftwareFunction

Σ

Noise Factorsinput signal frequency (rate)

input signal magnitudesensor input signal with disturbance

disturbance initial voltagedisturbance magnitude (voltage)

disturbance duration (loops)

Figure 10: Modified P – Diagram Sensor Processing

Figure 11 provides an illustration of how these disturbances affect the input signal. The slope of the signal with respect to time is proportional to the input frequency to the sensor. A disturbance can be viewed as a specific type of noise superimposed upon the input signal to the controller. The noise has characteristics of magnitude, duration, and an initial magnitude relative to the range of the signal. These characteristics are identified as noise factors and will be used as input to the Orthogonal Array tests.

Vol

tage

time

slope of signalproportional to

input freq or rate

disturbanceinitial voltage

disturbancemagnitude

disturbanceduration

SINGLE SENSOR VOLTAGE OUTPUTWITH SIMULATED DISTURBANCE

Figure 11 : Signal Disturbance Characteristics

ORTHOGONAL ARRAY EXPERIMENT SETUP

An L16(45) Orthogonal Array was chosen to setup the experiment test. This size matrix allows for 5 factors with up to 4 levels each. The factors of signal frequency, signal magnitude, disturbance initial voltage, disturbance magnitude, disturbance duration, and disturbance slope were assigned as factors to the orthogonal array. Dummy treatment was used for Factor E (disturbance slope) as only 2 levels 0 and 1 were applicable. For the 1st iteration, Level 1 values were derived from sample input data recorded from the ECU wherein the disturbance behavior had occurred. The remaining levels were selected as to provide a range around this single operating point which posed a problem. By setting up the orthogonal array in this manner, the 1st row experiment is trial is always expected to fail with the original ECU software. The remaining 15 additional experiment runs will reveal other interactions which also may cause the ECU software to fail. Table 1 below shows the table for the 1st iteration.

Table 1 : 1st Iteration Factor / Level Assignment

input signal freq / amplitude

initial disturbance

voltage

disturbance voltage

magnitude

disturbance duration cal

(mSec)disturbance slope

Level A B C D E1 1Hz +/- 100 deg 3.2 0.6 2 0 (neg)2 1.5Hz +/- 100 deg 2.8 0.65 3 1 (pos)3 0.5Hz +/- 100 deg 2.4 0.8 4 0 (neg)4 2.0Hz +/- 100 deg 1.8 1 1 1 (pos)

Factors

9

The factor assignments for the full L16(45) matrix are shown in Table 2. A SIMULINK s-function model was created to implement the ECU software. The various inputs specific to the factor levels identified were modeled in the SIMULINK environment and subjected to the ECU software routine under test.

L16 (45) Orthogonal Array Test

A B C D EDetermine Pass Fail Criteria

No.

input signal freq / amplitude

initial disturbance

voltage

disturbance voltage

magnitude

disturbance duration cal

(mSec)Slope Actual Result

1 = Fail

1 1Hz +/- 100 deg 3.2 0.6 2 0 (neg)

2 1Hz +/- 100 deg 2.8 0.65 3 1 (pos)

3 1Hz +/- 100 deg 2.4 0.8 4 0 (neg)

4 1Hz +/- 100 deg 1.8 1 1 1 (pos)

5 1.5Hz +/- 100 deg 3.2 0.65 4 1

6 1.5Hz +/- 100 deg 2.8 0.6 1 0

7 1.5Hz +/- 100 deg 2.4 1 2 1

8 1.5Hz +/- 100 deg 1.8 0.8 3 0

9 0.5Hz +/- 100 deg 3.2 0.8 1 1

10 0.5Hz +/- 100 deg 2.8 1 4 0

11 0.5Hz +/- 100 deg 2.4 0.6 3 1

12 0.5Hz +/- 100 deg 1.8 0.65 2 0

13 2.0Hz +/- 100 deg 3.2 1 3 0

14 2.0Hz +/- 100 deg 2.8 0.8 2 1

15 2.0Hz +/- 100 deg 2.4 0.65 1 0

16 2.0Hz +/- 100 deg 1.8 0.6 4 1

Table 2 : 1st Iteration Experiment Test

REALTIME HARDWARE IN LOOP EXPERIMENT SETUP

The setup for the HIL experiment consisted of the Pi Autosim, and utilized a controlled braking ECU and software (also shown in Figure 8). Input data files generated from the SIMULINK simulation of the orthogonal array test used to test the ECU software were ported to the AutoSim environment. The input data signals were generated / captured in a time stamped text data file. Through software mapping of the AutoSim analogue card outputs to the “command” data contained in the text data files, reproduction of recorded sensor was performed. The ability to utilize this external text data as real time variables in the simulation required additional AutoSim model and script functions to be developed by the testing team. The

output of the ECU software was also monitored using the Autosim. The test ECU has the ability to provide direct memory access to software data variables using the CAN Calibration Protocol (CCP). By monitoring and requesting specific memory location reads via the ECU CCP bus, internal software variables were captured in real time response to the corresponding controlled input data signals being generated by the text data files. The pass / fail criteria developed in the SIMULINK environment was also scripted within the AutoSim environment to automatically indicate the outcome of each trial. The resulting configuration allowed for a very repeatable, deterministic execution of the orthogonal array test matrix.

ITERATION TEST RESULTS

Three iterations were necessary to successfully bound the input range of factor conditions which indicated problem areas with the existing ECU software. Following the 1st trial, the factor identified as disturbance slope was removed and replaced with input signal frequency. This allowed the input signal frequency and amplitude to be set independently. For the 2nd iteration the range of amplitude was selected over a broader range from 25 up to 100 Hz. This selection revealed that lower magnitudes below 75 Hz did not appear to cause problems for the ECU software. A final range was chosen which concentrated input amplitudes around 100 deg combined with frequencies above 1Hz. Experimental results indicated interactions between the disturbance criteria at higher frequencies and signal magnitudes that yielded undesirable results.

With the problem successfully bounded, an alternative ECU software solution was developed and re-tested using the factors and levels that were used for the three iterations in the original tests. Response tables and signal to noise ratios were calculated based on the number of experiment trial failures. This metric was used to quantify the difference of performance between trials, and also the amount of improvement from the original design to the new design. An average 1.95 db S/N gain was observed between the two ECU designs. The final trial experiment with the redesigned ECU software still indicated a problem interaction with large magnitudes above 2 Hz. However, this condition was evaluated to be possible in simulation only, and outside the practical operating constraints for the application.

Table 3 provides these metrics for the various iterations for both designs of ECU software. Complete results for a sample iteration test are provided in the Appendix.

REALTIME HARDWARE IN LOOP TEST RESULTS

The various orthogonal array trials were performed using the HIL setup for iterations which illustrated the root issues with the ECU software as well as confirmation

10

iterations with improved ECU software. The results from the software simulation step correlated with those from the HIL test results. The total runtime for each L16(45) matrix iteration for the HIL setups takes approximately 3 min., and time for set-up and completion of 1 complete test matrix takes approximately 48 minutes. The development of the automated HIL test script becomes a check that can easily be applied to software packages as part of the final validation or detail software regression testing process.

Table 3 : Experimental Results

CONCLUSION

The application of robust engineering methodologies for ECU software testing provides a tool which allows for a more thorough investigation of possible interactions that may adversely affect software results. For the cases where a problem has been observed with existing ECU software, the approach may be applied to identify the range or noise factor range which must be addressed. This approach should help to prevent the undesirable cycle of creating and implementing solutions only to discover another undesirable scenario later during development or field operation. The method defines test

requirements that may be implemented in a software simulation environment and again confirmed with actual HIL testing of the product ECU. The approach serves a as a very repeatable, disciplined method to enhance the level of ECU software verification.

REFERENCES

1. ASI Consulting Group. Robust Engineering Practioner’s Course Class Notes. January 2004.

2. Deguchi, J, et. al. Efficient Tool for Debugging with the Orthogonal Array. R.E. for Software Application Course Handout. January 2004.

3. Goldstein, S. Robust Testing of EW Systems. R.E. for Software Application Course Handout. January 2004.

4. Taguchi, G., et. al.. Robust Engineering. McGraw Hill 2000.

5. Musa, J. Software Reliability Engineering. McGraw Hill 1999.

6. Wang, R., et. al. Unified Chassis Simulation Architecture Delphi TechCon February 2004

7. Schuette, H., et. al. Hardware-in-the-LoopTesting of Vehicle Dynamic Controllers – A Technical Survey SAE 2005-01-1660

CONTACT

Eldon G. Leaphart, Engineering Manager – Diagnostics, Communications & System Software / Controlled Brakes Delphi Corp., 12501 E. Grand River, MC 483-3DB-210, Brighton, MI, 48116-8326 Phone: (810)-494-4767, Fax:(810)494-4458 email: eldon.g.leaphart@delphi.com

L16 (45) Orthogonal Array Test Response Table Summary

Test Description

Interaction Failures

(initial design)

S/N gain (db)(initial design)

Interaction Failures

(re-design)

S/N gain (db)(re-design)

S/N improvement

Iteration #1 35 -2.3620 10 -0.5266 1.8354

Iteration #2 38 -1.9382 10 -0.5266 1.4116

Iteration #3 71 -4.5449 40 -1.9382 2.6067

11

APPENDIX ITERATION # 1: ORIGINAL DESIGN

input signal freq / amplitude

initial disturbance

voltage

disturbance voltage

magnitude

disturbance duration cal

(mSec)Slope

Level A B C D E1 1Hz +/- 100 deg 3.2 0.6 2 0 (neg)2 1.5Hz +/- 100 deg 2.8 0.65 3 1 (pos)3 0.5Hz +/- 100 deg 2.4 0.8 4 0 (neg)4 2.0Hz +/- 100 deg 1.8 1 1 1 (pos)

Factors

A B C D E

No.

input signal freq / amplitude

initial disturbance

voltage

disturbance voltage

magnitude

disturbance duration cal

(mSec)Slope 1 =Trouble

1 1Hz +/- 100 deg 3.2 0.6 2 0 (neg) 1

2 1Hz +/- 100 deg 2.8 0.65 3 1 (pos) 0

3 1Hz +/- 100 deg 2.4 0.8 4 0 (neg) 1

4 1Hz +/- 100 deg 1.8 1 1 1 (pos) 0

5 1.5Hz +/- 100 deg 3.2 0.65 4 1 0

6 1.5Hz +/- 100 deg 2.8 0.6 1 0 1

7 1.5Hz +/- 100 deg 2.4 1 2 1 0

8 1.5Hz +/- 100 deg 1.8 0.8 3 0 1

9 0.5Hz +/- 100 deg 3.2 0.8 1 1 0

10 0.5Hz +/- 100 deg 2.8 1 4 0 0

11 0.5Hz +/- 100 deg 2.4 0.6 3 0 0

12 0.5Hz +/- 100 deg 1.8 0.65 2 1 0

13 2.0Hz +/- 100 deg 3.2 1 3 0 0

14 2.0Hz +/- 100 deg 2.8 0.8 2 1 0

15 2.0Hz +/- 100 deg 2.4 0.65 1 1 0

16 2.0Hz +/- 100 deg 1.8 0.6 4 0 1

Sum Failures 5

S/N -2.361986242

EXPERIMENT RESPONSE TABLE

A1 A2 A3 A4

B1 1

B2 1

B3 1

B4 1 1

B1 B2 B3 B4

C1 1 1 1 1 1

C2

C3 1 1 1

C4

C1 C2 C3 C4

D1 1

D2 1

D3 1 1 1 1 1 1

D4 1 1

D1 D2 D3 D4

E1 1 1

E2 1 1 1 1

E3 1 1 1 1 1 1

E4

8 8 0 4 0 0 3 3 3 0 2 0 2 0 0 0 0 2 0 35

12

ITERATION #6 WITH REDESIGN (REPEAT ITERATION #1)

input signal freq / amplitude

initial disturbance

voltage

disturbance voltage

magnitude

disturbance duration cal

(mSec)Slope

Level A B C D E1 1Hz +/- 100 deg 3.2 0.6 2 0 (neg)2 1.5Hz +/- 100 deg 2.8 0.65 3 1 (pos)3 0.5Hz +/- 100 deg 2.4 0.8 4 0 (neg)4 2.0Hz +/- 100 deg 1.8 1 1 1 (pos)

Factors

EXPERIMENT RESPONSE TABLE

A1 A2 A3 A4

B1

B2

B3

B4 1

B1 B2 B3 B4

C1 1 1

C2

C3

C4

C1 C2 C3 C4

D1

D2

D3 1 1 1

D4

D1 D2 D3 D4

E1

E2 1 1 1 1

E3

E4

0 0 0 4 0 0 0 0 3 0 2 0 0 0 0 0 0 1 0 10

A B C D E

No.

steering signal freq / amplitude

initial glitch voltage

glitch voltage magnitude

glitch duration cal (loops) Slope 1 =Trouble

1 1Hz +/- 100 deg 3.2 0.6 2 0 (neg) 0

2 1Hz +/- 100 deg 2.8 0.65 3 1 (pos) 0

3 1Hz +/- 100 deg 2.4 0.8 4 0 (neg) 0

4 1Hz +/- 100 deg 1.8 1 1 1 (pos) 0

5 1.5Hz +/- 100 deg 3.2 0.65 4 1 0

6 1.5Hz +/- 100 deg 2.8 0.6 1 0 0

7 1.5Hz +/- 100 deg 2.4 1 2 1 0

8 1.5Hz +/- 100 deg 1.8 0.8 3 0 0

9 0.5Hz +/- 100 deg 3.2 0.8 1 1 0

10 0.5Hz +/- 100 deg 2.8 1 4 0 0

11 0.5Hz +/- 100 deg 2.4 0.6 3 1 0

12 0.5Hz +/- 100 deg 1.8 0.65 2 0 0

13 2.0Hz +/- 100 deg 3.2 1 3 0 0

14 2.0Hz +/- 100 deg 2.8 0.8 2 1 0

15 2.0Hz +/- 100 deg 2.4 0.65 1 0 1

16 2.0Hz +/- 100 deg 1.8 0.6 4 1 0

Sum Failures 1

S/N -0.526578774