Design of Experiments Introduction and

Applications

By: James Hurst

The Testing Process• The planning phase• Select which input variables to change• Select which output variables to measure• Determining the number of test point that can be executed

• Ideally determined by how much information about the system is needed• Realistically determined by the budget/schedule limitations of the program

• Use DOE processes to select which test points to test

The Testing Process (cont)• The execution phase• Adamantly follow the test plan as laid out in the planning phase

• Avoid changing the test plan for “convenience” as this may reduce the benefits of the DOE process

• The reporting phase• Conclude the testing process by properly documenting all results,

conclusions, and any interesting nuances about the system that we discovered during the testing• “A test not reported on is as good as test not done”

Types of Testing• One Factor At a Time (OFAT): Change one input variable at a time in

order to see how the output variable responds to each change• Simple and straightforward but does not show any compounding/negating

effects when two input variables are changed simultaneously• Do what we did last time: When programs are updated or similar

systems are developed, it is tempting to simply test the same way the previous system was tested• Experienced, “Tried and true”• Lessons learned from last test can be directly applied to this test• Information missed last time will be missed this time

• Design of Experiment (DOE): Selection of test points based on statistics and engineering optimization

Design of Experiment (DOE)• The choice of test points to run is rooted in statistics and optimization• This technique is most appropriate for large, complex systems or

systems with stringent budget and schedule restraints• In complex systems it is impossible or impractical to perform every possible

test point. Choosing which to test is crucial• In expensive programs (such as flight testing or weapons testing), the cost is

the driving force in minimizing the amount of testing necessary

• The test must thoroughly assess the system but may have harsh limitations to how much testing can actually be performed• DOE processes are critical to selecting which points to test

DOE Historical Background• Dr. James Lind, 1753: Experiment to cure scurvy• Scurvy was a common problem among the British Royal

Navy while serving long term missions at sea• Dr. Lind took 12 afflicted sailors, split them into groups of two, and gave each

pair a different treatment (including garlic, vinegar, lemons, or oranges)• Within a week those given citrus fruits were fit for duty

• This is the first documented case of a controlled clinical experiment• It is now known that scurvy is caused by a deficiency in vitamin C

(hence why citrus fruits, high in vitamin C, cured it)

DOE Historical Background (cont)• Sir Ronald A. Fisher (1920s-1940s): Father of the DOE process• Conducted agricultural experiments in Great Britain• Varied input parameters on some crops and compared them to control

groups of crops• Randomly assigned which crops would be in a given experimental group and

which would belong to the control group• This allowed him to average out any uncontrollable parameters such as soil richness in a

given area of the field• Laid the framework for factorial experiments (to be discussed later)

DOE Historical Background (cont)• George Box (1950s): First use of DOE in industy• Worked at a chemical industry company• Wanted to vary input parameters to increase chemical yields• Proponent of iterative testing

• Testing produces results that will lead to more questions or opportunities to learn more about the system or process. These new questions lead to more testing.

• Laid the framework for the response surface method process (to be discussed later)

DOE Historical Background (cont)• Genichi Taguchi (1970s): Applied DOE to quality improvement• Implemented DOE methods for process optimization and quality

improvement• Ideas and methodology made Toyota an international brand and jump

started Japanese quality production• Laid the framework for fractional factorial experiments (to be discussed later)

DOE Historical Background (cont)• Modern era: (1990-present): Companies develop specialized DOE

processes• Motorola has developed the six sigma program to product optimization• Many companies exist with the sole purpose of developing test plans using

DOE strategies• These companies have proprietary methods of DOE and are paid by other companies to

help optimize their testing process

DOE Basics Principles• Randomization: Randomly assigning which test point to assign to

which test item or the order in which the points are conducted• Pioneered by Sir Ronald A. Fisher• Eliminates bias by average out any uncontrolled variables• For example: if testing ten altered systems and ten unaltered systems for the

control group, the order in which they are tested should be random

• Replication: Repeating a test point to add confidence to a certain conclusion or to make sure the system’s response isn’t an anomaly• Important to raise confidence in the test, especially when the results of the

test are either controversial or important• May be too expensive to replicate test points

DOE Basics Principles (cont)• Blocking: Eliminating uncontrolled variables or other known biases

from the experiment• Minimizing uncontrolled variability makes sure the test results are as

accurate as possible• For example, in flight testing the testers may aim to have the weather as

constant as possible to ensure changing weather conditions don’t affect the test

• Orthogonality: When no compounding or negating effects exist• For example, if an aerially employed munition loses accuracy at higher

altitudes or higher Mach numbers but increasing both simultaneously does not compound the effect• Is not required or commonly used but simplifies the analysis process

DOE Implementation• The value of DOE methods lies in its ability to help pick test points to

conduct• Ideally there would be an unlimited budget with unlimited time to

conduct all possible test points to completely understand the system• Unfortunately budget and schedule restrictions limit the number of test

points that can be completed• DOE allows testers to conduct a thorough test even with these limitations

• Popular DOE methods include factorial experiments, fractional factorial experiments, response surfaces, and space filling

Factorial Experiments• Invented by Sir R. A. Fisher, all selected input variables are changed

orthogonally• For example, if X input variable are selected and two values of each input

variable are to be tested, then a “cube” of X dimensions with 2X test points will be used• If three input variables are used with two values per parameter, then eight test points

and a standard 3D cube represents the test points• The figure below graphically represents this

Factorial Experiments (cont)• Factorial experimentation is a very systematic, simple approach• Any number of input variables and any number of values for these

variables can be selected• i.e. a large complex system may have 10 input parameters with two values

each for a total of 210 total test points arranged in a “cube” of 10 dimensions• For more precision these ten input variables could have more values for even

more total test points

Factorial Experiments Example• Consider a chemical experiment in which the chemists want to test

the yield of the reactions when changing temperature and pressure• Temperatures of 50°C and 100°C and Pressures of 1 MPa, 2 MPa, and 3 MPa• Use blocking to the max extent possible to limit the effect of uncontrolled

variables• Test points in this full factorial experiment are shown in the table below

Test Point Temperature (°C) Pressure (MPa)

1 50 1

2 50 2

3 50 3

4 100 1

5 100 2

6 100 3

Fractional Factorial Experiment• Factorial experiments can become extremely large and expensive as

more input variables and more values for these input parameters are selected• Two input parameters with two values: 4 test points• Five input parameters with two values: 32 test points• Eight input parameters with two values: 256 test points

• Genichi Taguchi developed the fractional factorial experiment in order to lower the number of test points while still keeping the systematic approach of factorial experiments• Once the full factorial test points are selected, a geometrically

selected subset of these points are conducted

Fractional Factorial Experiment (cont)• Test points from the full factorial experiment are selected to

maximize diversity in the test while lowering test points to manageable number• Still has repetition inherently

• Figure below shows test point selection

Fractional Factorial Experiment Example• Consider an aerially launched munition• Too expensive to launch missiles for every test point in full factorial test• Assume altitude (10,000ft and 20,000ft), Mach number (0.5 and 0.75), and

which side of the aircraft the munition is on (left or right) are selected input variables• Table on the following page represents the full factorial experiment and

highlighted rows represent a fractional factorial experiment

Fractional Factorial Experiment Example (cont)

• Notice in full factorial, both altitudes, Mach numbers, and munition locations are tested four time each• Fractional factorial experiment tests both the altitudes, Mach numbers, and

munition locations twice• A smaller fractional factorial expereiment could be performed by using test

points 1 and 6

Test Point Altitude Mach number Munition side

1 10,000 0.5 Left

2 10,000 0.5 Right

3 10,000 0.75 Left

4 10,000 0.75 Right

5 20,000 0.5 Left

6 20,000 0.5 Right

7 20,000 0.75 Left

8 20,000 0.75 Right

Response Surface Method • Developed by George Box in the 1950s• Goal is to derive an analytical expression for the output parameter in

terms of the selected input variables• Allows testers to predict the systems response at any input parameter values• Analytical expression is easy to maximize or minimize in order to optimize the

system’s performance

• Number of test points depends on number of input variables selected and desired accuracy of the analytical expression

Response Surface Method (cont)

• Response surface method allows optimization• May require large number of test points as more input variables are selected• When multiple output variables are selected it may become complicated • Expression is only an approximation (only as accurate as the number of test

points allow

Space Filling Method• Meant for systems with little to no variability• Computer programs or simulations• Replication is unnecessary because the results will be identical to the original

run if the system has no variability

• Goal is to spread out the data points as much as possible in the area of interest• Specifically, the goal is to maximize the minimum distance between the test

points (see figure on following slide)

Space Filling Method (cont)

• Figure shows possible test points• Circles show how data points are spread out evenly• Larger circles on the left signify that fewer test points will be run

Optimal Design Procedures• Goal is to minimize cost b intelligently selecting test points• Useful for fractionating large factorial experiments or picking test points for

programs with large budget/schedule restraints

• Uses statistics to decrease variability in the test points• Allows fewer test points to be selected while still providing the same amount

of information

• Users to make sure they understand the programs performing these operations rather than using it as a “black-box”

Optimal Design Procedures (cont)• Programs that perform optimal design procedures are typically

proprietary to large companies or companies specializing in DOE• DOE specialized companies are hired by companies that require assistance in

designing test programs• Contracting out test designing like this may add cost to a program but is

hopefully made up through test cost savings and a quality test design

Uses of DOE• DOE is most useful in large or expensive testing such as flight testing

or testing where the test item is consumed or destroyed (such as weapons testing)• Flight testing costs ten of thousands in jet fuel, MX/support, etc• Minimizing the number of test points can save time and money

• Main uses of DOE include Product design/development, process optimization, and tolerance determination

Product design/development• DOE can be used to aid in the design or development of a new

product• During the design process, decisions must be made about component

or process specifications• DOE can help show how changing these component specifications

(input variables) affect the purpose of the system (out variable)

Product design/development Example• Consider a turbine being developed• Number of blades, geometry of the blades, exit pressure, max temperature

inside the turbine, working fluid, etc. can all be selected as input variables• Max power or efficiency may be selected as out put variable• Building/modeling and testing every possible design is impractical and much

too expensive• DOE should be used

Product design/development Example (cont)• Response Surface method may work well• Obtaining an analytical expression for the max power output would allow

designers to maximize their design• Selecting all desired input parameters and perform a full or fractional

factorial experiment• Using these data points a curve fit for max power can be computed• Number of test points depends on desired accuracy of expression and

number of input variables selected

Process Optimization• Just as a machine can be optimized, so can a process• Parameters of the process can be changed (input variables) to see

how the process’s outputs are affected• Consider a factory assembly line• Management wants to know which employees should be working to

maximize the yield of the factory• Simplest option is to use every combination of the employees to see which

combination makes the most parts• Could take months or longer to run that many shifts

• DOE processes can help solve this problem

Process Optimization (cont)• Response Surface methods can apply here• Each employee is an input variable while the number of parts built is the

output variable• Performing a full or fractional factorial experiment will gather data points to

calculate this analytical expression for the number of parts built in terms of which employees are working• Enforcing binary constraints on the variables (the employee can either be

working or not) and maximizing this expression will answer managements question of the optimal assembly line they can have

System Tolerance Determination• During the designing of a system, physical tolerances are required for

all parts to be built• Looser tolerances are easier (and cheaper) to build but can cause the

system to fail if they are too loose• Goal is to maximize tolerances while still having a system that works to some

confidence level

System Tolerance Determination (cont)• Consider the manufacturing of a piston for a car (or any physical part)• The component will be built in a CAD model early on• The worst case scenario size of the component would be it’s maximum

tolerance• A simulation can be built that operates a model of the system with varying

sizes of the component being assessed• Using this simulation and the size of the component as the input variable, the

space filling DOE method could be used

Summary• Testing is a critical part of the designing and development process• Testing gives knowledge about the system and acts to ensure the

system will work as designed when used in the operational environment• Testing can help in the designing of a system, in the optimization of a

process, the determination of tolerances, and more

Summary (cont)• Testing can be extremely difficult and complex• Large/complex or expensive programs may require the testing to be done

efficiently• Budget or schedule limitations may require less testing than may be desired

• Number of test points is often determined by budget and schedule considerations rather than the desired quality of the test plan

• DOE methods and procedures should be used for these complex systems to maximize the effectiveness of the test

Summary (cont)• Design of Experiment is a statistical and mathematical approach to

test planning• Often lowers the required number of test points while still offering a

reasonably thoroughly and informative test• Pioneered by individuals like Sir Ronald A. Fisher, George Box, and Genichi

Taguchi• Uses principles such as randomization, replication, blocking, and

orthogonality

Summary (cont)• One of the most important applications of DOE is its uses in choosing

which test points to compute. Major DOE methods include:• Factorial Experiments pioneered by Sir. Ronald A Fisher• Fractional Factorial Experiments developed by Genichi Taguchi• Surface Response methods used by George Box• Space filling strategies for systems with low variability• Complex techniques used by major companies in industry (such as Motorola’s

six sigma program) and companies specializing in Design of Experiment

• Which method to use depends on the goals of the test and the complexity of the program

Summary (cont)• Implementing DEO processes in a program takes experience and

should be tailored specifically to the program• While it may be complex to implement, Design of Experience can be

an invaluable tool in the testing of more complex and expensive systems