research methodology eph 7112 lecture 7: experimental design

Research Methodology

EPH 7112

LECTURE 7: EXPERIMENTAL DESIGN

Contents

Synthesis Implement Solution Design Experiments Conduct Experiments Reduce Results

Scientific Method

Analysis

Synthesis

Validation

Hypothesis

Analysis

Describe Problem

Set Performance Criteria

Investigate Related Work

State Objective

Hypothesis

Specify detail and comprehensive solution

Assert expected results Define factors that will be varied Measure against performance metrics

Synthesis

Implement the solution And experiment To accomplish the goals And validate the

hypotheses

Synthesis Steps

Implement Solution Design Experiments Conduct Experiments Reduce Results

Implement Solution

Implement solution to test hypotheses Methods:

AcquireConstructCombination of both

Implement Solution

ConstructCustom made to meet requirements Time consumingExpensive

Implement Solution

AcquireQuick solutionCheaperMay not meet requirements

Implement Solution

Consider strongly to acquire the solution

Even if part of entire solution Consultants – acquired solution?

Implement Solution

Example: Optical Amplification in S-band

Construct using Thulium doped fiber Problem: Fusion Splicing is not

possible Solution: Use Angled Connectors Issue: Not specified during order

Implement Solution

Is your solution right or is it the right solution?

Careful implementation Step-by-step Troubleshooting Example: Constructing an optical

amplifier Problem: WDM is faulty

Design Experiments

To design a series of experiments Results used to estimate how good

solution to solve problem An experiment acquires data to

measure the performance of the solution under controlled conditions in a laboratory

Design Experiments

Always good to have a check list Objective Unit under Test Inducers Sensors Supervisor Channels Domain Knowledge Range Knowledge Solution

Experiments

Is it really necessary? How about theoretical or simulation

work? Experiment = verification Example: Find solution of two-

dimensional plane that satisfy certain conditions

Experiments

Simulation and modeling Verify against experimental results Example: Modeling of Optical

Amplifier Advantages of modeling

OptimizationAnalyzing the physical phenomena

Design Experiments

Planning:Specification Experiment LaboratoryDesign of experiment blocksDesign of protocolsAcquiring and managing data

Experiment Laboratory

Laboratory is where the experiment takes place

Large room with test & measurement equipments, units under test, chemical & mechanical apparatus, computers

Laboratory

Experiments can also take place:In an officeFieldManufacturing Plant

Laboratory

Depends on the experiment:ObjectiveSample Unit under TestInducersSensorsSolution

Laboratory: Safety

Watch out for moving or revolving parts (they don’t like necklaces and neck ties!)

Watch out for Electro-Static Sensitive Devices

Limit personnel into the laboratory Maintenance and cleaning personnel

can cause mishaps

Design Experiment Block

Process that takes place in the laboratory during the experiment

Important termsSampleFactors (independent variables)InducersSensors

Sample

Task unit consisting of objects, living plants, animals, humans that is the subject in the experiment

Factors

Condition or parameter of a task whose value is intentionally varied to measure its impact on the results of the task

Inducers

A device or mechanism that alters the task unit/ subject during the experiment

Sensors

Device that capture the results from the task/ unit or subject

Extent

Each factor is assigned a set of values

Extent of the factor space is total number of unique combination of values that may be assigned to each factor

FVa = number of values for factor a

Extent = FVa x FVb x FVc

Treatment

Each one of the unique combination of values that may be assigned to every factor is called a treatment

One instance in the entire factor space

Case Study

FactorsTypeface: 2 types (Serif and Sans

Serif)Noise Level: 12 levelsCharacter: 36

Extent = 36 x 12 x 2 = 864 Each combination = treatment

Block Design

If sample is a single object or device, then all the possible treatment must be assigned to it during the experiment

Example: Characterization of a Thulium doped Fiber Amplifier for different pump powers and wavelengths

Block Design

If sample is more than one, then the treatments may be distributed in some way among the sample

Important terms:Experiment trialExperiment block

Experiment Trial

Complete set of treatments applied to a sample during the experiment (sample is more than one)

Example: The combination of typeface: serif, character <A, B, C> and noise level <130, 140, 150>

Experiment Block

Set of experiment trials that provides a cover of the factor space that is appropriate and adequate for achieving the task objective

Block Design

What is the appropriate set of experiment trials

that provides an appropriate cover of the factor space

for the experiment?

Block Design

Three basic strategies:Enumerated block designSystematic block designRandomized block design

Enumerated Block Design

Assigns every possible treatment to every sample

Obvious strategy if sample = 1 If sample > 1, this is not practical Because total number of trials =

extent of factor space x number of sample

Too large !!

Systematic Block Design

Uses a deterministic algorithm to assign treatments to different sample in a systematic way

Eventually covers the entire factor space

Problem: Unintentional resonance between sample and treatment can be sparked

Systematic Block Design

Example: A marketing survey is carried out to

every 100th telephone numberThe chances a treatment assigned to,

say a number 03-2698 1100 belonging to a business entity

Is higher than say 03-2698 1024A bias towards response of business

entity may occur in the survey

Systematic Block Design

This block design should be avoided Unless this bias can be ascertained

Randomized Block Design

Similar to systematic block design Except that the treatment assigned to

the sample are sequenced randomly This can also reduce the risk of

systematic bias

Case Study

Enumerated block design is not practical

Total trials = 864 x 14! Each sample has to respond to 864

treatments! Fatigue ‘Peak Performance’

Case Study

Another disadvantage: Humans are smartEasily guess that factor space include

10 decimal digits and 26 Latin characters

Guess from elimination processBias the results

Case Study

Since all the license plate inspectors had same recognition skills

Not all treatment need to be assigned to every sample/ subject

Divide 864 treatments equally Reduce time for each subject Can a systematic block design do it?

Case Study

Systematic block design also has setbacks

Subjects can also detect the periodicity

Biased improved performance Randomized block design is solution Computer generated pseudo-random

assignment of treatments

Case Study

Decide how many and which sets of treatments would be randomly assigned to subjects

Combined to cover enough sets for each factor

To make up set of trials that cover entire factor space

Representation Factor

A factor that is not intended as a basis for measuring performance

However they are necessary for assigning values of a parameter

Example:Characters

Performance factor

A factor that is used as a basis for measuring performance

Example:TypefaceNoise Level

Case Study

Either assign each of two typefaces to half the subjects

Or assign both typefaces to all Former solution better to avoid

confusion among subject and more realistic

Case Study

Noise Level range 130 to 240 with increments of 10

How to distribute the treatment to subjects?

Condition:Interval must be sameSubsets differentSame average

Case Study

Subsets suitable:{130, 150, …, 210, 230}{140, 160, …, 220, 240}{130, 140, …, …, 230, 240}

Case Study

How many subjects? Access to 14 trained license plate

inspectors 2 for OP Pilot Number of subjects will determine the

combination of performance factor values

Case Study

Statisticians require at least 30 responses over entire experiment for each typeface and noise level combination

Characters can be assigned to obtain response

Must use whole set (36 characters) or multiple of whole set to avoid character bias

Case Study

Decided:6 subjects for OP Pilot8 subjects for experiment trialUsing block design in column B

Control Trial

Measures the performance of one set of task in the absence of another to isolate the effects of the included components on performance

Control Trial

To identify possible bias in the processes of the project task

Bias is a consistent tendency to behave in an inconsistent way under certain conditions

Example: A spring loses its memory when

elastic limit is exceeded

Control Trial

Case Study:The ‘Listening Rat’Disabled Power Brakes

Control Trial

To establish performance baselines for comparison

Without baseline, it is impossible to test the hypothesis of a solution that suggests a certain improvement or behavior

Example:To test if a new hand lotion is better

than not using any hand lotion

Case Study

Two control task was introduced:With characters but without noiseWithout characters, only noise

Case Study

The first control trial to ensure that each subject had sufficient experience with interface during practice sessions

So that the effects of the learning curve is negligible during test trials

If learning bias occurs, repeat practice session

Case Study

Second control trial to measure selection time in the absence of any characters

Pure guessing The statistics of this selection time

used as baseline for computing the confidence that subject did not purely guess in the test trials

Protocols

Step-by-step procedure to be followed during preparation and conduct of experiment

Main purpose:To ensure that experiment can be

accurately and precisely repeated

Protocols

Check list can help ensure uniformity in preparation of lab before experiment trial begins

Everyone involved must carry out protocol accordingly

Pilot trials can be used to plan and debug the protocol

Protocols

Anyone in contact with human subjects in an experiment trial should not expose the objective of the experiment

Protocols can be printed as flowcharts, pseudo-code or lists

Case Study

Characterization of EDFA Steps include:

Measure input signalMeasure output signalOSA will compute Gain, NF, PASE

Sequence of measurement is important

Data Management

Most critical and frustrating task Protect data Maintain logs of data (where it is kept,

which file is for what) Record ALL experiment data

Data Management

Do not preprocess the raw experiment data in any way before recording it

Establish clear organizational and documentation conventions for data files

Back-up !!

Conduct Experiments

Time to follow your plans Resist temptation to

improvise on the fly If doesn’t run well, stop and

revise Consider failed experiment

as pilot trial

Reduce Results

Performance values to validate the hypothesis cannot be drawn directly from raw results

The raw results must be reduced, combined or transformed to be meaningful

Case Study 1

Identify Faster Traffic on Highway Need to measure speed This is not raw results Reduced from compression signals,

time two pulses occurred

Case Study 2

Characterization of EDFA Need to measure Gain (dB) and Noise

Figure (dB) Not raw results Reduced from among others; Input

Signal Power (dBm), Output Signal Power (dBm), ASE Level (dBm)

Reduce Results

Data Reduction methods may change Performance metrics may be altered Important to record both raw and

reduced results

Q&A