[email protected] universityofcastilla...

Optimal experimental design, an introduction

[email protected] of Castilla-La ManchaDepartment of Mathematics

Institute of Applied Mathematics to Science and Engineering

Inst. Appl. Math. Sci. & Eng. Optimal experimental design, an introduction, Jesús López Fidalgo

Books (just a few)

I Fedorov V.V. (1972). Theory of optimal experiments. Academicpress. New York.

I Pazman A. (1986). Foundations of Optimum Experimental Design.D. Reidel Pub. New York.

I Pukelsheim F. (1993). Optimal Design of Experiments. John Wiley& Sons. New York.

I Fedorov V.V. and Hackl P. (1997). Model-oriented design ofexperiments. Springer. New York.

I Atkinson A.C., Doney A.N., Tobias R. (2007). OptimumExperimental Designs, With SAS. Oxford science publications.Oxford.


OPTIMAL EXPERIMENTALDESIGN THEORY


Concepts and notation

Process:I Select a model,

E(y | x) = f T (x)θ, Var(y | x) = σ2 (linear model).I f continuous with components linearly independent.I Select an experimental condition x on a design space χ

(typically a compact on a Euclidian space).I Examples:

I y = α0 + α1x + ε, f (x) = (1, x)T , x ∈ χ = [a, b].I y = α1x1 + α2x2 + α3x3 + ε, f (x) = x = (x1, x2, x3)T ,

x ∈ χ = [a1, b1]× [a2, b2]× [a3, b3].I y = α0 + α1x + α2x2 + ε, f (x) = (1, x , x2)T , x ∈ χ = [a, b].I ANOVA (1-way, 3 levels):

y = β0 + β1x1 + β2x2 + ε, f (x1, x2) = (1, x1, x2)T ,x = (x1, x2) ∈ χ = {(0, 1), (1, 0), (1, 1)}


Process

I Perform the experiment and observe the response (r.v. y).I Repeat the process for the experimental conditions x1, . . . , xn

obtaining responses y1, . . . , yn.I Estimate the parameters (MLE) θ1, . . . , θm, σ2 and make inferences.


EXPERIMENTAL DESIGN

I Aim: Choosing x1, . . . , xn (exact design of size n).I Some may be repeated (probability measure):ξn(x) = nx

n , where nx = times x appears in the design.I Covariance matrix: Σθ = σ2(XTX )−1 = σ2n−1M−1(ξ),

M(ξ) =∑

x∈χ f (x)f T (x)ξn(x) (information matrix):I Symmetric.I Non-definite positive.I Singular if the number of different points in the design k < m.


Extended concept of experimental design

I Approximate design (Kiefer): probability measure ζ on the Borelfield generated by the open sets of χ.

I Information matrix: M(ζ) =∫χf (x)f T (x)ζ(dx).

I Convex set of approximate designs: Ξ.I If ζ is discrete and ξ the probability function associated:

M(ξ) =∑x∈χ

f (x)f T (x)ξ(x)

I If ζ is absolutely continuous there exists a density function (pdf), ξ,associated:

M(ξ) =

∫χ

f (x)f T (x)ξ(x)dx

I Compact and convex set of information matrices: M


Caratheodory’s theorem

I Caratheodory’s theorem: For any information matrix there isalways a design with at most 1

2m(m + 1) + 1 different points.I Finite support,

ξ =

{x1 x2 . . . xkp1 p2 . . . pk

}I Restricting Ξ to finite designsM is still the same (Caratheodory,

definition of integral and continuity).I In practice, perform ni ≈ nξ(xi ) experiments at xi ,

∑i ni = n.

I From now on we will use ξ or ζ as convenience.


Example

y = α0 + α1x + ε, x ∈ χ = [0, 1], f (x) = (1, x)T .

ξ =

{0 0.5 10.2 0.4 0.4

}.

M(ξ) =3∑

i=1

(1 xixi x2

i

)pi

=

(1 00 0

)0.2 +

(1 0.50.5 0.25

)0.4 +

(1 11 1

)0.4

=

(1 0.60.6 0.5

)


Example

ξ =

{0 0.5 10.2 0.4 0.4

}.

If just n = 12 experiments may be performed, possible rounding off:about 0.2× 12 = 2.4 (2, 3, 3) experiments at 0,about 0.4× 12 = 4.8 (5, 4, 5) experiments at 0.5,about 0.4× 12 = 4.8 (5, 5, 4) experiments at 1.Remark: It is not as simple as the usual rounding off.


CRITERION FUNCTION

I Φ : Ξ→ [0,+∞) or Φ :M→ [0,+∞)to be minimized (maximized).

I The first is more general.I The second

I has to be considered as a function of M−1,I has to be non-increasing (better estimates of the parameters):

M ≤ N ⇒ Φ(M) ≥ Φ(N),

I may be global or partial (interest on part of the parameters),I may be positive homogeneous: Φ(δM) = 1

δΦ(M), δ > 0,

I M is within the cone of the non-negative definite matrices in theEuclidean space of the symmetric matrices. Thus, the usual conceptof differentiability applies here.

I A Φ–optimal design ζ∗ minimizes Φ.I Convex: Φ[(1− ε)ζ + εζ ′] ≤ (1− ε)Φ(ζ) + εΦ(ζ ′).


Equivalence theorem for differentiable criteria

I Sensitive function:

ψ(x , ξ) = ∂Φ[M(ξ), f (x)f T (x)]

= f T (x)∇Φ[M(ξ)]f (x)− trM(ξ)∇Φ[M(ξ)].

I Equivalence theorem:ξ∗ is Φ–optimal if and only if ψ(x , ξ∗) ≥ 0, x ∈ χ.Equality for x ∈ Sξ∗ .


Comments on the equivalence theorem

I Just for approximate designs.I M is convex and it is the same for finite or general designs.I From now on we assume finite designs and use the symbol ξ (pdf)

instead of ζ (measure).I Still valid for a restricted search in a convex subset.


Checking condition

ψ(z , ξ∗) = 0, z ∈ S∗ξ(∂ψ(x , ξ∗)

∂x

)x=z

= 0, z ∈ S∗ξ ∩ Int(χ)

E.g., for a two parameter model assume ξ =

{x1 x2

1− p p

},

then there are two equations:

ψ(x1, ξ) = 0, ψ(x2, ξ) = 0

and 0, 1 or 2 of the equations:(∂ψ(x , ξ)

∂x

)x=x1

= 0,(∂ψ(x , ξ)

∂x

)x=x2

= 0

to be solved.


EFFICIENCY (goodness of a design)

effΦ(ξ) =Φ[M(ξ∗)]

Φ[M(ξ)].

If Φ homogenous, e.g. for 60% efficiency and n observations:

0.6 =Φ[M(ξ∗)]

Φ[M(ξ)]=

Φ(σ−2nΣ∗−1θ

)

Φ(σ−2nΣ−1θ

)=σ2n−1Φ(Σ∗−1

θ)

σ2n−1Φ(Σ−1θ

)=

Φ(Σ∗−1θ

)

Φ(Σ−1θ

).

For n observations with ξ and n∗ with ξ∗ the efficiency is 1 if:

1 =Φ(Σ∗−1

θ)

Φ(Σ−1θ

)=σ2n∗−1Φ[M(ξ∗)]

σ2n−1Φ[M(ξ)]=σ2n∗−1

σ2n−1 0.6,

n∗ = 0.6n (60%) observations would be enough with ξ∗.


General properties for global criteria

I A Φ–optimal design has m points at least:Any design with less than m points gives a singular matrix.

I For strictly convex criteria there is always a Φ–optimal design withno more than m(m+1)

2 points:I The optimal must be in the boundary (dimension m(m+1)

2 − 1).I Caratheodory in the boundary gives a limit of m(m+1)

2 points.I If σ2(x) is not constant, the whole theory is applicable for

f (x) = f (x)/σ(x).

I If the observations are correlated (Σy ), just exact designs.Information matrix: XT Σ−1

y X .


D–OPTIMALITY

I Determinant of the inverse:

ΦD [M(ξ)] =

{log detM−1(ξ) = − log detM(ξ) if detM(ξ) 6= 0,∞ if detM(ξ) = 0.

I Homogeneous definition:

ΦD [M(ξ)] =

{detM−1/m(ξ) if detM(ξ) 6= 0,∞ if detM(ξ) = 0.


Properties

I ΦD continuous onM.I Convex onM and strictly convex onM+ (nonsingular information

matrices).I Differentiable onM+:

5(− log detM) = −M−1

I Minimum volume of the confidence elipsoide:

(Θ−Θ)T (XTX )−1(Θ−Θ) ≤ mFm,n−m,γS2R ≡ c2.

Volume =cmVm[detM−1(ξ)]1/2, where Vm is the volume of them–dimensional sphere.


G–optimality

I Kirstine Smith (1918).I Kiefer & Wolfovitz gave the name.I

ΦG [M(ξ)] =

{maxx∈χ f T (x)M−1(ξ)f (x) if detM(ξ) 6= 0,∞ if detM(ξ) = 0.

I ΦG continuous and convex onM.I ΦG [M(ξ)] ∝ maxx∈χ varξ y(x).


A–optimality

I Trace of the inverse:

ΦA[M(ξ)] =

{trM−1(ξ) if detM(ξ) 6= 0∞ if detM(ξ) = 0,

}∝

m∑i=1

varξ θi .

The second equality is a consequence of

varξ θi ∝ eti M−1(ξ)ei

if the parameters are estimable.I ΦA continuous onM.I Convex onM and strictly convex onM+.I Differentiable onM+:

5(trM−1(ξ)) = −M−2(ξ)


L–optimality

I Atwood (1976):

ΦL[M(ξ)] =

{trWM−1(ξ) if detM(ξ) 6= 0,∞ if detM(ξ) = 0,

where W is a positive definite matrix of dimension m.I ΦL continuous onM.I Convex onM and strictly convex onM+.I Differentiable onM+:

5[trWM−1(ξ)] = −M−1(ξ)WM−1(ξ).


c–optimality

I Variance of the estimator of cT θ, linear combination of theparameters.

I Φc [M(ξ)] = cTM−(ξ)c .


Elfving procedure in practice

I Plot the curve x(t) = f1(t), y(t) = f2(t), t ∈ χ and its symmetricthrough origin.

I Plot the convex hull of both curves.I Plot the line through the origin defined by vector c .I Optimal design defined by the boundary point.


Elfving plot (One–point design)

{t∗

1

}


Elfving plot (Two–point design)

{a t∗

1− p∗ p∗

}


Elfving procedure for three parameters

00.5

1

1.5

2-1

-0.5

0

0.5

1

0

0.25

0.5

0.75

1

00.5

1

1.5

2

-1-0.5

00.5

1

-1

-0.500.5

1

-1

-0.5

0

0.5

1

-1-0.5

00.5

1

-1

-0.500.5

1

-1-0.5

00.5

1

-1

-0.500.5

1

-1

-0.5

0

0.5

1

-1-0.5

00.5

1

-1

-0.500.5

1

y = θ0 + θ1t + θ2t2 + ε, t ∈ [0, 1]

One, two or three design points.


Equivalence theorem for D– and G–optimality

ξ∗ is D–optimal iff it is G–optimal:

maxx∈χ

d(x , ξ∗) = m,

where d(x , ξ) = f T (x)M−1(ξ)f (x).

Efficiency bound: effD [M(ξ)] ≥ 2− maxx d(x,ξ)m


Example: Checking condition for a quadratic model

E [y ] = θT f (x) = θ1 + θ2x + θ3x2, σ2(x) = 1,

x ∈ χ = [−1, 2], f (x) = (1, x , x2)T .

M(ξ) =∑

x∈[−1,2]

f (x)f T (x)ξ(x)

=∑

x∈[−1,2]

ξ(x)

1 x x2

x x2 x3

x2 x3 x4

Assume the D-optimal design is of the type:

ξ =

{−1 z 21/3 1/3 1/3

},


Quadratic model

I Maximize the determinant:

detM(ξ) =13(−z2 + z + 2

)2.

The maximum is reached at z∗ = 0.5:

ξ∗ =

{−1 0.5 21/3 1/3 1/3

},

I Checking whether ξ∗ is the D-optimal:

d(x , ξ∗) = f T (x)M−1(ξ∗)f (x)

= 0.89(x2 − 3.43x + 3.3

) (x2 + 1.43x + 0.87

).

-1.0 -0.5 0.5 1.0 1.5 2.0

2.2

2.4

2.6

2.8

3.0


Example: Elfving procedure

E [y ] = θT f (x) = θ1x + θ2x2, σ2(x) = 1,

x ∈ χ = [0, 1], f (x) = (1, x , x2)T .

Plot the curve x(t) = t, y(t) = t2, t ∈ [0, 1]:

-1.0 -0.5 0.5 1.0

-1.0

-0.5

0.5

1.0


Symmetric through the origin

-1.0 -0.5 0.5 1.0

-1.0

-0.5

0.5

1.0


Convex hull

Tangential point, z :x(z)− x0

x ′(z)=

y(z)− y0

y ′(z)

z + 11

=z2 + 12z

, z =√2− 1


Line through the origin defined by vector c = (0.5, 0.3)

(λ0.5)2 = λ0.3, t = 3/5

ξ∗c =

{3/51

},


Line through the origin defined by vector c = (0.5, 1)

Cutting point: {y = 2x ,x−1z+1 = y−1

z2+1 ,

x =14

(2−√2)

= 0.15, y = 0.30.


Weights

Convex combination:

(0.15, 0.30) = (1− p)(−z ,−z2) + p(1, 1), p =14

(3−√2)

= 0.30

ξ∗c =

{ √2− 1 1

14

(1 +√2) 1

4

(3−√2) } .


THANK YOU


[email protected] universityofcastilla...

Documents