1 multivariate normal distribution shyh-kang jeng department of electrical engineering/ graduate...

93
1 Multivariate Normal Multivariate Normal Distribution Distribution Shyh-Kang Jeng Shyh-Kang Jeng Department of Electrical Department of Electrical Engineering/ Engineering/ Graduate Institute of Graduate Institute of Communication/ Communication/ Graduate Institute of Networking Graduate Institute of Networking and Multimedia and Multimedia

Upload: carol-chandler

Post on 23-Dec-2015

228 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

11

Multivariate Normal DistributionMultivariate Normal Distribution

Shyh-Kang JengShyh-Kang JengDepartment of Electrical Engineering/Department of Electrical Engineering/

Graduate Institute of Communication/Graduate Institute of Communication/

Graduate Institute of Networking and Graduate Institute of Networking and MultimediaMultimedia

Page 2: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

22

Multivariate Normal DistributionMultivariate Normal DistributionGeneralized from univariate normal Generalized from univariate normal densitydensityBase of many multivariate analysis Base of many multivariate analysis techniquestechniquesUseful approximation to Useful approximation to ““truetrue”” population distributionpopulation distributionCentral limit distribution of many Central limit distribution of many multivariate statistics multivariate statistics Mathematical tractableMathematical tractable

Page 3: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

33

Univariate Normal DistributionUnivariate Normal Distribution

xexf

N

x 2/]/)[(

2

2

2

2

1)(

),(

Page 4: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

44

Table 1, AppendixTable 1, Appendix

Page 5: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

55

Square of DistanceSquare of Distance(Mahalanobis distance) (Mahalanobis distance)

)('

)())((

1

122

μxμx

xxx

Page 6: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

66

pp-dimensional Normal Density-dimensional Normal Density

],,,['

vector random from sample a is

,,2,1,

2

1)(

),(

21

2/)()'(2/12/

1

p

i

p

p

XXX

pix

ef

N

X

x

Σx

Σμ

μxΣμx

Page 7: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

77

Example 4.1 Bivariate NormalExample 4.1 Bivariate Normal

)1(

1,

),Corr(/

)Var(),Var(

)(),(

2122211

2122211

1112

1222

2122211

1

2221

1211

2122111212

222111

2211

ΣΣ

XX

XX

XEXE

Page 8: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

88

Example 4.1 Squared DistanceExample 4.1 Squared Distance

22

22

11

1112

2

22

22

2

11

11212

22

11

11221112

22111222

2122211

2211

1

21

1

)1(

1],[

)('

xxxx

x

x

xx

μxΣμx

Page 9: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

99

Example 4.1 Density FunctionExample 4.1 Density Function

]}2

[)1(2

1exp{

)1(2

1),(

22

22

11

1112

2

22

22

2

11

11212

2122211

21

xx

xx

xxf

Page 10: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

1010

Example 4.1 Bivariate DistributionExample 4.1 Bivariate Distribution

11 = 22, 12 = 0

Page 11: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

1111

Example 4.1 Bivariate DistributionExample 4.1 Bivariate Distribution

11 = 22, 12 = 0.75

Page 12: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

1212

ContoursContours

pi

c

c

contourdensityyprobabilitConstant

iii

i

,,2,1,

:axes

at centered ellipsoidan of surface

)-()'(such that all

i

21

eΣe

e

μ

μxΣμxx

Page 13: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

1313

Result 4.1Result 4.1

definite positive

for ),(1/for ),(

1

definite positive :

1

1-

1-

Σ

ΣeΣe

eeΣeΣe

Σ

Page 14: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

1414

Example 4.2 Bivariate ContourExample 4.2 Bivariate Contour

]2

1,

2

1[',

]2

1,

2

1[',

rseigenvecto and seigenvalue

normal, Bivariate

212112

112111

2211

e

e

Page 15: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

1515

Example 4.2 Positive CorrelationExample 4.2 Positive Correlation

Page 16: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

1616

Probability Related to Probability Related to Squared DistanceSquared Distance

1y probabilit has

)()()'(

satisfying values of ellipsoid Solid21pμxΣμx

x

Page 17: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

1717

Probability Related to Probability Related to Squared DistanceSquared Distance

Page 18: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

1818

Result 4.2Result 4.2

),( bemust

every for )','(:'

)','(

:'

),(:

2211

ΣμX

aΣaaμaXa

Σaaμa

Xa

ΣμX

p

pp

p

N

N

N

XaXaXa

N

Page 19: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

1919

Example 4.3 Marginal DistributionExample 4.3 Marginal Distribution

),(

:in ofon distributi Marginal

),()','(:'

','

'],0,,0,1['

),(:]',,,[

111

111

1

21

iii

i

pp

N

X

NN

X

NXXX

X

ΣaaμaXa

Σaaμa

Xaa

ΣμX

Page 20: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

2020

Result 4.3Result 4.3

),(:

)',(:

),(:

11

2121

1111

ΣdμdX

AΣAAμAX

ΣμX

p

q

pqpq

pp

pp

p

N

N

XaXa

XaXa

XaXa

N

Page 21: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

2121

Proof of Result 4.3: Part 1Proof of Result 4.3: Part 1

)',(:every for valid

))'('),('(:)('

))'()'(,)'((:)'(

'

,')(' ncombinatiolinear Any

AAΣAμAXb

bAAΣbAμbAXb

bAΣAbμAbXAb

bAa

XaAXb

qN

N

N

Page 22: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

2222

Proof of Result 4.3: Part 2Proof of Result 4.3: Part 2

),(:

arbitrary is

)',''(:''

)','(:'

'')('

ΣdμdX

a

ΣaadaμadaXa

ΣaaμaXa

daXadXa

pN

N

N

Page 23: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

2323

Example 4.4 Linear CombinationsExample 4.4 Linear Combinations

322211

2

33232213222312

13222312221211

32

21

3

2

1

32

21

3

, with verifiedbe can

)',(:

2

2'

110

011

),(:

XXYXXY

N

X

X

X

XX

XX

N

AAΣAμAX

AAΣ

AX

ΣμX

Page 24: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

2424

Result 4.4Result 4.4

),(:

|

|

,,

),(:

1111

2221

1211

2

1

)1)((2

)1(1

ΣμX

ΣΣ

ΣΣ

Σ

μ

μ

μ

X

X

X

ΣμX

q

qp

q

p

N

N

4.3Result in |Set :Proof))(()()(

qpqqqpq0IA

Page 25: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

2525

Example 4.5 Subset DistributionExample 4.5 Subset Distribution

4424

2422

4

221

4424

242211

4

21

4

21

5

,:

,,

),(:

N

X

X

N

X

ΣμX

ΣμX

Page 26: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

2626

Result 4.5Result 4.5

22

11

2

1

2

1

2222211111

1221

2221

1211

2

1

2

1

)(21

)1(2

)1(1

|'

|

,:

tindependen ),(:),,(: (c)

0 ifonly and ift independen:,

|

|

,:--- (b)

),Cov( t,independen :, (a)

21

21

2121

Σ0

μ

μ

X

X

ΣμXΣμX

ΣXX

ΣΣ

ΣΣ

μ

μ

X

X

0XXXX

qq

qq

qq

qqqq

N

NN

N

Page 27: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

2727

Example 4.6 IndependenceExample 4.6 Independence

) also and oft independen is (

tindependen are and

tindependennot :,

200

031

014

),(:

213

32

11

21

3

XXX

XX

X

XX

N

X

Σ

ΣμX

Page 28: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

2828

Result 4.6Result 4.6

211

221211

221

2211

221

22

2221

1211

2

1

2

1

covariance

and )(mean with normal

is given ofon distributi lconditiona

0,

|

|

,),,(:

ΣΣΣΣ

μxΣΣμ

xXX

Σ

ΣΣ

ΣΣ

Σ

μ

μ

μΣμ

X

X

X

pN

Page 29: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

2929

Proof of Result 4.6Proof of Result 4.6

22

211

221211

22

221

221211

12212

|

'|

'

covariance with normaljoint

:

)(

)(

,

|0

|

Σ0

0ΣΣΣΣ

AAΣ

μX

μXΣΣμX

μXA

I

ΣΣI

A

Page 30: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

3030

Proof of Result 4.6Proof of Result 4.6

)),((

:given

),0(:)(

))()((

)

|)()((

)()(/),()|(t independen ,

tindependen are and )(

211

221211221

22121

221

211

221211221

221211

221

221211221

221211

22

221

221211221

221211

22221

221211

ΣΣΣΣμxΣΣμ

xXX

ΣΣΣΣμXΣΣμX

μxΣΣμxμXΣΣμX

xX

μxΣΣμxμXΣΣμX

μXμXΣΣμX

q

q

N

N

f

f

APBPBAPBAPBA

Page 31: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

3131

Example 4.7 Conditional BivariateExample 4.7 Conditional Bivariate

)),(()|(

thatshow

),(:

22

212

112222

12121

2212

1211

2

12

2

1

xNxxf

Nx

x

Page 32: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

3232

Example 4.1 Density FunctionExample 4.1 Density Function

]}2

[)1(2

1exp{

)1(2

1),(

22

22

11

1112

2

22

22

2

11

11212

2122211

21

xx

xx

xxf

Page 33: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

3333

Example 4.7Example 4.7

)1(2/))(/(

21211

22121

2221211

2122211

22

222

2

2222

12112

1211

22

22

11

1112

2

22

22

2

11

11212

21211

222221211

)1(2

1

)(/),()|(

2)1(2)1(2

)(

2

1)(

)1(2

1

]2[)1(2

1

xxe

xfxxfxxf

xxx

xxxx

Page 34: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

3434

Result 4.7Result 4.7

ondistributi theof percentile

)th (100upper thedenotes )( where

,1 is )}()()'(:{

ellipsoid solid theinsidey probabilit The (b)

: )()'( (a)

0),,(:

2p

2p

2p

1

2p

1

μXΣμXx

μXΣμX

ΣΣμX pN

Page 35: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

3535

22 Distribution Distribution

)2,12/ withondistributi(Gamma

0,0

0,)2/(2

1)(

(d.f.) freedom of degrees :,

)1,0(:);,(:

,),,(:),,(:

2

22/12/22/2

2

1

2

2

2222

2111

2

ef

x

NX

ZNX

NXNX

i i

ii

i

iii

Page 36: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

3636

22 Distribution Curves Distribution Curves

Page 37: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

3737

Table 3, AppendixTable 3, Appendix

Page 38: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

3838

Proof of Result 4.7 (a)Proof of Result 4.7 (a)

2

1

21

2

2

1

1

1

'

'

2'2

1'1

1

2

2

1

'

'

1

1

:)()'(),1,0(:

/

/

/

'

)',0(:)(,)(1

)()'(1

)()'(

p

p

iii

p

pp

iiii

pp

p

p

ii

p

ii

i

ii

p

i i

ZNZ

NZ

μXΣμX

Ieee

ee

e

e

e

AAΣ

AAΣμXAZμXe

μXeeμXμXΣμX

Page 39: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

3939

Proof of Result 4.7 (b)Proof of Result 4.7 (b)

1)()()'(

by ddistribute

variablerandom new )()'(

),(:by ellipsoid the toassigned

yprobabilit theis)()'(

21

2

1

21

p

p

p

P

N

cP

μXΣμX

μXΣμX

ΣμX

μXΣμX

Page 40: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

4040

Result 4.8Result 4.8

n

jj

n

jj

nn

n

j

n

jjjjpnn

jp

n

b

c

bbb

ccNccc

N

1

2

1

2

122112

1 1

222111

j

21

)()'(

)'()(

matrix covariancewith

normaljoint are and

)(,:

),(:

tindependenmutually :,,,

ΣΣcb

ΣcbΣ

VXXXV

ΣμXXXV

ΣμX

XXX

Page 41: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

4141

Proof of Result 4.8Proof of Result 4.8

ΣAAΣ

ΣΣAAΣ

AAΣAμV

VAX

III

IIIA

Σ00

0Σ0

00Σ

Σ

μ

μ

μ

μ

ΣμXXXX

X

X

X

X

X

)(:' rmsofdiagonalteoff

)(,)(:' of termsdiagonalblock

)',(:,

,

),(:],,,['

1

1

2

1

2

22

1

21

21

2

1

''2

'1

n

jjj

n

jj

n

jj

pn

n

n

npn

bc

bc

Nbbb

ccc

N

Page 42: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

4242

Example 4.8 Linear CombinationsExample 4.8 Linear Combinations

312123

22

21

321

1

34321

2223'

3'

)','(:'

201

011

113

,

1

1

3

),( identicalt independen:,,,

aaaaaaa

aaa

N

N

Σaa

μa

ΣaaμaXa

Σμ

ΣμXXXX

Page 43: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

4343

Example 4.8 Linear CombinationsExample 4.8 Linear Combinations

0ΣVVXXXXV

ΣΣΣ

μμμ

ΣμXXXXV

V

V

VV

4

12143212

4

1

2

4

1

343211

)(),Cov(,3

201

011

113

)(

2

2

6

2

),(:2

1

2

1

2

1

2

1

1

1

11

jjj

jj

jjj

bc

c

c

N

Page 44: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

4444

Multivariate Normal LikelihoodMultivariate Normal Likelihood

estimates likelihood Maximum

estimation likelihood Maximum

likelihood

,,, fixedfor and offunction a as

2

1

,,,

ofdensity Joint

),( from sample random:,,,

21

2/)()'(

2/2/21

21

1

1

n

nnpn

pn

n

jjj

e

N

xxxΣμ

ΣXXX

ΣμXXX

μxΣμx

Page 45: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

4545

Maximum-likelihood EstimationMaximum-likelihood Estimation

Page 46: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

4646

Trace of a MatrixTrace of a Matrix

k

i

k

jij

k

iiiij

kk

a

cc

caa

1 1

2

1

1)(

)'tr()e(

)tr()tr()d(

)tr()tr()c(

)tr()tr()(tr)b(

)tr()tr()a(

scalar a is ;)tr(

AA

AABB

BAAB

BABA

AA

AA

Page 47: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

4747

Result 4.9Result 4.9

k

ii

k

kk

1

)tr( b)(

)'tr()'tr(' (a)

vector1 :

matrix symetric :

A

AxxAxxAxx

x

A

Page 48: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

4848

Proof of Result 4.9 (a)Proof of Result 4.9 (a)

)'tr()')tr(()'tr(

)tr()tr(

)tr(

)tr()tr(

matrix :,matrix :

1 11 1

1 1

AxxxAxAxx

BCCB

BC

CBBC

CB

m

i

k

jjiij

k

j

m

iijji

m

i

k

jjiij

cbbc

cb

mkkm

Page 49: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

4949

Proof of Result 4.9 (b)Proof of Result 4.9 (b)

k

ii

kdiag

1

21

)tr()'tr(

)'tr()tr(

,,,

','

ΛΛPP

ΛPPA

Λ

IPPΛPPA

Page 50: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

5050

Likelihood FunctionLikelihood Function

2/''tr

2/2/

1

1

1

1

1

1

1

1

1

2

1),(

''

'

'

'tr'

n

jjj n

nnp

n

jjj

n

jjj

n

jjj

n

jjj

n

jjj

eL

n

μxμxxxxxΣ

ΣΣμ

μxμxxxxx

μxxxμxxx

μxμx

μxμxΣμxΣμx

Page 51: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

5151

Result 4.10Result 4.10

Σ

B

b

ebe

b

pp

pp

bppb

bb

2/1for only holding

equality with , definite positive allfor

211

scalar positive :

matrix definite positive symmetric:

)(

2/)tr( 1

Page 52: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

5252

Proof of Result 4.10Proof of Result 4.10

IBΣBBΣ

BBΣ

ΣBBΣBΣ

BΣB

BΣBBBΣBΣ

bb

ebebebe

eee

bppb

bbbbb

p

i

bipb

bp

ii

b

p

ii

p

i

i

i

p

ii

2such that )2/1( when attained is boundupper

211

2at 2 maximum a has

11

/,tr

positive all , of seigenvalue:

trtrtr

2/112/1

2/)tr(2/

1

2/2/

12/)tr(

11

1i

1

2/112/1

2/112/12/12/111

1

11

Page 53: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

5353

Result 4.11 Maximum Likelihood Result 4.11 Maximum Likelihood Estimators of Estimators of and and

SXXXXΣ

ΣμXXX

n

n

n

N

j

n

jj

pn

1'

ˆ

),( from sample random:,,,

1

21

Page 54: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

5454

Proof of Result 4.11Proof of Result 4.11

SxxxxΣ

ΣΣμ

μxΣμxxxxxΣ

Σμ

xxxxΣ

n

n

n

eL

n

L

n

jjj

nnp

n

jjj

n

jjj

1'

2

1),ˆ(

ˆ

'2

1'tr

2

1

:),( ofExponent

1

'tr

2/2/

1

1

1

1

1

Page 55: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

5555

Invariance PropertyInvariance Property

)Var( of MLE1

ˆ

ˆ of MLE

ˆˆ'ˆ' of MLE

:Examples

)( ofestimator likelihood maximum:)ˆ(

ofestimator likelihood maximum:ˆ

1

2

11

i

n

jijiii

iiii

XXXn

hh

μΣμμΣμ

Page 56: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

5656

Sufficient StatisticsSufficient Statistics

population normal

temultivaria a of statistics sufficient are and

and through ,,,

nsobservatio ofset wholeon the depends

2

1

,,,

ofdensity Joint

21

2/''tr

2/2/

21

1

1

Sx

Sxxxx

Σ

XXX

μxμxxxxxΣ

n

n

nnp

n

n

jjj

e

Page 57: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

5757

Distribution of Sample MeanDistribution of Sample Mean

4.8Result cf.

)/,(:

:case teMultivaria

)/,(:

1 :case Univariate

),( from sample random:,,,

2

21

nN

nNX

p

N

p

pn

ΣμX

ΣμXXX

Page 58: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

5858

Sampling Distribution of Sampling Distribution of SS

)|)1((on distributiWishart :)1(

),(:

:case teMultivaria

),0(:,)1(

:)1(

1 :case Univariate

),( from sample random:,,,

11

'

2

1

222

21

2

1

22

21

ΣSZZS

Σ0XXZ

ΣμXXX

n-Wn-

N

NZZsn

XXsn

p

N

n

n

jjj

pjj

j

n

jj

n

n

jj

pn

Page 59: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

5959

Wishart DistributionWishart Distribution

)'|'(:')|(:

)|(:

)|(:),|(:

:Properties

definite positive:

21

2)|(

2121

2211

1

2/)1(4/)1(2/)1(

2/tr2/)2(

1

21

21

1

CCΣCACCACΣAA

ΣAAAA

ΣAAΣAA

A

Σ

AΣA

mm

mm

mm

p

i

nppnp

pn

n

WW

W

WW

in

ew

Page 60: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

6060

Univariate Central Limit TheoremUnivariate Central Limit Theorem

size sample largefor normalnearly also is

on distributi normalnearly a has

ty variabilisame the

ely approximat having variablesrandom :

,,, causes

tindependen ofnumber large aby determined:

21

21

X

X

VVVX

V

VVV

X

n

i

n

Page 61: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

6161

Result 4.12 Law of Large NumbersResult 4.12 Law of Large Numbers

nε-μY-εP

n

YYYY

YE

YYY

n

i

n

as 1][

0, prescribedany for is,That

to

yprobabilitin converges

)( with normal) benot (may population

a from nsobservatiot independen:,.,

21

21

Page 62: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

6262

Result 4.12 Multivariate CasesResult 4.12 Multivariate Cases

ΣS

μX

μX

XXX

y toprobabilitin converges

y toprobabilitin converges

)(mean with

normal) temultivaria benot (may population

from nsobservatiot independen ,,,

i

21

E

n

Page 63: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

6363

Result 4.13 Central Limit TheoremResult 4.13 Central Limit Theorem

normal)nearly is populationparent the

when moderatefor ion approximat good (quite

size sample largefor

),(ely approximat is )(

covariance

finite and mean with population

a fromn observatiot independen:,,, 21

n

pn

Nn p

n

Σ0μX

Σ

μ

XXX

Page 64: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

6464

Limit Distribution of Limit Distribution of Statistical DistanceStatistical Distance

n-p

n

n

n-p

n

pnn

N

p

p

p

largefor

ely approximat:)()'(

large is

y whenprobabilithigh with toclose

largefor

ely approximat :)()'(

size sample largefor )1

,(nearly :

21

21

μXSμX

ΣS

μXΣμX

ΣμX

Page 65: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

6565

Evaluating Normality of Univariate Evaluating Normality of Univariate Marginal DistributionsMarginal Distributions

sticcharacterith for theon distributi

normal assumedan from departure indicate

628.0)046.0)(954.0(3954.0ˆor

396.1)317.0)(683.0(3683.0ˆeither

)22(in lying data ofportion :ˆ

)(in lying data ofportion :ˆ

data, ofsymmetry checkingAfter

2

1

2

1

i

nnp

nnp

sx,sxp

sx,sxp

i

i

iiiiiii

iiiiiii

Page 66: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

6666

Evaluating Normality of Univariate Evaluating Normality of Univariate Marginal DistributionsMarginal Distributions

0026.0]3ˆ[

),( is ˆ ofon distributi The

),( large, is When

on distributi binomial

:intervalan within samples ofNumber

n

pqppP

n

pqpN

n

yp

npqnpNqpy

nn

qpy

n

yny

yny

Page 67: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

6767

Q-QQ-Q Plot Plot

ondistributi normal a

from are data theif since linear,

ely approximat are they if see to,Plot

2/1

2

1][

/)2

1(/: ofPortion

20 e.g., large, tomoderate anddistict be Let

on nsobservatio:

)()(

)()(

2/)(

)(

)(

)()2()1(

2)(

jj

jj

zq

j

j

j

in

qx

xq

n

jdzeqZP

njnjxx

nnx

Xxxx

j

Page 68: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

6868

Example 4.9Example 4.9

Page 69: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

6969

Example 4.9Example 4.9

Page 70: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

Histogram of MidTerm Scores Histogram of MidTerm Scores of Students of This Course in of Students of This Course in

2006 2006

7070

Page 71: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

Q-Q Plot of MidTerm Scores of Q-Q Plot of MidTerm Scores of Students of This Course in 2006Students of This Course in 2006

7171n = 33, rQ = 0.946652

Page 72: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

7272

Example 4.10 Radiation Data of Example 4.10 Radiation Data of Closed-Door Microwave OvenClosed-Door Microwave Oven

Page 73: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

7373

Measurement of StraightnessMeasurement of Straightness

4.2 Tablein value

eappropriat thebelow falls if cesignifican

of levelat hypothesisnormality Reject the

)()(

))((

1

2)(

1

2)(

1)()(

Q

n

jj

n

jj

n

jjj

Q

r

qqxx

qqxx

r

Page 74: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

7474

Table 4.2 Table 4.2 Q-QQ-Q Plot Correlation Plot Correlation Coefficient TestCoefficient Test

Page 75: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

7575

Example 4.11Example 4.11

hypothesisnormality reject not Do9351.0

10.0,10

994.0,795.8

472.8,584.8

0,770.0 4.9, Example from dataFor

10

1

2)(

10

1

2)()(

10

1)(

Q

Qj

j

jjj

jj

r

n

rq

xxqxx

qx

Page 76: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

7676

Evaluating Bivariate NormalityEvaluating Bivariate Normality

)5.0()()'(such that all

bygiven ellipse in the lie

nsobservatio sample of 50%roughly ifCheck

22

1 xxSxxx

Page 77: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

7777

Example 4.12Example 4.12

Page 78: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

7878

Example 4.12Example 4.12

conclusion reach the tosmall toois )10( size sample However,

normality bivariatereject 50%an Greater th

39.1 with are nsobservatio 10 ofout Seven

39.134.4]4224,974.126[',

102927

309.62

128831.0003293.0

003293.0000184.0'

2927

309.62

39.1)5.0(

1030.1476.255

76.25520.005,10,

2927

309.62

2

221

5

2

1

2

12

22

5

n

d

dxx

x

x

x

xd

Sx

Page 79: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

7979

Chi-Square PlotChi-Square Plot

)/)2

1(1()/)

2

1(( that Note

1 slope havingorigin the

throughlinestraight a resemble shouldplot The

)),/)2

1((( allGraph

freedom of degrees on with distributi square-chi

theof quantile /)2

1(100:)/)

2

1((

distance squared Order the

distance squared :)()(

2,

2)(,

,

2)(

2)2(

2)1(

12

njnjq

dnjq

p

njnjq

ddd

xxSxxd

ppc

jpc

pc

n

Page 80: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

8080

Example 4.13 Chi-Square Plot Example 4.13 Chi-Square Plot for Example 4.12for Example 4.12

Page 81: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

8181

Example 4.13 Chi-Square Plot Example 4.13 Chi-Square Plot for Example 4.12for Example 4.12

Page 82: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

8282

Chi-Square Plot for Computer Chi-Square Plot for Computer Generated 4-variate Normal DataGenerated 4-variate Normal Data

Page 83: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

8383

Steps for Detecting OutliersSteps for Detecting OutliersMake a dot plot for each variableMake a dot plot for each variableMake a scatter plot for each pair of Make a scatter plot for each pair of variablesvariablesCalculate the standardized values. Calculate the standardized values. Examine them for large or small Examine them for large or small valuesvaluesCalculated the squared statistical Calculated the squared statistical distance. Examine for unusually distance. Examine for unusually large values. In chi-square plot, large values. In chi-square plot, these would be points farthest from these would be points farthest from the origin.the origin.

Page 84: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

8484

Helpful Transformation to Helpful Transformation to Near NormalityNear Normality

Original ScaleOriginal Scale Transformed ScaleTransformed Scale

Counts, Counts, yy

Proportions, Proportions,

Correlations, Correlations, rr

p

pp

ˆ1

ˆlog

2

1)ˆlogit(

y

r

rrz

1

1log

2

1)( sFisher'

Page 85: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

8585

Box and Cox’s Box and Cox’s Univariate TransformationsUnivariate Transformations

n

jj

n

jj

n

jj

xn

x

xxxn

n

λ

x

xx

1

)(_____

)(

11

2_____)()(

)(

1

ln11

ln2

)(

maximize to Choose

0,ln

0,1

Page 86: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

8686

Example 4.16 (Example 4.16 () vs. ) vs.

Page 87: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

8787

Example 4.16 Example 4.16 Q-QQ-Q Plot Plot

Page 88: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

8888

Transforming Multivariate Transforming Multivariate ObservationsObservations

p

jpjjj

n

jjkk

n

jjk

n

jkjk

k

p

pxxx

xn

x

xxxn

n

p

ˆ1

ˆ1

ˆ1

'

1

ln11

ln2

maximize toSelect

sticscharacteri the

for ssformationpower tran:,,,

)ˆ(

2

)ˆ(2

1

)ˆ(1)ˆ(

1

)(_____

)(

11

2_____)()(

k

21

21

λx

Page 89: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

8989

More Elaborate ApproachMore Elaborate Approach

p

jpjjj

p

k

n

jjkkp

p

p

pxxx

xn

p

111'

from computed is )(

ln1)(ln2

,,,

maximize to',,,Select

sticscharacteri the

for ssformationpower tran:,,,

)(

2

)(2

1

)(1)(

1 121

21

21

21

λx

λS

λS

λ

Page 90: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

9090

Example 4.17 Original Example 4.17 Original Q-QQ-Q Plot for Plot for Open-Door DataOpen-Door Data

Page 91: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

9191

Example 4.17 Example 4.17 Q-QQ-Q Plot of Plot of Transformed Open-Door DataTransformed Open-Door Data

Page 92: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

9292

Example 4.17 Contour Plot ofExample 4.17 Contour Plot of for Both Radiation Datafor Both Radiation Data),( 21

Page 93: 1 Multivariate Normal Distribution Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking

9393

Transform for Data Including Transform for Data Including Large Negative ValuesLarge Negative Values

2,01log

2,0)2/(11

0,0)1log(

0,0/11

2)(

xx

xx

xx

xx

x