the method of state space mixtures - university of...

75
Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References The Method of State Space Mixtures Michael D. Hunter Department of Pediatrics University of Oklahoma Health Sciences Center Modern Modeling Methods (M 3 ) Storrs, CT; May 20, 2015 Michael D. Hunter University of Oklahoma Health Sciences Center State Space Mixtures

Upload: hoangdieu

Post on 04-May-2018

217 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

The Method of State Space Mixtures

Michael D. Hunter

Department of PediatricsUniversity of Oklahoma Health Sciences Center

Modern Modeling Methods (M3)Storrs, CT; May 20, 2015

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 2: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

OpenMx 2.0

Now on CRAN!

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 3: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Outline

I Introduction, Background, and Motivation

I State Space Models and Kalman Filters

I Simulation

I Discussion, Conclusions, and Future Work

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 4: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Perspective

−4 −2 0 2 4

−4

−2

02

4

Income

Hap

pine

ss

●●

●●●

●●●

●●●●

●●●●●

●●

●●

● ●●●

●●

●●

● ●

●●●●●●●

●●

●●●

●●

●●

● ●●

●●

●●●●

●●●

●●●●

●●

●●

●●

●●

●●

●●●●●

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 5: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Where to go from here?

I Between-person models are valid, but (generally) onlybetween people.

I Model individuals and processes.

I Balance the Idiographic/Nomothetic trade-off

I Options

1. Assume complete heterogeneity: Separate processes2. Assume complete homogeneity: Same process3. Assume a mixture of homogeneous groups

I How do you model variability within people?

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 6: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Where to go from here?

I Between-person models are valid, but (generally) onlybetween people.

I Model individuals and processes.

I Balance the Idiographic/Nomothetic trade-off

I Options

1. Assume complete heterogeneity: Separate processes2. Assume complete homogeneity: Same process3. Assume a mixture of homogeneous groups

I How do you model variability within people?

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 7: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Where to go from here?

I Between-person models are valid, but (generally) onlybetween people.

I Model individuals and processes.

I Balance the Idiographic/Nomothetic trade-off

I Options

1. Assume complete heterogeneity: Separate processes2. Assume complete homogeneity: Same process3. Assume a mixture of homogeneous groups

I How do you model variability within people?

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 8: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

State Space ModelMeasurement

I Structural Equation Measurement Model

yi = Ληi + Kxi + εi with εi ∼ N (0,Θ) (1)

I State Space Measurement Model

yi = Ληi + Kxi + εi with εi ∼ N (0,Θ) (2)

I OpenMx Notation

yi = Cxi + Dui + ri with ri ∼ N (0, R) (3)

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 9: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

State Space ModelMeasurement

I Structural Equation Measurement Model

yi = Ληi + Kxi + εi with εi ∼ N (0,Θ) (1)

I State Space Measurement Model

yi = Ληi + Kxi + εi with εi ∼ N (0,Θ) (2)

I OpenMx Notation

yi = Cxi + Dui + ri with ri ∼ N (0, R) (3)

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 10: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

State Space ModelTransition/Structural

I Structural Equation Structural Model

ηi = Bηi + Γxi + ζi with ζi ∼ N (0,Ψ) (4)

I State Space Structural Model

ηi = Bηi−1 + Γxi + ζi with ζi ∼ N (0,Ψ) (5)

I OpenMx Notation

xi = Axi−1 + Bui + qi with qi ∼ N (0, Q) (6)

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 11: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

State Space ModelTransition/Structural

I Structural Equation Structural Model

ηi = Bηi + Γxi + ζi with ζi ∼ N (0,Ψ) (4)

I State Space Structural Model

ηi = Bηi−1 + Γxi + ζi with ζi ∼ N (0,Ψ) (5)

I OpenMx Notation

xi = Axi−1 + Bui + qi with qi ∼ N (0, Q) (6)

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 12: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

State Space ModelDiagrams

x1

x3

x2

x3

x2

x1

y3

y4

y5

y6

y7

y8

y2

y9

y1

u1 u2

C

R

yD

x

u

AQ

B

Time t Time t+1

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 13: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

State Space ModelDiagrams

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 14: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

State Space ModelAvailable now in OpenMx

I Differential Equation in Discrete Time

I Implemented by me in OpenMx 2.0 Release

I Continuous Time is in OpenMx 2.1 Release

d

dtx(t) = Ax(t) + Bui + q(t) (7)

I Uses Kalman filter

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 15: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Kalman Filter BenefitsB-B-B-Benny and the Fits

I Designed for non-stationary time seriesI Cf. block-Toeplitz autocovariances (Molenaar, 1985)I Cf. lagged observed variables (Song & Zhang, 2014)

I Gaussian noise: gives ML estimates

I Non-Gaussian noise: becomes least squares optimal

I Latent State Estimates are factor scores (Priestley &Subba Rao, 1975)

I Latent Covariace Estimates ⇒ Reliability (Hunter, InPreparation)

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 16: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

6 8 10 12 14 16

1020

3040

5060

7080

Age

Rea

ding

Rec

ogni

tion

Kalman PredictionKalman UpdateObserved Data

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 17: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

6 8 10 12 14 16

1020

3040

5060

7080

Age

Rea

ding

Rec

ogni

tion

Kalman PredictionKalman UpdateObserved Data

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 18: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

6 8 10 12 14 16

1020

3040

5060

7080

Age

Rea

ding

Rec

ogni

tion

Kalman PredictionKalman UpdateObserved Data

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 19: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

6 8 10 12 14 16

1020

3040

5060

7080

Age

Rea

ding

Rec

ogni

tion

Kalman PredictionKalman UpdateObserved Data

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 20: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

6 8 10 12 14 16

1020

3040

5060

7080

Age

Rea

ding

Rec

ogni

tion

Kalman PredictionKalman UpdateObserved Data

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 21: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

6 8 10 12 14 16

1020

3040

5060

7080

Age

Rea

ding

Rec

ogni

tion

Kalman PredictionKalman UpdateObserved Data

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 22: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

6 8 10 12 14 16

1020

3040

5060

7080

Age

Rea

ding

Rec

ogni

tion

Kalman PredictionKalman UpdateObserved Data

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 23: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

6 8 10 12 14 16

1020

3040

5060

7080

Age

Rea

ding

Rec

ogni

tion

Kalman PredictionKalman UpdateObserved Data

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 24: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

6 8 10 12 14 16

1020

3040

5060

7080

Age

Rea

ding

Rec

ogni

tion

Kalman PredictionKalman UpdateObserved Data

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 25: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

6 8 10 12 14 16

1020

3040

5060

7080

Age

Rea

ding

Rec

ogni

tion

Kalman PredictionKalman UpdateObserved Data

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 26: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Kalman FilterEquations

I Predict Step

xt|t−1 = Axt−1|t−1 + But (8)

Pt|t−1 = APt−1|t−1AT + Q (9)

I Update Step

yt = Mean(yt) = Cxt|t−1 + Dut (10)

yt = Residual(yt) = yt − yt (11)

St = Cov(yt) = CPt|t−1CT + R (12)

K = Pt|t−1CTS−1t (13)

xt|t = xt|t−1 + K yt (14)

Pt|t = Pt|t−1 −KCPt|t−1 (15)

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 27: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

What about multiple individuals?

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 28: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Independent Mixture Distributions

P(y1)...

P(yN)

=

P(y1|C = 1) . . . P(y1|C = c)P(y2|C = 1) . . . P(y2|C = c)

.... . .

...P(yN |C = 1) . . . P(yN |C = c)

P(C = 1)

...P(C = c)

(16)

P(yi) =c∑

j=1

P(yi|C = j)P(C = j) (17)

From probability theory: P(yi|C = j)P(C = j) is the jointprobability of yi and C = j, and also that the summation overall possible values of C marginalizes this joint probability

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 29: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Kluckhohn and Murray (1948)

Every man is in certain respects

a. like all other men,b. like some other men,c. like no other man.

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 30: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

All other humans

All

p1 p6p5p4p2 p3

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 31: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Some other humans

All

p1 p6p5p4p2 p3

Some2Some1

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 32: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

No other human

All

p1 p6p5p4p2 p3

Some2Some1

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 33: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Exploratory VersionEFA and Mixture Models are Duals

ModelB

p1 p6p5p4p2 p3

ModelCModelA

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 34: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

ModelB

data1 data6data5data4data2 data3

ModelCModelA

LA1

LA2

Mixture Classes

People ( T > 1)

LC5

LC6

p A LA1+ pB LB1+pC LC1 ...p A LA2+ pB LB2+ pC LC2 p A LA5+ pB LB5+ pC LC5 (LA6LB6LC6

)T

(p ApBpC)

(LA1 LB1 LC1LA2 LB2 LC2⋮ ⋮ ⋮LA6 LB6 LC6

)(pApBpC)=(L1

L2

⋮L6)→∑ Li=L

Mixture Model

Total Model

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 35: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Simulation StudyData Generation Parameters

I Number of Groups/Clusters (4 levels): 1, 3, 5, 8

I Number of People Per Group (3 levels): 1, 10, 100

I Number of Occasions (4 levels): 5, 12, 50, 200

I Number of Factors (4 levels): 1, 3, 4, 8

I Number of Variables Per Factor (4 levels): 1, 3, 4, 6

I Goal: 1,000 reps of each of the 768 conditions

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 36: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Simulation StudyKey Questions

I Can state space mixture models be estimated at all?

I Can parameters be estimated at all?

I What influences the quality of parameter estimation?

I Can underlying groups be recovered at all?

I What influences the quality of group membershiprecovery?

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 37: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Can state space mixture models be estimated?Condition Completion

0 200 400 600

020

040

060

080

010

00

Condition Number

Rep

etiti

ons

Don

e

Reps: M = 630.7, Med = 1000; 484,396/768,000=61.6%.Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 38: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Can state space mixture models be estimated?Condition Completion

●●●●●

●●

●●●●●

●●

●●●●●

●●

●●●●●

●●

●●

●●●●●

●●

●●

●●●●●

●●

●●●●

●●

●●●●●

●●●●●

●●

●●

●●

●●●●●

●●

●●●●●

●●

●●

●●

●●●●●●●●

●●

●●●●●

●●

●●

●●●●●

●●

●●●

●●●●●

●●

●●●●

●●●●

●●

●●

●●

●●

●●●●●

●●

●●

●●●●●

●●

●●●●

●●

●●●

●●●●●●●●

●●●●

●●●●●●●

●●

●●●●●

●●●●

●●

●●●

●●●●●

●●

●●●●●

●●

●●●●●

●●

●●

●●

●●

●●

●●

●●●●

●●●●●

●●

●●●●●

●●

●●

●●●●●

●●

●●

●●●●

●●●

●●●

●●

●●●

●●●●

●●●●●

●●

●●●●●●●●

●●●●

●●●●●●●●

●●

●●

●●●●●●●

●●●●

●●

●●●●

●●●

●●●●●

●●●

0 200 400 600

−2

02

46

810

12

Median Time to Estimate Models for Each Condition

Condition

Loga

rithm

of W

all T

ime

1 Group 3 Groups 5 Groups 8 Groups

1 Person Per Group

10

100

Range: 0.1 seconds to 48 hours.Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 39: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Can state space mixture models be estimated?Status Codes

0 200 400 600

0.0

0.2

0.4

0.6

0.8

1.0

Proportions of Convergence Codes for Models by Condition

Condition

Pro

port

ion

of M

odel

s

Code 0: ConvergedCode 1: Status GreenCode 4: Status BlueCode 6: Status RedMissing: Terminated

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 40: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Can parameters be estimated at all?

Overall True−Start vs True−Estimated Correlation

Correlation

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

050

000

1000

0015

0000

2000

00

Overall True−Start vs True−Estimated RMS

Root Mean Square Difference

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

020

000

4000

060

000

8000

010

0000

1200

00

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 41: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Influences on parameter estimationNumber of Times

numOccasions = 5

Root Mean Square Difference

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

010

000

2000

030

000

numOccasions = 12

Root Mean Square Difference

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

010

000

2000

030

000

numOccasions = 50

Root Mean Square Difference

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

010

000

2000

030

000

numOccasions = 200

Root Mean Square Difference

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

010

000

2000

030

000

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 42: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Influences on parameter estimationNumber of People Per Group

numPeoplePerGroup = 1

Root Mean Square Difference

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

020

000

4000

060

000

numPeoplePerGroup = 10

Root Mean Square Difference

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

010

000

3000

0

numPeoplePerGroup = 100

Root Mean Square Difference

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

010

000

3000

0

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 43: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Influences on parameter estimationNumber of Groups

numGroups = 1

Root Mean Square Difference

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

020

000

6000

0

numGroups = 3

Root Mean Square Difference

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

010

000

2000

030

000

numGroups = 5

Root Mean Square Difference

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

050

0015

000

2500

0

numGroups = 8

Root Mean Square Difference

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

040

0080

0012

000

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 44: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Can underlying groups be recovered at all?Micro-Accuracy

Histogram of Group Recovery AccuracyMore Than One Group

Accuracy

Fre

quen

cy

0.6 0.7 0.8 0.9 1.0

010

000

2000

030

000

4000

050

000

mean = 0.86

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 45: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Can underlying groups be recovered at all?Kappa

Histogram of Cohen's kappa

Cohen's kappa

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

020

000

4000

060

000

8000

0

mean = 0.58

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 46: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Influences on group recoverySimulation Factor Level Kappa AccuracyNumber of Groups 3 0.63 0.84

5 0.57 0.868 0.50 0.89

Number of People Per Group 1 0.58 0.8610 0.57 0.86

100 0.59 0.84Number of Occasions 5 0.52 0.84

12 0.56 0.8650 0.61 0.87

200 0.63 0.87Number of Factors 1 0.43 0.81

3 0.59 0.874 0.65 0.888 0.73 0.90

Number of Variables Per Factor 1 0.63 0.883 0.57 0.864 0.56 0.856 0.52 0.84

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 47: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Influences on group recovery

Estimate Std. Error z value Odds Ratio(Intercept) -0.58 0.0030 -194.5 0.56numGroups5 -0.34 0.0016 -208.2 0.71numGroups8 -0.62 0.0015 -405.9 0.54numPeoplePerGroup10 0.05 0.0025 20.7 1.05numPeoplePerGroup100 0.12 0.0025 47.4 1.13numOccasions12 0.30 0.0016 187.9 1.35numOccasions50 0.54 0.0017 312.4 1.71numOccasions200 0.69 0.0019 357.3 2.00numFactors3 1.05 0.0015 688.5 2.85numFactors4 1.42 0.0017 845.5 4.14numFactors8 2.02 0.0024 826.4 7.53numVarPerFactor3 -0.09 0.0016 -56.8 0.91numVarPerFactor4 -0.10 0.0017 -58.5 0.91numVarPerFactor6 -0.17 0.0018 -96.0 0.84

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 48: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

NLSYNational Longitudinal Survey of Youth

I Original 1979 Generation 1 SampleI Nationally representative household probability sampleI 12,686 young men and womenI 14-22 years old

I Children (NLSYC, Generation 2)I Children of Generation 1 FemalesI Beginning 1986I 3276 different mothersI 11,075 NLSY-Children kinship links

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 49: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Present Analyses

I Goal: look at longitudinal variation in cognitive abilityusing biometrically informed models

I 5 cognitive variablesI PIAT Math, Reading Comp, Reading Recog (Ages 5-14)I PPVT (Ages 3-14)I WISC Memory for Digit Span (Ages 7-11)

I Modal Number of Occasions: 2-4

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 50: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Parameter EstimationNLSY: numGroups (1,3), numOccasions (5), numFactors (4, 8),numPeoplePerGroup (100), numVarPerFactor (1, 3)

Overall True−Start vs True−Estimated Correlation

Correlation

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

010

0020

0030

0040

00

Overall True−Start vs True−Estimated RMS

Root Mean Square Difference

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

050

010

0015

0020

0025

00

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 51: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Group RecoveryNLSY: numGroups (1,3), numOccasions (5), numFactors (4, 8),numPeoplePerGroup (100), numVarPerFactor (1, 3)

Histogram of Cohen's kappaCases Matching the NLSY

Cohen's kappa

Fre

quen

cy

0.0 0.2 0.4 0.6 0.8 1.0

050

100

150

200

250

300

350

mean = 0.84

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 52: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Summary

I Between- and Within-person variabilities are distinct

I Within-person models are needed for within-personconclusions

I Proposed method that hybridizes within- andbetween-person modeling.

I Simulation found parameters can be recovered, samplesize helps the most.

I Simulation found group membership can be recovered,dimension of dynamics helps the most.

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 53: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Weaknesses

I Many alternative techniques are not triedI Latent Differential EquationsI N -way factor analysisI Cross-sectional Pooled time seriesI Multilevel (structural equation) modelsI Machine learning

I Simulation studyI Violation of assumptions not tested: Normality,

Linearity, Model ErrorI Dynamics generated are simple

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 54: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Conclusions

I A topic of growing interestI Gonzales and Ferrer (2014)I Voelkle, Brose, Schmiedek, and Lindenberger (2014)I Zhang and Wang (2014)I Voelkle and Oud (2014)I Steele, Ferrer, and Nesselroade (2014)I Song and Zhang (2014)I McArdle, Hamagami, Chang, and Hishinuma (2014)I Babbin, Velicer, Aloia, and Kushida (2015)

I Preliminary evidence is extremely encouraging

I A promising beginning

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 55: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Acknowledgments

I Joseph L. Rodgers, Hairong Song, & David Bard

I OpenMx Core Development Team

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 56: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Thank [email protected]

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 57: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

Babbin, S. F., Velicer, W. F., Aloia, M. S., & Kushida, C. A.(2015, Jan). Identifying longitudinal patterns forindividuals and subgroups: An example with adherenceto treatment for obstructive sleep apnea. MultivariateBehavioral Research, 50(1), 91-108. doi:10.1080/00273171.2014.958211

Gonzales, J. E., & Ferrer, E. (2014, May). Individual poolingfor group-based modeling under the assumption ofergodicity. Multivariate Behavioral Research, 49(3),245-260. doi: 10.1080/00273171.2014.902298

Kluckhohn, C., & Murray, H. A. (1948). Personality in nature,society, and culture. AA Knopf.

McArdle, J. J., Hamagami, F., Chang, J. Y., & Hishinuma,E. S. (2014, Jul). Longitudinal dynamic analyses ofdepression and academic achievement in the hawaiianhigh schools health survey using contemporary latent

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 58: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

Introduction Kalman HOWTO Simulation Study NLSY Simulation Discussion References

variable change models. Structural Equation Modeling:A Multidisciplinary Journal, 21(4), 608-629. doi:10.1080/10705511.2014.919824

Molenaar, P. C. M. (1985). A dynamic factor model for theanalysis of multivariate time series. Psychometrika, 50,181-202. doi: 10.1007/BF02294246

Priestley, M., & Subba Rao, T. (1975). The estimation offactor scores and kalman filtering for discrete parameterstationary processes. International Journal of Control,21(6), 971–975. doi: 10.1080/00207177508922050

Song, H., & Zhang, Z. (2014, Jan). Analyzing multiplemultivariate time series data using multilevel dynamicfactor models. Multivariate Behavioral Research, 49(1),67-77. doi: 10.1080/00273171.2013.851018

Steele, J. S., Ferrer, E., & Nesselroade, J. R. (2014, Oct). Anidiographic approach to estimating models of dyadic

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 59: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

interactions with differential equations. Psychometrika,79(4). doi: 10.1007/s11336-013-9366-9

Voelkle, M. C., Brose, A., Schmiedek, F., & Lindenberger, U.(2014, May). Toward a unified framework for the studyof between-person and within-person structures:Building a bridge between two research paradigms.Multivariate Behavioral Research, 49(3), 193-213. doi:10.1080/00273171.2014.889593

Voelkle, M. C., & Oud, J. H. L. (2014). Relating latent changescore and continuous time models. Structural EquationModeling. doi: 10.1080/10705511.2014.935918

Zhang, Q. J., & Wang, L. P. (2014, Mar). Aggregating andtesting intra-individual correlations: Methods andcomparisons. Multivariate Behavioral Research, 49(2),130-148. doi: 10.1080/00273171.2013.870877

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 60: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

require(OpenMx)

data(demoOneFactor)

nvar <- ncol(demoOneFactor)

unam <- colnames(demoOneFactor)

amat <- mxMatrix("Full", 1, 1, TRUE, .3, name="A",

labels="a11")

bmat <- mxMatrix("Zero", 1, 1, name="B")

clab <- paste("load", 1:nvar, sep="")

cdim <- list(unam, "F1")

cmat <- mxMatrix("Full", nvar, 1, TRUE, .6, name="C",

dimnames=cdim, labels=clab)

dmat <- mxMatrix("Zero", nvar, 1, name="D")

qmat <- mxMatrix("Diag", 1, 1, FALSE, 1, name="Q")

rlab <- paste("resid", 1:nvar, sep="")

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 61: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

rmat <- mxMatrix("Diag", nvar, nvar, TRUE, .2,

name="R", labels=rlab)

xmat <- mxMatrix("Zero", 1, 1, name="x0")

pmat <- mxMatrix("Diag", 1, 1, FALSE, 1, name="P0")

umat <- mxMatrix("Zero", 1, 1, name="u")

ssModel <- mxModel(model="State Space Manual Example",

amat, bmat, cmat, dmat, qmat, rmat,

xmat, pmat, umat,

mxData(observed=demoOneFactor, type="raw"),

mxExpectationStateSpace("A", "B", "C", "D",

"Q", "R", "x0", "P0", "u"),

mxFitFunctionML()

)

ssRun <- mxRun(ssModel)

summary(ssRun)Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 62: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 63: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

ds <- mxGenerateData(ssModel, nrows=30)

ssModel2 <- mxModel(ssModel, mxData(ds, 'raw'))

ssRun2 <- mxRun(ssModel2)

summary(ssRun2)

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 64: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 65: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

0 20 40 60 80 100 120

−5

05

10

Time

x1

0 20 40 60 80 100 120

−5

05

10

Time

x2

0 20 40 60 80 100 120

−5

05

10

Time

x3

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 66: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

0 20 40 60 80 100 120

−5

05

10

Time

y1

0 20 40 60 80 100 120

−5

05

10

Time

y4

0 20 40 60 80 100 120

−5

05

10

Time

y7

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 67: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

●●

● ●●●●

●● ● ●● ●●● ●

● ●

●●

● ●

●●

● ●●

● ●

●●

●● ●

● ●

● ●

● ●

●●●● ● ●●● ● ●● ● ● ● ●● ● ●

● ●

● ●

●●● ●● ●

● ●● ● ● ●● ●● ● ●

●●

● ●●

●●

● ●

●●● ●●

● ●

●●

●●

●● ● ● ●

● ●●

●●●

●●

●● ●●● ●

●● ● ●

● ● ●●●● ●

●●

●● ● ●●●

● ●

● ●

●● ●

●●● ●

●●

●● ●● ● ●● ●●

●●● ●

● ●

●●

● ●●

● ●● ● ●●

● ●

● ●

●●

●●

● ● ●

●●●

●●

● ●●

● ●

● ●

●●

● ●● ●

● ●● ●

●●

● ● ●●● ●

●● ● ●●

●●

●●

●●● ●

● ●

●●

●●

●● ●

●●●

● ●●

●● ● ●

●●

● ●

●●●●

●●●

●● ● ●●● ●

●● ●●● ●

●●

● ●

● ●

●● ●●

●● ● ●

●●

●●●

● ●

● ●●

●● ●

●● ● ●

● ● ●●

● ●

●●

●● ●●

●●

●●●

● ● ●●● ●

● ●

● ●

●●●

● ●●

●●● ●

●● ● ● ●

● ●

● ●●●

● ●

● ● ●●

●●● ● ●

●●

● ●●

● ●

● ●● ●●

●●● ● ●

● ●

● ●● ●

●●

●●● ●

●●

● ● ●●●

●●

●● ● ●● ●●● ●●

● ●● ●

●● ● ●●

●●

● ●

●●●

●● ●

● ●● ● ●

● ● ●

●● ●●●● ●

●● ●●●

●●●●●

●●

● ●●

●●

● ●

● ●●

● ●●

● ●●

● ●

●● ● ●●●

●● ● ●●●

●● ●● ●

●●

●●

● ●● ●

● ●●

● ●

● ●● ●

● ● ●

● ●

● ●

●● ●● ●

● ●

●●●●

● ●

● ●●● ●● ●

● ●

● ●

● ●

●● ●

● ●

●● ● ●●

●●● ●

● ● ● ●

●●

●●

● ●

●● ● ●

● ● ●

●●

●● ●● ●●

●●

● ●● ●●

● ●● ●

● ●

● ●

● ● ●

●● ●

● ●

●●● ●

● ●●

●●

●● ●● ●

●●●● ● ●

● ● ●

●●

● ● ●●

●●

● ●

●●●●

● ●● ●● ●●

● ●● ● ●

● ●

●●● ●

●●

●●● ●●

●● ●●

●● ●● ●● ●

●●●● ●

●●●

●● ● ●

● ●●

●●●

● ●●

●●● ●● ●

●● ●

●●●

●●

●● ● ●●●●●

●● ●

●● ● ●●

● ●

●●●

● ●

●● ●●

●● ●●● ●●● ●

● ● ●●

●● ● ●● ●

●● ● ● ●●

● ● ●

● ●

● ●●

● ●●

●● ●●

●●

● ● ●●● ●

●●

●●● ●

● ●

●●●

● ●

● ● ●

● ●

●● ●

●● ●● ●● ● ●

● ●

● ●● ● ●● ●●

● ●●

● ● ●●

● ●● ●

●●

● ● ● ● ●●

● ●● ●●● ● ●

●● ●●● ●

● ●

● ●

●●

● ●

● ●●●●● ●

● ●

●●

● ●●● ●●

● ●

●● ●

●●

●●

●●

● ●

●●

● ●● ● ●●●●

● ●

● ●●●● ● ● ●● ●

●● ● ●● ●

●●

●●

●●

●●●●

●●

● ●● ●

●● ●

●●● ●

●● ●●

●●● ●

●● ●

●●

●●●

● ●●●

● ●● ●●●

● ●

●●●● ● ●

● ●

● ●● ●

●● ●●●●

● ●●● ●

●●

● ●● ●●

●●● ●

●● ●● ●

● ● ●

●● ● ● ●●

● ●● ● ●●● ● ● ●

● ●● ●

●● ●

●●

● ●

●●● ●●●

●●●● ● ●

●●●

●●● ●●

●●●●

●●

●●

●● ● ●

●● ●

● ● ●●

● ●●

● ●● ●

● ●

● ●

●● ● ● ● ●

● ●

●● ●

●● ● ●● ●

●● ●

●●● ●●

● ●

●● ●

● ●

● ● ●

●● ●●● ●

●●

● ●

●●● ●

●●●●

●● ● ●● ●

●● ●● ●●

● ●

●●

● ●

●●

● ● ●

●●● ● ● ●

● ● ●

●●

●●● ●

● ●

●●

●●

●●

● ●

●●

●●● ●●

●●

● ● ●● ●

●●

●●

●● ●● ●

● ● ●● ●

● ● ●

● ●

● ● ●●

● ● ●

●●

●●● ●●●

● ●● ●

● ●●● ● ●●

●●

●● ●●●●

●● ●

● ●● ● ●●

● ●

● ●●

● ●

●●

●● ●

● ●

●●

● ●

●● ●

●●● ●●●● ●● ●

● ●

● ●●

●●

●● ●●●

● ●

● ●

● ●

●●

● ●

●● ●● ●

● ●●●

● ●

●● ●● ● ●●

●● ●●

●●

● ●

●●●

●● ● ●● ●

● ●●● ● ●

●●●●● ●

● ●

● ● ●●●● ●

● ●

● ● ●●

● ●

●● ● ●●●● ●

● ●●●●

● ● ●●

●● ●

●● ●●●●●

● ●●● ●

● ● ●●● ●

●● ●●

● ●● ●

●● ●

●● ●●

●●

● ●

●● ●

● ●

●●

●● ●●● ● ●●

●● ● ●

● ● ●

●● ●● ●●

● ●

●● ●

● ● ●●

● ●

●●

● ●●●●

●● ● ●●● ●

●●

● ●● ●

●●

● ●

●●

● ●●● ●●●●

●● ●●●

●●

●● ●● ●

●● ●

● ●● ●

●● ●

●● ●●

●● ●

●● ●

● ●

●●● ●

● ● ●●

●● ●● ● ●

●● ●●

● ●

● ● ● ●

● ●●● ● ●

●●

●●●

●●●

●●● ● ● ●●●●

● ● ●

●●● ●● ●

● ●

●● ●

●●● ●●

●●

●●

● ●● ●● ●●

●●●

●●●● ●

●● ●● ●●

●●

● ●

● ●

●●

●●

● ●●

●● ●● ●

●●

● ●

● ● ●

● ●● ●

● ●

● ●

●●

● ●

●●●● ●●

●● ●

● ●● ●

● ●

●●

● ●

●●

●●

●●●

●●● ●

● ●

●●

●●

● ●●

● ●● ● ●● ● ● ●

●● ●● ●

● ●●

●●

●● ●

●● ●●● ●● ●● ●●●● ● ●● ●●

● ● ●● ●●●

●●

●● ●

●● ●

●● ● ●

●● ●●●● ●● ●

●●

● ●●●

● ● ●

● ●●

●● ●

●●

●●●

● ● ●●●

●● ●

●●

●● ● ●

● ● ●● ●

● ●

● ●

●●

●●

● ●

● ●

●●● ●●● ●●

● ●●

● ●●

●● ●● ● ●●●

●●● ● ●

●●

●●

●●

●●

●● ● ● ● ●●● ●

●● ● ●

● ●

● ● ●● ●

●●

●●

● ●● ● ●

●●

● ●

● ● ● ● ●● ● ●

●●

● ●

● ● ●

●●

● ●● ●● ● ●● ● ●● ●

● ●●

●●●● ●● ● ●

●●●

● ●● ●●

●●

● ●● ●● ●● ● ● ●●● ●●●

●● ●● ●● ● ● ● ●

● ●●

● ● ●

●● ●● ●●

●● ●●

●● ●●

●● ●● ●

●●

● ● ●

● ●●

● ●● ●

●● ●

●●

●● ●

● ●

●●●● ● ●● ●●●

● ●● ●

●●

●●● ● ●

●●

●● ●● ● ●

● ●● ●●

●● ●● ●

● ● ●●●● ●

● ●

●● ●

● ●

●● ●●

● ● ●● ● ●

● ●

● ●●

●●● ● ●

●● ●

● ● ●● ●●●

●● ●

●● ● ●

● ●

●●

●● ●●●

● ●● ● ●●● ●● ●

●●●

●● ●

●●

● ●●

● ●

●●

● ●

● ●

●●

● ●● ●●

●●

●● ●

●● ●● ●

● ● ●● ● ●●

●● ● ●●

●●

● ●●● ● ●

● ●

● ●

● ● ●●

● ●●

●●● ●

●●● ●●

●●● ●

● ●●● ●●

● ●● ● ●

● ●●

● ●●

● ● ●

● ● ●● ●●●●● ●

● ●● ●● ●●

● ●

● ●

● ●●

●●● ●●

● ●

● ●●●

● ●● ●

● ●●

● ●

● ●

● ●● ●

● ●●● ●●● ● ●

● ● ●

● ●

● ●

● ● ● ●

●●●

●●

● ● ●● ●●● ● ●●● ● ●

● ●

● ● ●●●● ●●●

● ●●

● ● ● ●● ●●●

●●

●● ●● ●● ●● ●● ● ●●

● ● ●●

● ● ●

● ●●●● ● ●● ●● ●

●●

● ●

●●● ● ●

●● ●

● ●● ●●● ●● ●

●● ●

● ●

● ●

●●

● ●

●● ●●

●●

● ● ●

●●

●● ●● ●●

●● ●

●●

● ● ● ● ●● ●●

● ●

●● ●●

●●●

● ●

●●●

●●●●

● ●● ● ●● ●●

● ● ● ●● ●●

● ● ●

● ● ●●● ● ● ●●

● ●●●● ●

●●

● ●●

●●● ●

●●

● ● ●

● ●

● ●

●●

● ● ● ●●

●● ● ●● ●● ●●

● ●

●●

●●

● ● ●●●

●● ●

● ●●

●●

● ●● ●

● ● ●●● ●●●

● ●●

●●● ●

●●●

● ●

●●

●● ●

● ●

●● ●●

●● ●

●●

● ●

●●● ●

● ●●●

●●

●●● ● ●●

● ●

●●● ●

● ●

●●●

●● ●

●●

● ●● ●

●● ●

●●● ●

● ●

●● ●● ●

●●●

●●●● ●●

● ●●

●● ●● ●● ● ●●

●● ●

● ●

● ●●

●● ● ●●

● ●

● ●

●●

● ● ●

●● ●

●●

● ● ●

●●

● ●

●●

●●

●● ●

●●●

● ●

● ●● ●

● ●

● ●● ●

●●

●●●

●●

●●

● ●●●●

●● ●●● ●● ●

●●

● ●

● ● ●● ● ●● ●● ●● ●

● ●

●●● ●

● ●

● ●●

● ● ●

● ● ●●●

● ● ●

● ●●● ●

● ●

● ● ●●

● ●

●●●

● ●

● ●● ● ●● ● ●●

● ● ● ●●

●●●

● ●

●●

●●● ●● ●

● ●●

●●

●●

●●● ● ●

● ● ●

● ●

●●

●● ●

●●

●● ●● ● ●● ● ●● ● ●

● ●

● ●●●

● ●●●

● ●● ● ●●

●● ●

●● ●

● ●

●●●

● ●

●●

● ●

●● ●

● ●

● ●

●● ●●

● ●●●

● ● ●●●

● ● ●● ●●

●●● ●

●● ●● ● ●

●● ●

● ●

●● ●● ●● ●● ●

● ●

● ●

● ●

● ●●

●● ●● ●

● ●

●●

●●

●●

● ● ●

● ●●● ●

●● ●● ●

●● ●

●●

●●●

● ● ●●● ●● ●

●● ● ●● ●●

●●

● ●●

● ● ●

●●● ●●

● ●

● ●

●● ●

● ●

●●● ●

● ●●●●

●●●●●

● ●

● ●● ●●● ●● ●● ● ●

● ●

●●

●● ● ●●

●● ●

●●

●●

● ●

●●

●●

●●

● ● ●● ●●

●●

● ●

●●

●● ●

●●

● ●

● ●● ●

● ●

● ●

● ● ● ●● ●● ●

●●● ● ●● ●●

●● ● ●● ●

●●● ●● ●

● ●●

● ●

●●

● ● ●

●●

● ●●

● ●

●● ●

● ● ●

● ●

●●●

●●

● ●

● ●●●

● ●●●

●●

●●●

●● ●

●●●

● ●

●●●

●●●●

● ●

●● ●●

●● ●

●●● ●

● ●

●● ●

●●●

● ●● ●

●● ● ●●

● ●

● ●

● ●

●●● ● ●

●●

●●

●● ●

●●● ●● ●

● ●●● ● ● ●

●●

● ● ●● ● ●●

● ●●● ●●

●● ●

● ● ●●

●● ●

●●

●● ●

●● ●●

● ● ● ●●

● ●

●●

● ●

●● ● ●● ●

● ●●

● ●●

●● ●●

● ●

●●

● ●● ●●

●● ●

● ●

● ●●●

● ● ●● ●● ●● ●● ●● ●

● ●

● ●● ●

●●

●●

● ●●

●●

●●● ●● ●●●

●●

● ●●●

●●

●● ●●

● ●

● ●

●● ●● ● ●● ●● ●

●●

●● ●●●

●●

●●

●●

●● ● ●

● ●●●●

●●

●● ●

●●●

● ●● ●● ●

●● ●● ●

●● ●●

● ● ●● ●

● ●

●●● ●●●

● ●

● ● ●●●● ● ●●

●●●

● ●

●● ●●● ●● ●●

●●

● ●

●● ●●●

●●●● ●● ● ●● ●●●

● ●● ●●● ● ●●● ●●

●● ● ●● ●●●

●●●

● ●

●● ●●

● ●

● ●●

●● ●●

● ● ●

●●

● ●

●●● ● ●●

●● ● ●

●●

● ●● ● ●● ●●

●●●●●

●● ●● ●● ● ● ●● ●

● ●● ● ●●

● ● ● ●

● ● ●

● ● ● ●

●● ●

● ●●

● ●● ●

●● ● ●●

●●

● ●

● ●

● ●● ● ●

● ●●

● ●● ●

● ●● ●● ● ●

●●● ● ●●● ● ●● ●

● ●●

● ●

●●

●●● ●

● ●

●● ●

●● ●●● ●● ●

●●●●

● ●

●●●●

●● ●● ●

● ●●

●●

●●

●●

●●●

● ●● ●●

●● ●

●● ●

● ●● ●●

●●

●●

●● ●

● ● ●

●●

●●●

●● ● ● ●●●

●●

●● ●●

●●

●● ●●

●●

● ●

●●

●● ●

●● ● ●

●●

●●

●● ●●

●● ● ●● ●

●●●

●● ●

●● ●●●● ●

●● ●●

● ●●

●●

● ● ● ●● ●

● ●●

● ●●● ●●● ●

● ●●

● ●●

●● ●● ●

●●

● ● ● ●● ●●● ●● ●

●● ●●

● ● ●● ●●●

● ●

● ●●

●●● ●

● ●●● ●

●● ●

●● ●

●● ● ●

●● ●

● ●● ●●

● ●

●● ●● ●

● ●● ● ●●●

●●

●●●● ●● ●●

●●

●●●

● ●●

●● ●●●

●●

●●

● ●●●

● ●●

● ● ●● ● ●●● ●●

●● ●

●● ●

●●●● ●● ●

●● ●●● ●●

● ●

●●● ●●● ●●

● ●●

●● ●

● ●●● ●

●●

● ●●●

●●

●● ●●●

● ● ●● ●

●●●

● ●

● ●●●

● ●

●● ● ● ● ●

●●●●● ● ●

●●

● ●●

● ●●● ●

●● ● ●●

●●

●● ●●

● ● ●● ● ●●

●● ●●

● ● ●●● ●●

●● ●● ●

● ●

●●●

● ●●

●● ●●

● ●

●●●

●●

●●

● ● ●●

●●●

● ●●

● ●●

●● ●

● ● ●● ● ●● ●● ● ● ● ●● ●● ●

● ● ●● ● ● ●●

●● ●●

● ●

● ●

● ●●

●●● ●● ●●●

●●

● ● ●

●●

● ●●

● ●● ●●● ●● ●

●●● ●●

● ●

●●

●●

● ●●● ●● ●●

●●● ●● ●

● ●

●●

● ●● ●

●●

● ●

● ●

●● ●●● ● ●

● ● ●

●●

● ●●

●●

● ●● ●● ●

● ●● ●● ● ● ●●●

●● ●●● ●● ●

●●●

● ●● ●● ●●● ●● ●●● ●

●● ● ●

●● ● ● ●

● ●

●●

●● ● ●●

●●●● ●

● ● ●

●●

●● ●● ● ●

● ●● ● ●● ● ●●

●● ●●

●● ● ●

●●●

●●● ●

● ●

● ●●

● ●

● ●

● ●

●● ●● ●●

●●

● ●

● ●

● ● ●●●

● ●

● ● ●●

● ● ●

●●

● ●●

●● ●●

●●●● ●●

● ● ●

●●● ● ●

●● ●

●●

●● ●

● ●

●●

● ●

●● ●

●●●

● ●●

● ●● ● ●● ●● ●

● ●

● ●●

●●

● ●

●●●● ● ●

● ●

●●

●● ●

●●

●● ● ●● ● ●● ●●

●●●

● ● ● ● ●●

●●

● ● ●

● ●

●●● ● ● ●●

●●

● ●● ● ●●

●●● ●

●●● ●

●● ●●

● ●

●●

● ● ●

●●

● ●●

● ● ●● ● ● ●

● ●● ●●

● ●

● ●

●●

● ●

●●

● ●

●● ●●●● ●●●

●●

● ●

●●● ● ●●● ●

●● ● ●●

● ●

●● ●

●● ●

● ●●

● ●●

●● ● ●●

●● ●●

●● ●●●●●●● ●

●● ● ●● ●

● ●● ● ●

●●● ● ●

●● ●

● ●●

●● ● ●● ●

● ●●

●● ●●● ●● ●●

● ●

● ●● ●

● ● ●

●●●

● ●

●● ●● ●

●●●

● ● ●

●● ●●

●●

● ●

● ●

●●● ●●●● ●

●● ● ●●●

●●

●● ●

●● ●●

● ● ●●

●●● ●

● ●●

● ●●

●● ● ● ●●●

● ●● ●●

● ● ●

● ●

●●

● ●

● ● ●

● ●

●●

●●

● ●● ●● ●●● ●● ●

● ●

●●●● ●

● ●●● ●● ●

● ●●

● ●●

● ● ●●

●● ●

●●● ● ●

●●

● ●

● ●

●●

●● ●● ● ● ●● ●

● ●

●●●

●● ●

●●●

● ●● ●

●● ● ●● ●

●●

● ●●●●

● ●

●● ●● ●

● ●

●●● ●

●●●●●● ●● ● ● ●

● ●

● ● ●●

● ●● ●●

● ●

●●

● ●

●●

● ●

●●

● ●

●●● ● ●● ●

●●● ● ●

●● ● ●●

●●●

●● ●●

● ● ●● ●

●●●

●●●

● ●

●● ● ●●● ●

●●

● ●

● ●● ●

● ●●

●●

●● ● ●

● ● ●●

● ●●

● ●●●

● ●

● ● ●●

●● ●

● ●● ● ●● ●

● ●●

●● ● ● ● ●

●●●

● ●●

●● ●

● ● ●

● ● ●● ●

●● ●

● ●

●●

●●

●●

●● ●●

●●

● ●●

● ● ●● ●●●● ●● ●

●● ● ●●● ●

●●

●●●

● ●

●●● ●

●●

● ●●

● ●

● ●●

●● ●●

●● ●●●

●●●

●●●

● ●

● ●●

●●

● ● ●

● ●●

● ●

●●●

● ● ●● ●● ●

● ●

●●

● ●

● ● ● ●● ●●●

●●

● ● ●

● ●●

● ●

●● ● ●●● ●

● ●● ● ●●

●●●● ●●

●●● ●

●●● ●

●●

● ●● ●

●●

●● ●

●● ●

●●●

●● ● ●●

● ●●

●●●

● ●●

● ● ●● ●●● ●

● ●

●● ●●

● ● ●

●● ●●

● ●● ●●

●● ●

●●●

●●

●●

● ●●●

●●

● ● ●●

●● ●●

● ● ●●●

● ●●

●●● ●● ●

●●

●● ●●●

●●

●● ●

● ●

●● ● ●● ●

● ●

●● ●●●● ●●●● ●

●● ● ●●

● ●●

● ● ● ●

● ●

● ●●

●● ● ●●

●●●

● ●

● ● ● ●

● ●

●●

●● ●

●● ●●

●● ● ●●●●

● ●

●●

● ●● ●

●●

● ●● ●●●

● ●

●●● ●● ● ● ●

● ●●

●●● ●

●●● ●

● ●●

●● ●

● ●●● ●● ●●● ●●● ●●● ● ●●

●●

●●

●●● ●

●● ●●

●● ●●●

● ● ●

●●

●● ● ●●

●●

● ●

● ●●●

● ● ● ●●●

● ●

●●● ● ●●●●●

●●

●●

● ●● ●●● ● ●● ●●●●

● ●

●● ●

●● ●

● ●●

● ●●

● ●●

●● ●

● ● ● ●●

● ●● ●●

● ●

●● ● ●

● ●

●●

● ●

● ●

●● ●

● ●

● ●●

●●

● ●

● ●● ●

●● ● ● ●

●● ●●

● ●

● ●

●●

●●●

● ●

● ● ●● ●

●●

●●

● ●●

● ●●● ●

●●

●●

●●

●● ● ●

●●

●●●

●●

●●● ●

● ●●

● ●●● ●

● ● ●●

● ●

● ●

●● ●

●●● ●

● ●

●●

● ●● ●●

●●

● ●●

●●● ● ● ●

● ●

● ●

●●

●●

● ● ●

●●

●●

● ●●

● ●

●●

● ●

●●●

● ●●●

●● ●● ●

● ●

●●

●●● ●

●●

● ●

● ●

●● ●●

● ●●

● ●●

●● ●

●● ●●

●●

●●

● ●

● ● ●

● ●

● ●●

●● ●

● ●●

● ●

●●

● ●

●●

●●

● ● ●●● ●

● ●

● ●●

●●●

●●

●●●

●●

●●

● ●

● ●●

● ●● ●●

● ●

● ●●●

●● ● ●

● ●

●●

●●

● ●

●●

●●●● ●●● ●● ●

● ● ●

● ●●

●●

● ●

● ● ●

● ●

●●

●●

●●●

● ●

●●

● ●● ●●

● ● ●

●●

●●

● ●●

● ●

● ● ●

● ●

● ●●●

●●●

●● ●●

●●

● ●

● ● ●

● ●

●● ●

● ●●

● ●

●● ●

●● ●●

● ●

●● ●

●●

● ●

●●

● ●● ●

●●●

●●

●●

●●

●● ●● ●●

●●

●●

● ●

● ●● ●

● ●

● ●

● ●●

● ●

● ●

● ● ●

● ●

● ● ●● ●

●●● ● ●●

● ● ●

● ●

● ● ●●

●●●

● ●● ●

●●

●●

● ●

● ●● ●

●●

●●

●●

● ●●●

● ● ● ●

●● ●●

●●●

● ●

● ●●

●●

● ● ●

● ●● ●

●● ●● ●● ●●

● ●● ●●

●●

● ●

● ●

● ● ● ●

● ●● ●● ●● ●●

●● ●

● ●●

●● ●● ●●

● ●

● ●●

●●

● ●

●● ●

● ●● ●

●●

●●

●●

● ●

● ●● ●

● ●

●● ● ●●●● ●

● ●

● ●

● ●

●●● ● ●

● ●●●●

● ●

● ●●

● ●

●●

●●

●●

●● ●●●● ●

●●

● ● ●

● ●

●●

● ●●

●●

●●

●●

● ●● ●●●●

● ● ●●

● ● ●● ●

●●

●●

●● ●

●●

● ●

●●

●●

●●●●

●● ● ●

●● ●

●●

● ●●

●●

● ● ●

●●

● ●

● ●●●

●●●

●●

●● ● ●●

●●

● ●

●●

●● ● ●●●

● ● ●●

● ●

● ●

●●

● ●

●● ● ●●●

● ● ●● ● ●

●●

● ●● ●

● ●●●

●●

●● ● ●

● ● ●

● ●

● ●●●

● ●● ●

●●

● ●

● ●

● ●

●●

●● ●

● ●

●●●

● ●●● ●

● ●●

● ●●●

●● ●

●●●

●●

●●

● ●

● ●

●●

● ●● ●

●●

● ●

● ●

●●

● ●

●●

●●

●●

● ●

●●

● ●● ●

●●● ●

● ● ●

● ● ●

●●

● ●●

● ●● ●

● ●●● ●●

● ● ●

●●●● ●●

●●

● ●

●●

●●●

●●

● ●

● ●●

●●

● ● ●

●●

●● ●●●

●●

●●

● ●

●● ●

●● ● ●

● ●

●● ●● ●

●●

● ●

●●

●● ● ●● ●

● ●●

● ●

●● ●●

● ●

● ●● ●●● ●

● ● ● ●

●●

● ● ● ●●●

● ●

●●

● ● ●●

●●

● ●●

●●

●●●

● ● ●●

●●

●●●

●● ●

● ●

●●

●●

● ● ●●

● ● ●

●● ●

●●●

● ●

●●

● ●● ●

● ●

●●

●●

●● ●● ●

● ● ● ●

●●

● ●

●●

●● ●●● ●●

● ● ●●

● ●●●

● ●

● ● ●

●● ●●

● ●

● ● ●

●●●

● ●

● ● ●

● ●

●● ●●

●●

●● ●

●● ●● ● ●

●●

●●●

● ●●

●●● ● ●● ● ●●

● ●●

●●● ● ● ●

●●● ●

●●

● ●

● ● ●●

●● ●

●● ●●

● ●● ●●●

●●

●●

● ●●

●●●

● ●

● ●

● ●● ●

●●

● ●

● ●● ● ●●

● ● ●

● ● ●

●●

●●

●●●

●●

● ●

●●

● ●

● ●

●●

●● ●● ●

●●

●●

●● ●

●●

●●

● ●●●

●● ●

●●●

● ●

●●

●●

●●● ●

● ●

●●

● ●●●

● ●●

●● ●

●●

●● ●

● ●

●●

●●

● ●

● ● ●● ●

● ●

● ●

●●

●●

● ●

● ●

● ●● ●●

●●

● ●●

● ●●

● ●●●

● ●●

●●

●●

●● ● ● ● ●

● ●

●●

●● ●● ●

● ● ●● ●

●●

● ●

●●●●

●●

●●

● ●● ● ● ●

●●

●● ●● ●

● ●

● ●● ● ●

●●

●● ●● ●●

● ● ● ●

● ● ●

●● ●● ●●

●● ●●

●●

● ●

●●

● ●●

● ●● ●

●●

● ● ●

●● ●●

● ●

● ●●●

● ●

● ●●

●● ●●

●●

●● ●● ●

●● ●

●● ●●●

● ●

● ●

● ●

●●

● ●

● ●●

●● ● ●●

●●●

●● ●●●●

●● ●

●● ● ●

●●

●●● ●●

●● ● ● ●● ● ●● ●

●●

●● ●

● ●●

●●

●●

● ●

●●

● ● ●●

●●

●●

● ● ● ●

●●

● ●●

● ●

●●● ●●

●●

● ●

● ●●

●●● ●

●●●

●●●

● ●●● ●

● ●●

● ●● ●● ●● ●●

● ●

●●●●

● ●

● ●

● ●

● ●● ●●

●●

● ●●●

● ●

● ●●

●●

●●

● ●●●

● ● ●●● ●

● ●

● ● ●

● ●

● ● ●

●●

● ●

●●

●●● ● ●●● ●●

● ●● ●

●●●

● ●●

●● ●

● ●●

●●

● ●

● ●

● ●

● ●●

● ●

● ●

● ●● ●● ● ●●● ● ●

● ●

●● ● ●

●● ●

●●●● ●

● ●

●●

● ●

●● ●●

● ●

●●

●●

●●

●● ●

● ● ●

● ●

● ●● ●

● ●●

● ●

● ●●

●●

●●●

●●●●

● ●● ●

●● ●

●● ● ●● ●

● ●

● ●●●● ● ●●

● ●● ●● ●

● ●

●● ●

● ●

● ●

●●

●● ●●

● ●●●● ●●

● ●

●●

●●

●●

● ●

●● ●

●●

● ●

●● ● ●

● ●

● ●

●● ●

● ●

●●

●●

●●

●●

●●

●●●

●●● ●

●●● ●

●●

●● ●

●● ●

●● ●

●●●●

●●

●●

● ●

● ● ●●● ● ●● ●

● ●●

● ● ● ●●

● ●

●●

●●

●● ●

● ●

● ●

● ●

● ●

● ●● ●

●●

●●

●●

● ●●●

●●

● ●●

●●

● ● ●

● ● ●●

●●●●

● ●●

● ● ●

● ● ●●●

●● ●

● ●

●●

●●●

●●

● ●●

● ●

● ● ●

●● ●●

● ● ●●●

●● ●

● ●

●●

●●

●● ●

●●

● ● ●●

● ●

●● ●

● ●●● ●●

● ●● ●

● ●

●● ●●

●● ●

● ●

●●●

● ●

●●

●●

● ●●

● ●

● ●

●● ●

● ●●

●●

●●

●●

●● ●● ● ●

●● ●

● ●

●●●● ●● ●● ●

● ●

● ●●

● ●●●

●●

● ●

●●

●●

●● ●

● ●

●●●

● ●● ●● ●● ●

● ● ●●●

●●

●●

● ●●

●●

● ●

● ●

●●● ●

●●

●●●

● ●● ●●

● ●

●●

● ●● ●●

● ● ●

●●

● ●●

●●

● ●

● ●

● ●

● ●●

● ●

● ● ●●●

● ●

●●

●● ●

●●●

●●

● ●●

●●

●●●

●● ●

● ●

● ●

● ●

● ●

● ●

●●

●●

● ●

●●●

●● ●

● ●

● ●●

●●●

● ●

●● ●

● ●

●●

●● ●●

●●

●●

● ●

● ●

●●

●●

●●●

● ●

● ●

● ●● ●●

●●

●●

● ● ●●

● ●●

●● ● ●

● ● ●

●●

●● ●

●● ●●

● ●● ●

●●

● ●

●●

● ● ●

● ●●

●●

●●

●●● ●

●● ●

● ●

●● ●● ●● ●● ●● ●● ●

● ●

●●

●●

●● ●

●●●●

●●

● ●● ●

●●

● ●●●

● ●●

● ●●●

● ●

●●● ●●

●●

●●

●●

●●

● ●

●● ●

● ● ● ●● ●

● ●● ● ●

●● ● ●

●● ●● ●

●●●● ●●

●●

● ●●

● ●

●● ●

● ●

●● ●●

●● ●●

● ●

●● ●●

●●●

●●● ●

● ●●●●

● ●● ●

●● ● ●●● ●

●● ●

●●

●● ●

● ●●

● ●

●●

●●

● ●●

● ●

● ● ●

● ●●

●●●●●

●● ●● ●● ●● ●● ●

● ● ● ●●

● ● ●●

● ●

● ●● ●

● ●

●● ●

● ●

● ●

● ●●

● ●

●● ● ●● ●●

● ●

● ● ●

●●● ●

●●

●●

● ●●

● ●

●● ●

●● ●

● ●●

● ●

●●

● ●

●●●

●●

●●

●● ●

●● ●

● ●●

●●

●●

●● ●

● ●

● ●

● ●● ●●

●●

●●● ●

● ●

● ●●

●●

●●

●●

●● ●

● ●●

● ●●

●● ●

● ●● ●

●● ●

● ●

●●

● ●

● ●●

●●● ●

● ●

●●

● ●●

● ●

●● ● ● ●● ● ●

●● ●

●●

● ●

● ●●

●●

●● ●● ●

● ●●

●●●

●●

● ●

●●

● ●

● ●● ●

●●

●●●

● ● ●

●● ●●●

●●

●●

● ●●●

● ●● ●● ●

●●

● ●

●● ●

●● ●● ●● ●

●● ●●● ●

●●

● ●● ●●

● ●●

●●

●●● ●

● ●

●●●

● ● ●

● ●●

● ●●

● ●● ●

●● ●

●●● ● ●

● ●●

● ●●● ●

●●

● ●

● ●

●● ●●

● ● ●

●●●

●●

●●

●●

●●●

● ●

●●●

●●

●● ● ●

● ●

● ●

●●

●● ●

●●●

● ●

● ●●●● ●● ● ●● ●● ●

●● ●● ●● ●●

● ●

● ●

●● ●●●

●●

●●

●●

● ●

● ● ● ●●

●●●●

● ●

●● ●● ●●

●●

●●

●●

● ●●●

● ●

● ●

●● ●●● ● ●

●●

●●

●● ●

● ●

● ●● ●

●● ● ● ●●●

●● ●● ●● ●

●●●

●●

●● ●● ● ●

●●● ●

●●● ●

● ●●

● ●

●● ● ●●

●●

●●

●●

●●● ●

●●

● ●

● ●

● ●●

●● ● ●

●●●

●●

● ●

● ●●

● ●

●●

●● ●●● ●

● ●

● ●

● ●

● ●

● ●

●●

● ●●

● ●

●●

● ● ●

●●● ● ●

● ●

● ●

●●●

● ●

● ●

●●

● ●

● ●●

● ●

● ● ●

●●

●●

● ●

●●

● ●

● ●● ●

●● ●●

●●

● ●●

● ●

● ●●

●● ● ●●

●●

● ●● ●● ●

●● ●

●● ●●

●●

● ●

●●

●● ●

●●

● ●

●●●● ●

●●● ●●

● ●

●●

● ●

●●

●●● ●●● ● ●

●●

●●

● ● ● ●●●● ●

● ● ● ●●

●●●

● ●

●●

● ●● ●

●● ●●

●●● ●

●●

●●●

●● ●

● ●

● ●● ●

●●

● ●

●● ● ●

●●

●●●

● ●

● ●

●●●

●●

●●

●●

● ●●● ●

●●

●● ●

● ●

● ●●

●●●

●●

●●●●

●● ● ●●

● ● ●

●●

● ●

● ● ●

●●

●● ● ●● ●●

● ●

● ●

● ●●

●●

● ●

● ●●●

● ●●

●●●● ●

●●

●●

●●

●●

●●● ●

●●

●● ●

●● ●

● ●●

●● ●

●●

● ●●●●

● ●

●● ●

●●● ●

●●● ●●

●● ● ●●

● ●

● ● ●

● ● ●● ●

● ●

● ●

●●

● ●

●●

● ●

● ●

● ●● ●

●●

● ●

● ●●

●● ●

●●

● ●

● ●

●● ●

●●●

●● ● ●●● ●

● ●

● ●●●

● ●

●●

● ● ●

● ● ●

● ●●

● ●●

● ●

● ● ● ●

● ●

● ●● ●

● ●●

●●

●●●

●●●

● ●●

●●

● ●

●●

●●

●●

● ●●

●● ●●●● ● ●●

●●● ●●●●

●●

●● ●

●●

● ●●

●●●

●●

● ●● ●●

●●

●●

● ●●

●●

●●

● ●●

● ●

●●

● ●

● ●

●●

●●

● ●●

●●●

●● ● ●

●●● ●● ●

●●

● ●

● ●●●

●●

●●

● ●● ●

●●

●●

● ●●

●● ● ●●

●●●

● ●●

● ●

● ●●● ●

●●

●● ● ●

●● ● ●●

●● ●

●●

●●

●● ●●

●●

● ●●

●● ●●

● ● ●● ●

●●

● ●

● ●

●●

●● ●●●

●●

●●

●●

● ●●●

● ●

●●

●●● ●●

●●● ●●

● ● ●

●● ●●

●●

●● ●

● ●●

●●

● ●●

● ●

●●

●●

● ●●● ●●●

● ●

● ●

●●

●●●

● ●

● ●●

● ●

●●●

●● ● ●

●●

●●●

● ●●

●●● ●● ● ● ●● ●●

● ●●

● ●

●●

● ● ●●

● ●●

● ●●●

●●

● ●

● ●●●

● ●

●●

● ●●●●

● ●

● ●

● ●

● ●●● ●●

● ●

●● ●

● ●●

●● ●

●● ●

● ●●

●● ●

●● ●● ●● ●● ●●

● ●

● ● ●● ●●● ● ● ●● ● ● ●● ● ●● ●

●●● ●●● ●● ● ●●● ●● ●● ● ●● ●●● ●

● ●●● ● ●● ●● ●●●

●● ●●●● ● ●● ●● ●

●●● ●●

●●● ● ●●● ●●●● ● ●●● ● ●●●

● ●● ●● ●● ●

● ●●● ●

●● ●● ●

●● ●● ● ● ● ●●● ● ●●●

● ● ●

● ● ● ●● ● ● ●●● ●●● ●●

● ●●● ● ●●●● ●●● ●●●●● ● ●

●● ●

● ● ●●● ●● ●●● ● ●●● ● ●

●●● ● ●● ●● ●●

●●● ● ● ●

● ●

● ●

● ● ●●●

● ●●●● ● ●●● ●● ●● ●● ●●● ●● ●● ● ●● ●● ● ● ●● ●

●●● ● ●

●● ●● ●

●● ●

●● ●● ●●●● ●● ●● ●● ● ●●● ● ●●● ●● ● ● ●

● ●●● ● ● ●● ●● ● ●● ● ● ●●● ● ●● ●●●

●● ●● ● ●● ●● ●●● ●●

● ● ● ●● ●●● ● ● ●

●● ●●●●●● ●● ●●

● ●

●● ●● ●●● ●●● ●●

●● ●● ● ●●●

●●●● ●● ●●●● ●

●● ●● ●●●●● ●● ●●

●● ● ●

● ●●●

● ●● ●

● ● ●●● ●

● ●●● ●●● ● ●● ●

● ● ● ●●

● ●●

● ●● ●● ● ●● ●●

●● ●●● ●●

●●● ●

●●●● ●

● ●● ● ● ● ●

●● ●● ● ●●

●●

● ●● ●●●

● ● ● ●●● ● ● ●●● ●●● ● ●● ●● ●

●● ● ●● ●

● ●●● ● ●

●● ● ●●● ● ●●● ● ● ● ● ●● ● ●●●

●●● ●●● ●● ●● ●●

●●● ●● ●● ● ●●● ●●● ●●

● ● ●●●● ●●

●●● ●● ● ● ●●● ● ●●●

● ●●● ●

●● ●● ●●●● ●

●●

●●● ●●● ● ●● ● ●● ●● ●

● ● ● ●● ●● ● ●● ●●●● ●● ●● ●● ●●● ● ●● ● ●

● ●●

●● ●●●

●●

●●●● ●

● ● ●

●● ●● ●●● ● ●●

● ●●● ●●● ● ●● ● ●● ●

● ● ●●

● ●● ●● ●● ●● ● ●●● ●

● ● ● ●●

● ● ● ● ●●● ●● ● ● ●●● ● ●● ● ●

● ●

●● ● ●●●●

●● ●●

●● ● ●●

●●●

● ●● ●

●● ●

●● ●● ●●● ● ●●

●●●● ●

●● ●● ● ●● ●●● ●● ● ● ●

● ● ●● ● ● ●● ● ● ●

● ●●● ●● ● ●

● ●● ●● ●● ● ●● ●●

● ●●●●● ●● ● ● ●● ●

● ●●● ●

●●● ●●● ● ●

●●● ●● ●● ● ●● ●● ●● ● ●●●●

● ●● ●●

● ●●●●

● ●● ●●

● ●● ● ●● ●● ● ●● ●●

●● ●

●●

● ●● ●● ● ●

● ● ● ●

● ●

●●●

● ● ●●

●●● ● ●● ● ●● ●● ●●● ●●

●●● ●● ●● ●

● ● ●●

●●● ●

● ●

●●● ● ●●

● ● ●● ●● ● ●● ●●●● ●● ● ●●● ●● ●●●

●● ●●● ● ●●● ● ● ● ●●● ●●●●● ●● ●

● ●●● ● ●

● ●

●● ● ● ●

● ●●

●● ●● ● ●●● ● ●●● ● ● ● ●●● ● ●● ●

●● ●● ●● ● ● ● ●● ●● ● ●●● ●

●●

● ●●

●●●● ● ●● ●

●● ●●●● ● ●●● ●●●

●● ● ●●

●●

●●

●● ●●● ●●● ● ●● ●●●

●● ● ● ● ●● ● ●● ● ●●

●● ● ●●● ●●

●●●● ● ● ●

●● ● ●

● ●●●● ●● ●● ● ●

●● ●●● ● ●● ●●

●● ● ● ●● ●● ●●

● ●

● ● ●● ●

● ●● ●

●● ● ●● ● ● ●● ● ●

● ●●●

● ●●● ●● ● ●● ●● ● ●●● ●● ●

●● ●● ● ●●● ● ● ●● ●● ●● ●●●● ●●● ● ●● ●

●●

● ● ●

●● ● ●●● ● ●● ● ●● ●

●● ●●● ●● ● ●●●

●●●

●● ●●● ●●

●●●

● ●

●●● ●●● ●● ●

● ●● ●

●●

●● ● ●●

●● ● ● ●●● ●

●●

●● ● ●● ● ●●

●●● ●

●●● ●

●● ●●● ● ●● ●●●● ●● ●● ● ●

● ● ●● ●●

●●

● ●

●● ●● ● ●●

●● ●● ● ●

●●●●

● ●● ●● ●● ●● ●●● ●

● ● ●

● ●● ●● ●● ●●● ● ● ●●● ●● ●● ● ●● ●●● ●●● ●● ●●●● ●● ●●

● ●●● ●●●●

● ● ●●● ● ●●● ●● ● ●●●● ●● ● ●●

●●

●● ● ●● ●● ● ● ●●● ●● ●●● ●●●

●●● ●● ● ●

●● ●●● ●● ●●●

● ● ●● ●●

●● ●●

● ●●● ●●● ● ●● ● ●● ●● ● ●

●● ●●●●

●●

● ●● ●● ●●● ●

● ●● ● ● ●●●● ● ● ●● ●●●● ● ●● ●● ● ●●●● ●● ● ●● ●● ●●

● ● ●●

●● ●● ●●● ● ●

● ●● ●●●

●●

● ● ●● ●

● ● ●● ●

● ●● ● ●● ●●

●●●

●●

● ● ●● ●●● ● ●● ● ●● ● ●●●●

● ● ●

● ●

● ●●

●●

●●

●● ● ● ●●● ●●

● ● ●●

● ● ●● ●

● ●●●● ●● ●

●●

● ●

● ● ●●●● ● ●●● ●● ● ● ●● ● ●● ●● ●●● ● ●●● ●●●●

● ●● ● ●● ●● ● ●● ●

● ● ●●● ●●●● ●● ● ●● ●● ●●●● ● ● ●● ●●●●

● ● ●●

● ●●● ●●●

●●

●● ●

● ●●● ●● ●● ●●

●●●

●●●

●●

● ●

● ●● ●●● ● ●● ●

● ● ●● ●● ●● ● ●● ●●●● ●

●● ● ●● ●●●

●● ●● ● ●

●●● ●●

●● ●● ●

●●● ● ●● ● ●● ● ●●●● ●●●

●● ● ●● ●

●●●

● ●● ● ●●● ●● ●●● ●

● ●●

●● ●

● ● ●●

●●●

● ●● ● ●● ●

●●● ● ●● ●●● ●●● ●● ●● ● ●●

●● ●●● ● ● ●

●●● ●●●

●● ● ●● ●

● ●● ●● ● ●

● ●

●● ●●● ●●

● ● ● ●●

●●● ●● ●● ●● ● ●● ●●

●●● ● ●●

● ●● ● ●

● ● ● ● ●

● ●● ●●●● ●

● ●

●● ●● ● ● ●

● ●● ●● ● ● ●●●●

●● ●● ● ●●●

●● ● ●●

● ●●

●●

● ●

● ● ●●● ●● ●●● ● ●

●● ● ● ●● ● ● ●● ●●●● ●● ●

● ● ●● ●● ●● ●● ● ●● ●● ●● ● ●●● ●●

● ●

● ●● ●● ● ●● ● ●● ●●● ●● ● ●●●●●

● ●

● ●● ● ●●

● ●●● ●● ● ●● ● ●●

● ●● ●●●● ●● ●● ●● ● ● ●●●●●● ● ●

● ●●

●● ●●●

● ●●● ● ●●●● ● ●● ● ●●

●●●● ● ●●● ● ● ●●● ●● ●●

●● ●●●● ●●

●● ● ●●● ●● ●●● ●●●● ●●●

●● ●●● ● ●● ●●●● ● ●●● ● ●● ●●

●●●●

● ● ● ●●● ●●● ● ●● ●● ●●

● ● ●●

● ●●

●●

●●● ●● ● ● ●

● ●●●

●● ●● ●

●●●● ●● ●● ●●

● ●●● ●

●●● ●● ●●

● ● ●●

●●

●● ●● ●

● ●●● ●●●

●●● ●● ●●

● ●

●●

● ●● ●● ●

●● ●● ● ●●● ● ●● ●● ● ●● ●● ●●● ●● ●

●●

●● ●● ●●● ●

● ●

●● ●● ●●● ●●● ● ●● ●● ●● ●●

● ● ●● ● ●● ● ●● ●● ● ●●

●● ●●

●● ●● ●

●●● ●

● ●●●

●●● ● ●● ● ●●● ● ●●● ●

● ● ●● ●● ●●

●● ●●●●

●● ● ●

● ●● ● ●

●● ●●● ●● ●● ●● ● ●● ●

●● ●●●● ●● ●● ●● ● ●●● ● ●● ●● ●● ●● ●● ●●● ● ●● ● ●●● ●● ●●● ●● ●● ●

●● ●

●● ●●●● ●●

●●●● ●

● ● ●●

● ● ●● ●● ●

●● ●●●● ●● ●●● ● ●● ● ●●●●

●● ●● ●● ● ● ●●

● ● ●●

● ●

● ● ●● ●● ●●● ●● ● ●● ●●●

●● ●● ●●● ●● ●● ● ●● ● ● ● ●●● ●●●●● ● ● ●●● ●

●●● ●● ●●

● ●●● ● ●● ● ● ●

●● ●●●

● ● ●● ●● ● ●● ● ●

● ●

●● ●● ● ●● ●● ●● ● ●● ●●

● ●●●● ● ●●●

● ●● ●●● ●● ●● ●● ●● ●

● ●● ●●● ●● ●● ●●●● ●●● ●●●●

●●● ● ● ●

●●● ● ●● ● ● ●●● ●

● ● ●

● ● ●●● ●● ● ●●

● ● ●●● ●●●● ●●● ●● ●●

●●

● ●●●

● ● ●

●● ●●

● ●

●●● ●●● ●●● ●

●● ● ●●●

● ●● ●● ●●

● ●●

●●● ●

● ● ●● ● ● ● ●● ● ●● ●

●●● ● ●●

● ●● ● ●● ●● ●● ●●● ● ● ●●●● ●

● ● ●●●●

● ●●●●● ●● ●● ●●

● ● ●●●● ● ●● ● ●●● ●●● ● ●● ● ●●●● ●● ●● ●●● ●

● ●● ● ●●●● ●●

●● ●●● ●● ●● ●

●● ●●●●

● ●●● ● ●●●●

● ●● ●

● ● ●● ● ●●● ● ●

●●●● ●● ● ● ●●●

● ● ● ●●●

● ●●●● ●●

● ●●● ●● ● ●●● ● ●●● ●●●

●●

●● ● ●● ●●● ●●● ●● ●● ●● ●● ●●

●●● ●●

● ●● ●

● ●●

● ● ●●● ● ●●●● ●● ● ●●●●

● ●● ●●●

● ●● ●

●●● ●●●●● ● ● ●●

●● ● ●● ● ●● ●●● ● ●●

●● ●● ● ●● ● ● ●● ●●● ●● ●●●

● ●● ●

●●●● ●● ●● ●●●

● ●●

● ●●● ● ●● ● ●● ●

● ● ●● ●●

● ●●●● ● ●● ●● ●● ● ● ●● ● ● ●

● ● ●● ● ●●

● ● ● ●●●● ●● ● ●● ●●● ● ● ● ●●

● ●● ● ● ● ● ●

●●

● ●● ●● ●● ●● ●

● ●

● ● ●●

● ●

● ●●

●● ●● ● ●●●● ●●

● ●●●●

● ●●● ● ●●

●●

●● ● ●● ● ●● ● ●

● ●

●●● ●

● ● ●●●

●●

●●● ● ● ●●

● ●● ● ●

●●

● ●

●● ●●

● ● ●● ● ●● ●● ●●● ● ●●● ●● ●● ●● ●● ● ●

●●● ●● ●● ● ● ● ●●● ● ●●● ● ●

●● ●●

● ● ●● ●● ●● ●●

●●● ●● ●● ● ●● ● ●●●● ●●● ●●

●● ●●●● ● ●●

● ●●● ●● ●● ● ●●● ● ●

●●● ●●● ●● ● ●●

●●● ●●

●●● ● ● ●● ●● ●

●●●●

● ●●● ●● ●● ●

● ● ●

●●● ● ●● ●

●● ● ●●● ● ●● ● ● ●

● ●● ●● ●●●●

● ● ●● ●

● ●●●

● ●● ●● ● ●●● ● ● ●● ● ●

●●

● ● ● ●●●

● ●●

●● ●●● ● ● ●● ●●

● ●●● ● ●● ●●

● ●●● ● ● ●● ●● ●●

●●●

●●● ●● ●●● ●● ●● ●● ●●●●

● ●● ● ●● ●● ●● ● ●● ●● ●●● ●●● ● ●

● ●● ●● ●●● ● ●

● ●

●● ●●● ●● ● ●● ●●●● ●● ● ●●● ●●● ● ●●●● ● ● ●● ●

●● ●● ● ●● ● ●●●

●● ●

● ●● ●● ● ●● ●●● ●● ●

● ● ●●

● ●●● ●

●● ●●● ●●●

●●

●●● ● ●● ● ●● ● ●●● ● ●● ●● ●

● ●

●●● ●

●●●

●● ●●●●● ●● ●●

● ●● ●● ● ●●● ●●●

● ●

● ●

● ●● ● ●

● ●

● ●● ●

● ● ●● ● ●● ●● ● ● ● ●● ●● ● ●●●

● ● ●

● ●●●●● ● ●● ●

●●●● ●● ● ●

●●● ●●●● ●●●

● ●●● ●

●●● ●

●● ●● ●● ●● ●●●●● ● ●● ● ●●● ●●●

●● ● ●● ● ●● ● ●●●● ● ● ● ●●●

●● ● ●● ●● ● ●●● ●● ●●● ●● ●

● ● ● ●● ● ●

● ●●

●●● ●

●● ●● ●●●

●●● ●● ● ●● ●● ●●● ● ●● ●●●

● ●●● ●● ●● ●

● ● ● ●● ● ●● ● ●●● ●●

●● ●

● ●●● ●● ● ● ●● ●● ●● ● ●

● ●● ●

●●

●● ● ●● ●

●● ●

●●●● ●●● ●●● ●● ●

● ● ●●● ●●● ●●●

●● ●●●● ● ● ●●● ●● ●●● ●

●● ● ●

●●●● ●●●

●●● ●

●●

● ●●● ●

●●

● ●

● ● ●●●● ●

●●●● ●● ●●● ●● ●

●● ● ●

● ●●● ● ●● ● ●●

● ● ● ●● ●●

● ●●

●● ●●● ●● ● ●●●

●● ●●

●●● ● ●

●●

●●●

● ●

● ●

● ●

●●●● ●● ● ●●● ● ●

● ●●● ● ●●●

● ●● ●

●● ● ●●●● ● ●●●● ●●● ●

●● ● ●● ●●

● ●● ●●

● ● ● ●● ●● ●● ●●● ●

●● ●● ● ●●

● ●●● ●● ● ●●● ● ●● ● ●●

●● ●

●●●●● ●● ●● ● ● ●● ●●●

●●● ●●●

●● ● ● ●

● ●

●●●● ●● ● ● ●●

● ● ● ●

● ●

●● ●● ● ●

● ●● ●

●● ●● ●

●●●

●● ● ●●● ●● ●● ●● ● ●●● ●● ● ● ● ●●● ●●● ●● ●● ●●

● ●●

●●● ● ● ●● ●● ● ●● ●●

● ●● ● ●●●

● ●●

●● ●● ● ●

● ●●● ● ●

● ● ●●

● ●●

● ●● ●●● ● ●●● ●●● ●●●

● ● ● ●● ●● ●●●● ● ●●● ●●● ● ●●● ●●● ●● ●

●● ●● ●●● ● ●● ●

● ● ● ●● ●●● ●● ●● ●●● ●● ●●●● ●●

●●● ●

●● ● ●● ● ● ●● ● ● ●● ●● ● ● ●●● ●● ●● ●● ● ●● ● ● ●●

●●●

●●● ● ●● ●●

● ●● ●●●

● ● ●● ● ●● ● ● ●●

●● ●●

●● ●●● ●

●● ●● ●● ● ●●

●●

● ●● ● ●●

● ●

● ● ●●● ●● ●● ● ●● ●●● ●

●● ● ● ●●●● ●

● ● ●● ●●● ● ● ●● ●

●●●

● ●●●

● ● ●● ●● ●●● ●●● ●● ● ●● ● ● ●● ● ● ●● ●● ● ●● ●● ●● ● ● ● ●

● ●●● ●

●● ● ●

● ●●

●●

● ●

● ●●

● ●● ●●● ● ●● ● ● ●●● ● ●●● ● ●● ● ● ● ●

● ●●● ● ●● ●● ● ●●

●● ● ●●

● ●● ●● ● ● ●● ●

●●● ● ●● ●

●● ● ●

●● ●●● ● ●● ●● ● ●● ●●● ●

● ● ●

●● ●

●●

●●●●● ●● ●●● ● ●●● ●● ●●● ●● ●● ●● ● ●

● ● ●● ●

●●● ●●● ●● ●● ●● ●

●●●● ●● ●

●● ●● ● ●●

●●

●● ●●●●●● ● ● ●● ●

● ● ● ●● ●● ●

● ●● ●●

●● ●● ●● ● ●●

● ●

● ●● ●

●● ●● ●● ● ●●● ● ●● ● ●●●● ●●●

●● ● ● ● ●● ●● ● ●●

●●● ●●

● ●

● ●

●● ● ● ● ●● ● ●

●●● ●● ●●● ●● ●●● ●●

● ●● ●● ●

● ●●●

● ●●● ●●

● ● ●●●

●● ● ●● ●● ●

● ● ●●

●●

● ●●● ● ●

● ● ● ●● ●●

● ●● ●●

●● ● ●●● ● ●● ●

● ● ●●● ●● ●

●● ●● ●● ●● ●●● ●● ●●● ● ●●

● ●●●● ● ●● ●

●● ●● ●●● ● ●●● ●

●●● ●●●

●●

●● ●● ●● ●● ●●● ●● ● ●●●

●● ●

●● ●● ●● ● ●

● ●● ● ●●● ●

● ●● ●● ● ●●● ● ● ●● ● ●● ● ● ●●

●●

● ● ●● ●● ● ●●●

●● ●● ●●● ●●●● ●● ●●● ●

●●

●●●

●● ●●● ●

● ● ●●●●● ●● ● ●●● ● ●●● ●●

●● ●

● ●● ●● ●● ●● ● ●● ● ●● ● ●● ●●● ●● ● ●● ●

● ● ●

●● ●●●●● ●

●●●● ●●●

● ●● ● ● ●●● ●

● ● ●

● ●●● ● ●●● ●● ●●● ●● ● ● ●● ●●● ● ● ●●

● ●●● ●● ●●

● ●

●● ● ●● ●●●

●● ●●

●● ●● ●● ● ●● ●●

● ● ●●● ● ●

● ● ●●●

●●● ● ● ●● ● ●● ● ● ●●● ●●●

●● ●●● ●●●● ● ● ●● ●●●● ● ●

●●

●●●● ● ●● ● ●●●●

● ●● ●

● ● ● ●●

●● ●●● ●● ●● ● ● ●

●●

●●●● ●●● ● ● ● ● ●● ●● ● ●● ●● ●●●● ●

● ● ●●●

● ●●

● ●●● ●●● ● ●●●● ● ●●● ● ●● ●●● ● ●●● ●●

●●

● ●●●● ●

●● ● ●● ● ●●

● ●● ● ●● ● ●● ●

● ●●

●● ●● ●● ●●● ●● ●● ●● ●● ●●● ●● ●● ●●● ●● ●●● ●● ●● ●●

●● ● ● ●● ●● ● ● ●●● ● ●●● ●

● ●●

●●

●●●● ● ●●● ● ●●● ●●

●● ●● ●●● ●● ●●

●● ●● ●

●● ●●

●● ● ●

● ● ●●●● ●● ●●

● ●● ●

●● ● ●●● ●●● ●

●●● ● ●● ● ●●●

● ●● ● ●

● ●● ● ●●●● ●● ●●●● ●●

●● ●

● ●●

● ● ● ● ●

●● ●●●● ●

●● ● ●● ●

● ●●

● ●● ●● ●● ●●● ●●● ●

●● ●

● ●

● ● ●

● ●● ● ●●

● ●● ●●● ●●● ● ●●● ● ● ●● ●●● ● ● ●●

●● ●●●●● ●

● ●●

● ●●●●

●●●● ● ●●● ● ●●●●

● ●

● ●● ● ●

●● ● ●●

● ●●

●● ●

● ●● ●● ● ● ●●

● ●● ●● ●

●● ●● ● ● ●● ●● ●

● ●● ● ● ●● ●● ●●● ●●●●● ●

● ●●

●●● ●●●

●● ● ●●●● ●●

● ●● ●

● ●● ●●● ●●

●●● ●

●●● ●●●●

● ●●●

● ● ● ●●

● ●● ● ●● ● ●●● ●● ●●● ●● ● ●●● ●

● ● ●●●● ● ●● ●● ●●

● ●●

● ●● ● ●●● ●● ●●

● ●●●

● ●● ●● ●● ●●

●● ● ● ●●● ●● ●●● ● ●● ●●● ● ●●

●● ●

● ●● ● ● ●● ●●● ●● ● ●● ●● ● ●

● ●● ●● ● ● ● ●● ●

● ● ● ●● ●●

● ● ●

●●

●● ●● ●● ●

●●

●●● ●●● ●●

●●●●

● ●● ●

● ●● ●●● ●● ● ●●● ● ● ● ● ●

●●●● ●● ● ●

● ●●● ● ● ●● ● ●●● ● ●●● ●● ● ●

●● ● ● ●

●● ●●

●● ● ●● ● ●●● ●● ●● ●● ●

● ● ● ●●

● ●● ●●● ● ● ● ●

●● ● ● ●● ●●

● ● ●● ●● ●

●●● ●● ●

●●

● ●●●● ● ●●

●● ●● ●● ● ●● ●● ● ●● ●●● ●

● ● ●● ● ●● ● ●● ●● ●● ● ●●● ●●

● ●●

● ●●●● ● ● ● ●● ●●● ●

●●

● ●● ● ●● ●

● ● ●●

● ●

●●● ●●

● ●●● ●

● ● ●● ● ●●●

●●● ●● ● ●●● ●● ● ●●● ● ● ●● ● ● ●● ●●● ●● ●● ●● ●●●● ●●● ● ●●●● ● ●●● ●

●●● ●● ●

●●● ●● ●

●● ● ●

●● ●●●● ● ●● ● ●●●● ● ●●● ● ●● ●

●●● ●●● ●

● ●● ●● ●●● ●● ●●

●●●● ●●●● ●● ●● ●●● ●●

●● ●● ●

●●●

● ● ●●● ●●

● ●● ●

●● ● ●● ●● ● ●

●●

● ● ●● ● ●● ●●●● ●● ●●

● ●●● ●● ●● ● ●●●

● ●●

●●● ● ● ● ●●

● ●● ●

●●● ● ●● ● ●● ●● ● ●

● ● ●● ●●● ●●●● ● ●● ●● ●●

● ●● ●●●

● ●●●

●●

● ●● ● ●●●

●● ●● ●●● ●● ●● ●

● ●●

●●● ●●● ● ●● ●●

●●

●● ●●● ● ● ●● ●●● ● ●●● ●

●● ●●

● ●

●● ●

● ●●● ● ● ●●● ●●● ● ●●

●●● ● ●●● ●

● ●

● ● ●●● ●● ● ●●● ● ● ●●● ●● ●● ●● ● ●

● ●● ● ● ●

●● ●

● ● ●● ●● ●

● ● ●●●

●●● ●●● ●● ●

●● ●● ● ●● ● ●● ●●● ●● ●●●

●● ●●●●

● ●

● ●● ●● ●●● ●● ●●

● ●●●

● ●● ● ●● ● ● ● ●●

●●

●● ●●● ●

●● ●●● ●

●●● ● ●● ●● ● ●● ● ●●● ● ●●● ● ● ●

● ● ●● ● ● ●● ●●●● ●

●● ●● ● ●● ●

● ●

●● ● ●● ●● ●●

● ● ●● ●● ● ● ●● ●● ●● ●● ●● ●

●● ● ● ● ●

●●● ●●● ●● ● ●● ●

● ● ●

●●● ●●● ●●●

● ● ●● ●●●●●● ●

● ●●●● ●● ●

●●

● ● ● ●●

●● ●●● ●●

●●●● ● ●●● ● ●

●● ● ●●● ●

●●

● ●

● ●● ● ●● ● ●● ● ●● ●

● ●●

● ● ●●● ●● ●● ● ● ●●●● ●

●● ●● ●● ● ●● ●●● ●

● ●● ●●●● ● ●● ● ●● ●● ●● ● ●● ● ●

● ●● ● ●● ●

● ● ●● ● ●● ● ● ●

●● ● ● ●●●●● ●

●● ●●●

●● ● ●

●●● ●●● ● ● ●

● ●

● ●● ● ●

● ● ●● ●●

● ●● ●● ●●● ● ●

●● ●

●● ● ●● ● ●● ●● ●●● ●●

● ●●●

● ● ●●

● ●● ●●●●

●●

● ●

● ●●

● ● ●● ●● ● ●●●●● ●● ●

●● ●● ●● ●

● ●

●●

●●● ●●● ● ●●

●● ●● ● ●● ●●●

●● ●● ●●

● ●● ●● ●● ●● ●

● ●● ●●● ●●● ●●●● ●● ● ●●● ●●

●● ●● ● ●● ●●

● ●● ●● ● ●● ●● ●● ● ●●

●●● ●●● ● ●●● ●● ●●

●●● ● ●● ● ● ●●●● ●● ●● ●● ●● ●● ● ●● ●

●● ●● ●

●●

● ●

● ●● ● ●● ●●● ●● ● ●● ● ● ●●● ●● ● ●● ●●

●●● ● ●● ●● ● ●

● ●●● ●● ●●● ●● ● ●

●●● ● ●● ●● ● ●●● ● ● ●●●● ● ●●●●●

●● ●● ●

● ●●●

●● ● ● ●

●● ●●

● ●● ● ●● ●● ●● ● ● ●● ● ●●●

●● ●●

● ● ●● ●● ●

●● ●

● ●● ●●●

● ●● ●● ● ●

● ●●●

●●● ● ●

●● ●●

● ●

● ● ● ● ●● ● ●●● ●● ● ●●

●● ● ● ●●●

● ● ●

● ●●

● ● ●●● ●● ●●

● ● ●● ● ●●

●● ●●● ● ● ●● ●●

● ●●● ●●

●● ●● ● ●●

● ●● ● ●●● ●● ● ●●● ● ●●● ● ●●

● ● ●●●

● ●● ●●● ●●●

●●●● ● ●● ● ● ●● ●● ● ●●● ●●● ●● ●●●● ●●●●

● ● ●● ●● ● ● ●● ● ● ●●● ●● ● ●● ●

●●●●● ● ●● ●●● ● ●●●

● ●● ●●

● ● ●● ●●● ●●● ●●

● ●●

●● ●●● ●● ● ●● ●●

● ●●● ●

● ●

● ●

● ●● ●

● ●● ●● ●●● ●

●●●● ● ● ● ●● ●●● ● ● ●●●● ● ●●●● ● ●● ● ●● ●●● ●

● ●● ●● ●● ● ●●

●●

●●● ●● ●● ● ●● ● ●●

●●● ● ●

● ● ● ●●

●● ●

●● ●● ●● ●●● ●● ● ●● ● ●●● ●

● ●●

●●● ●● ● ●● ●● ●● ●● ●●● ●● ●●

●● ● ●● ● ●● ● ●●● ●

●● ● ●●● ●● ●● ● ●● ●●●

● ●●● ●● ● ●●● ● ● ●● ● ● ● ●● ● ●●● ●● ●● ●● ●● ●● ● ●●●●● ●● ● ●● ●● ● ●●● ●

●●● ● ●●●

● ●●

● ●● ●● ●●

● ●●● ● ●●

●●●

●● ● ●● ● ●●●●● ●

● ●

●● ● ●●● ● ●● ● ● ●●

●● ● ●●● ●●●● ●● ● ● ●●

●● ●● ●●● ● ● ● ● ●

●● ● ●● ●●

● ●● ●● ●● ●●

●● ●●● ●● ●●● ●●●●

● ● ●● ● ●●● ● ●

●● ●● ●●● ●● ● ●● ●● ●● ● ●● ●●●

●●

● ●● ●● ●●● ● ● ●●●●

● ●●● ● ● ●● ● ●● ●●● ●● ●●● ●●●●

●●

● ●●

●● ●● ●● ●● ●● ●

●● ● ●● ●

●● ●●

●● ● ● ●●● ●

● ● ●● ●● ●● ● ● ●●● ●● ● ●

●● ●●●● ● ● ● ●●● ● ● ●● ● ●●●● ●

●● ● ●● ● ● ●● ●● ●● ●● ●● ● ●●●

● ●●

● ●●●● ●

● ● ●● ●

● ●●● ● ●

● ●●● ●● ● ●●

● ●

● ●●

●● ● ●●● ●● ●●● ● ● ●●● ●● ●

● ●●

●●●● ●● ● ●●●

● ●● ●

●●

● ● ●● ● ●● ●● ●● ●● ●●● ●

●●● ●●●●●

● ● ●●●

●● ●● ●●●● ● ●

● ● ●●● ●●

●● ●●● ●●● ● ● ●● ● ●●● ●● ●

●●

● ●● ●● ●●●● ● ● ●●

● ●● ●● ●●● ●● ● ● ●● ●●● ●● ●● ●● ●●●

● ●

● ●● ● ●●● ●● ● ● ●● ● ●

●● ●● ●●

●● ●● ● ●●●●

● ● ●● ● ●● ● ●●● ●●●

● ● ●●● ● ● ●●●

●● ● ● ●●

● ●●● ● ● ●●

●●● ●●● ●● ● ●● ●●● ●●●

● ● ●●● ● ●● ●● ●●

●● ●●● ●● ●● ●● ●

●●●●

●● ●● ●● ●

●● ●● ● ●●● ●

● ●

●● ● ●● ● ● ●● ●●● ●● ●●● ●

● ●● ●

●● ●●● ●●●

● ●● ● ● ●●● ● ● ●●●

● ● ●●● ●● ●●● ● ●●●● ●● ● ●● ● ●● ● ●● ● ● ●● ●●● ●● ● ●● ●● ●● ●●

● ●● ●

●● ● ●

● ●

● ●● ● ●●

●●● ● ●●

●● ●● ●● ●●

● ● ●●●

● ●●●●● ●●● ●● ●●

●●

● ● ●●● ●●● ●●

●● ● ●● ●● ●● ● ●● ●●

● ●

●●●● ● ● ●●● ● ●● ●● ● ●● ●

● ● ●● ● ●● ●● ●● ●● ● ●● ●●●

●● ●●● ● ● ●● ● ●

● ●● ●●● ●●● ●

● ● ●●

●●

●● ● ●● ●● ● ●

● ●● ●● ● ●●●●

● ●● ●

●●●● ● ●● ● ●● ●● ●● ●●● ●

● ●●●● ●

●●● ●●● ●●

● ●● ●● ●●● ●● ●● ●● ●● ●

● ●● ●●

●● ● ●●● ● ●● ●● ●

● ● ● ●● ●

● ●

●●● ● ●● ● ●● ● ●● ●● ●●●● ●●●●● ●● ● ●● ●

● ●●

● ●

−1.0 −0.5 0.0 0.5 1.0

−1.

0−

0.5

0.0

0.5

1.0

Dynamics Eigenvalues

Real PartIm

agin

ary

Par

tHistogram of Magnitude of Eigenvalues

Modulus of Eigenvalue

Fre

quen

cy

0.0 0.5 1.0 1.5

050

010

0015

0020

0025

0030

0035

00

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 68: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

Matrix Value

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

020

040

060

080

0

Matrix Value

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

020

040

060

080

0

Matrix Value

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

020

040

060

080

0

Matrix Value

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

020

040

060

080

0

Matrix Value

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

020

040

060

080

0

Matrix Value

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

020

040

060

080

0

Matrix Value

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

020

040

060

080

0

Matrix Value

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

020

040

060

080

0

Matrix Value

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

020

040

060

080

0

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 69: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

Overall True−Start vs True−Estimated Correlation

Correlation

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

050

000

1000

0015

0000

2000

00

Overall True−Start vs True−Estimated RMS

Root Mean Square Difference

Fre

quen

cy0.0 0.2 0.4 0.6 0.8 1.0

020

000

4000

060

000

8000

010

0000

1200

00

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 70: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

Dynamics True−Start vs True−Estimated Correlation

Correlation

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

050

000

1000

0015

0000

2000

00

Dynamics True−Start vs True−Estimated RMS

Root Mean Square Difference

Fre

quen

cy0.0 0.2 0.4 0.6 0.8 1.0

020

000

4000

060

000

8000

010

0000

1200

00

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 71: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

Dynamics True−Start vs True−Estimated Correlation Second Group

Correlation

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

020

000

4000

060

000

8000

010

0000

1200

00

Dynamics True−Start vs True−Estimated RMS Second Group

Root Mean Square Difference

Fre

quen

cy0.0 0.2 0.4 0.6 0.8 1.0

5000

1000

015

000

2000

025

000

3000

035

000

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 72: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

Factor Loadings True−Start vs True−Estimated Correlation

Correlation

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

050

000

1000

0015

0000

Factor Loadings True−Start vs True−Estimated RMS

Root Mean Square Difference

Fre

quen

cy0.0 0.2 0.4 0.6 0.8 1.0

050

000

1000

0015

0000

2000

00

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 73: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

Residual Variances True−Start vs True−Estimated Correlation

Correlation

Fre

quen

cy

−1.0 −0.5 0.0 0.5 1.0

050

000

1000

0015

0000

2000

0025

0000

3000

00

Residual Variances True−Start vs True−Estimated RMS

Root Mean Square Difference

Fre

quen

cy0.0 0.2 0.4 0.6 0.8 1.0

050

000

1000

0015

0000

2000

0025

0000

3000

00

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 74: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

GLM QQ PlotOutcome=Kappa

●●

●●

●●

●●

●● ●●

●●

●●●●

●●

●● ●●●●

●●●

●●

●●

−2 −1 0 1 2

−15

0−

100

−50

050

100

150

Up to Two−Way Interactions Normal Q−Q Plot

Theoretical Quantiles

Sam

ple

Qua

ntile

s

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures

Page 75: The Method of State Space Mixtures - University of …dev1.education.uconn.edu/m3c/assets/File/M3_2015... · The Method of State Space Mixtures ... Every man is in certain respects

R Code to Fit SSM R Code to Generate SSM Dynamics Oddball Dynamics Generation Further Results

GLM QQ PlotOutcome=Kappa

●●

●●

●●

●●

●●

●●

●●

●●

●●●●●●

●●

● ●

●●●●●●●

●●●

●●

●●

●●●●●

●●

●●

●●

●●

●●

●●

●●

●●

● ●●●

●●

●●

●●

●●

●●

● ●●●

●●

●●●

●●

●●

●●

●●

●●

●●

●●

●●●

●●

●●

●●

● ●●

●●

●●

●●●

●●●

●●●

●●

●●

●●

●●

●●

●●●

●●

●●

●●

●●

−3 −2 −1 0 1 2 3

−50

050

100

Up to Three−Way Interactions Normal Q−Q Plot

Theoretical Quantiles

Sam

ple

Qua

ntile

s

Michael D. Hunter University of Oklahoma Health Sciences Center

State Space Mixtures