Detecting Intervention Effects Using a Multilevel Latent

Transition Analysis with a Mixture IRT model

Sun-Joo CHO

Allan COHEN

Brian BOTTGE

2

Context

• If you are asked by national Department of Education to test the efficacy of a new instructional method which was designed to improve learning disabilities students’ performance. Students’ performance were measured in two time points.

• People may concern that • (1) How much progress do student make? • (2) What’s the intervention effects?

3

• (3) What does the progress depend on? (intervention, teacher or both)

• (4) How do children’s response process change over time?

• (5) Do student perform differently?

4

Measuring Change

• CTT for measuring change

• 1. repeated ANOVA or MANOVA: mean change of a total score as a manifest variable across time points

• However, the assumption of sphericity (homogeneous and all correlation between any pair of measures equal) is often violated.

5

LTA-LCM

• Latent Transition analysis with Latent Class Model

• LTA-LCM assumes that there is no variability on the latent trait within classes.

LCM

LTA-LCM

6

LTA-MixIRTM

• Latent transition model with a mixture response model

Mixture Model

LTA_MixIRTM

7

Multilevel LTA-MixIRTMMixture Proportion in teacher level latent class

Mixture Proportion in individual level latent class

Transition proportion IRT model

8

• Proportion of the population in group-level latent class

• Proportion of the population in individual-level latent class at Time 1

• Transition proportion for t=2,…,T

9

10

Estimation

• Assumption: Item parameters are invariant across time and across group-level latent class (item parameters are estimated using post-test data)

• Anchor items (scale comparability among latent class?)

• Mplus (TWOLEVEL MIXTURE)

11

Empirical Study Results

• 310 students nested under 49 teachers answer 20 items at both pre-test and post-test occasion

12

• (1) Multilevel 1PL

ICC= 0.117 (pre-), 0.406 (post-)

(2) Detection of Latent Classes

A. Only one latent class at pre-test (very few students answer correctly)

B. Two latent class at post-test for student-level

C. Two latent class at post-test for teacher-level after the latent class at student level were determined

13

14

15

16

• Comparison between multilevel LTA-MixIRTM and LTA-MixIRTM: students were clustered into four transition patterns in multilevel LTA MixIRTM whereas there was two in the LTA MixIRTM.

17

Do students engage in problem solving differently?

18

How do students’ response change over time?

• On the post-test, 27.4% of the students were classified into student-level Class 2 (pattern 112 and 212). Higher transition proportion of pattern 212 indicate intervention effect.

19

• Pattern 212 has larger mean and large variance, indicating much progress.

How much progress? Teachers/intervention?

20

• In, Pattern 111 (pho=0.856), students tended to retain their relative positions.

• In Pattern 211 (pho=0.004), little variability for these student on post-test.

21

• Questions:

• 1. How about item parameter recovery?

• 2. Anchor item is necessary?

• 3. Achilles’ heel in mixture model