NOT YOUR GRANDPA'S STATISTICS: NEW MODELING APPROACHES TO STUDENT ACHIEVEMENT & RTI

Jill PentimontiAdrea Truckenmiller

Jessica LoganSara Hart

Discussion: Grandpa Schatschneider Presented Feb 6, 2014

Pacific Coast Research Conference, San Diego

Individual differences in response to intervention: An application of Integrative Data Analysis in Project KIDS

Sara A. Hart&

GrandpaFlorida State University

Expanding our search for moderators of intervention

• A little about me– Behavioral genetics background– PCRC participant

• Even with modest effect sizes, individual differences in intervention response

• Bioecological model (Bronfenbrenner & Ceci, 1994)

– Provides framework for differentiating students based on non-intervention related traits

Integrative Data Analysis (IDA)

• Item-level pooled data (Curran & Hussong, 2009)

• Capitalizes on cumulative knowledge – Longer developmental time span– Increased statistical power– Increased absolute numbers in tails

• Controls for heterogeneity – Sampling, age/grade, cohort, geographical, design,

measurement

Project KIDS

• Expanded definition of moderators of response to intervention– Cognitive, psychosocial, environmental, genetic risk

• IDA across 9 completed intervention projects – Approximately 5600 kids

• Data entry of item level data common across at least 2 projects– ~30 different assessments

• Questionnaire data collection

Proof of Concept

• Behavior problems and achievement are associated

• More behavior problems are typically seen in LD populations

• Is adequate vs inadequate response status differentiated by behavior problems?

Method

• Participants– 2005-2006 ISI intervention project (Connor et al., 2007)

• RCTish : 22 treatment, 25 contrast teachers, 3 pilot• 821 first graders• A2i recommendations vs standard practice

– 2007-2008 ISI intervention through FL LDRC (Al Otaiba et al., 2011)

• RCT: 23 treatment, 21 contrast teachers• 556 kindergarteners • A2i recommendations vs enhanced standard practice

Method• Measures– WJ Tests of Achievement Letter-Word

Identification (LWID)• Pre- and post-intervention testing periods

– Social Skills Rating Scale: Behavioral Problems subscale • Teacher completed during intervention year

05/06 1 ISIMean (SD)

07/08 K ISI LDRCMean (SD)

WJ LWID Fall 24.28 (7.97) 11.96 (5.53)

WJ LWID Spring 36.54 (7.39) 21.64 (7.10)

SSRS .53 (.44) .48 (.44)

Results: Calibration LWID

• Randomly selected 1 time point/child/project to form “calibration sample” for LWID

• IRT with decision to include only items > 5% endorsement rate

• Reduced item sample from 75 36 – Items 8 to 44

Results: Calibration LWID• Generalized linear factor

analysis (GLFA)– Combines latent factor

analysis and 2-PL IRT model• Here, equivalent of 2-PL

IRT model with DIF

• No significant DIF was found

Results: Second data sample LWID

• Using remaining data, GLFA model run again, setting parameters based on calibration sample

• Separately by project– If significant, add DIF estimates to parameters

Results: SSRS

• IRT to GLFA model with Project DIF on full data

Results

Results: Response

• Proc mixed: covariance adjusted LWID score– 1169 children

Results: Response

• 648 treatment children

Results: Response

• 648 treatment children

UnresponsiveCutoff < 20%

N=110!

Results: Response

• 648 treatment children

UnresponsiveCut offFall SS = 95Spring SS= 104

MeanFall SS = 86Spring SS = 96

Results: Response

• 648 treatment children

UnresponsiveCut offFall SS = 95Spring SS= 104

MeanFall SS = 86Spring SS = 96

Responsive MeanFall SS = 99Spring SS = 111

Results

• Logistic regression– SSRS behavior problems significant predictor of

response status (OR = 1.45, CI = 1.12-1.88)• average behavior problems = 19% probability of being

“unresponsive”• greater than average behavior problems(+ 1SD) = 29%

probability of being “unresponsive”• Less than average behavior problems (-1SD) = 12%

probability of being “unresponsive”

Conclusions

• Response status is differentiated by behavior problems– Mo’ behavior problems, mo’ (reading) problems!

• The questionnaire data we will be adding will be real test of bioecological model on response to intervention

Overall IDA conclusions

• IDA is a “cheap” way to get more power, more n at tails, and show more generalizable effects

• Given how similar many of our projects are, consider doing item-level data entry – Easy potential to combine data– Can you do factor analysis and IRT? You can do IDA!

• These data are more useful together than apart– IRT within and between samples?– Treatment effectiveness across samples?– Characteristics of lowest responders?

Acknowledgements • Stephanie Al Otaiba • Carol Connor• Chris Schatschneider• Great staff & grad students, and a small army of data

enterers

NICHD grant HD072286