deborah t. carran, jacqueline nunn, sara hooks johns hopkins university stacey n. dammann

1

Uses of a Statewide Longitudinal Data System to Evaluate and Inform Programs,

Policies, and Resource Allocations

Presented February 13, 2013

26th Annual Management Information Systems [MIS] Conference

Deborah T. Carran, Jacqueline Nunn, Sara HooksJohns Hopkins University

Stacey N. DammannYork College

2013 MIS Conference 2

Background: Linking Data Sets

Part C Early Intervention Services data (Birth – 3)Part B Special Education Services data (4, 5 – 21)General Education Services dataK Children assessed for K Readiness

WSS-K, scaled then scores at 3 levels Readiness (Developing, Approaching, Fully Ready)

Maryland State Assessment (MSA), scaled then scored at 3 levels (Advance, Proficient, Basic)– Math and Reading assessments administered annually in

grades 3 through 8


Benefits of Using the Maryland IDEA Scorecard

• Data at the State, District, and School levels that drills down to the student level

• Data that allows users to identify students in need of targeted interventions in the alert categories (attendance, academics, suspension, and mobility)

• Data to create an action plan to monitor student progress within targeted interventions

• Reporting functions to support monitoring of progress towards targets of interventions


How to use these data at the state level?

• Descriptive– Who do we serve?

• Longitudinal– What is the educational placement of children served in EIS by K

and Grade 3?– Is Fall K WSS is a successful predictor of later standardized test

performance?• Comparative– Is there a difference in standardized achievement performance

within educational services (Gen Ed and Sp Ed) for children served in EIS?


Tracking Plan

Birth Cohort09/01/2001-08/31/2002

Yes EIS

Gen Ed Grade 3

Sp Ed Grade 3

No EIS

Gen Ed Grade 3

Sp Ed Grade 3


Method

• PARTICIPANTS– 42% tracking match– Missing data

• Outcomes– WSS-K (2006-07)– RSAA and MSAA (Spring 2011)

• Procedure• Three studies presented


Missing Data

• N = 5,328 participants received EIS– 42.1% (n = 2,245) matched – 57.9% (n = 3,083) unmatched (missing data)

• Is there a significant difference between matched and unmatched participants on characteristics: gender, Part C eligibility category, Part C MA, Race, age of entry, age of exit, and/or months in EIS?


Gender

Female33%

Male67%

Unmatched

Female34%

Male66%

Matched

Results of the Chi-Square statistic assuming equal probabilities indicated no significant difference between Unmatched and Matched participants for gender [X2 (1, N = 5,328) = 0.93, p > .05]. Inspection of cell counts and percentages indicated that the distribution for gender was similar for both groups.


Part C Eligibility 25% Delay Atypical Develp/Behav High Probability

N (%) N (%) N (%)

Unmatched 1,902 (61.9%) 463 (15.1%) 707 (23.0%)

Matched 1523 (67.8%) 376 (16.7%) 346 (15.4%)

Total 3,425 (64.4%) 839 (15.8%) 1053 (19.8%)

Results of the Chi-Square statistic assuming equal probabilities indicated a significant difference between Unmatched and Matched participants for Part C Eligibility [X2 (2, N = 5,317) = 47.23, p < .001]. Inspection of cell counts and percentages indicated that a greater proportion of participants in the Unmatched group were eligible due to a Condition with a High Probability than Matched participants. Conversely, a greater proportion of Matched participants were eligible based on 25% Delay than Unmatched participants.


Part C Eligibility cont.

62%15%

23%

Unmatched25% Developmental DelayAtypical Devel or BehaviorHigh Probability

68%

17%

15%

Matched25% Developmental DelayAtypical Devel or BehaviorHigh Probability


Part C Medical Assistance

Yes MA39%

No MA61%

Unmatched

Yes MA33%

No MA67%

Matched

Results of the Chi-Square statistic assuming equal probabilities indicated a significant difference between Unmatched and Matched participants for Part C Medical Assistance [X2 (1, N = 5,278) = 17.49, p = .001]. Inspection of cell counts and percentages indicated that a greater proportion of participants in the Unmatched group were eligible for Medical Assistance while receiving EIS.


Race

Am Ind/Alaska Native

Asian or Pacific

Islander Black/AfAm Hispanic Multiple White

Unmatched 2 (0.1%) 121 (3.9%) 971 (31.5%) 237 (7.7%) 112 (3.6) 1640 (53.2%)

Matched 5 (.2%) 91 (4.1%) 606 (27%) 201 (9%) 75 (3.3%) 1267 (56.4%)

Total 7 (.1%) 212 (4%) 1577 (29.6%) 438 (8.2%) 187 (3.5%) 2907 (54.6%)

Results of the Chi-Square statistic assuming equal probabilities indicated a significant difference between Unmatched and Matched participants for Race [X2 (5, N = 5,328) = 16.76, p < .05]. Inspection of cell counts and percentages indicated that a greater proportion of participants in the Unmatched group were Black or African American while a smaller proportion of participants in the Unmatched group were Hispanic than in the Matched group.


Entry/Exit Age in Months

EntryAge ExitAge Months in Program0

5

10

15

20

25

30

35

16.53

30.38

13.7412.23

28.9

16.71 UnmatchedMatched


Summary – Missing Data• Gender – the two groups are similar• Eligibility – significantly greater proportion of Unmatched

participants in High Probability Category• Medical Assistance – significantly greater proportion of

Unmatched participants received MA• Race – significantly greater proportion of Black or African

American participants in Unmatched group with significantly greater proportion of Hispanic participants in Matched group

• Age – Matched participants were significantly younger at age of entry than Unmatched participants. Matched participants received EIS longer than Unmatched participants. Age of exit was similar for both groups.


Outcome Instruments• Work Sampling System-Kindergarten (WSS-K)

Assesses 7 Domains, scaled then scored at 3 levels (Proficient, In Process, Needs Development)• Personal and Social Development• Language and Literacy• Mathematical Thinking• Scientific Thinking• Social Studies• The Arts• Physical Development

• Reading State Accountability Assessment (RSAA)– scaled then scored at 3 levels (Basic, Proficient, Advanced)

• Math State Accountability Assessment (MSAA)– scaled then scored at 3 levels (Basic, Proficient, Advanced)


Procedure• Student data tracking system links records of students

in General Ed, Special Ed (Part B) and EIS (Part C)• Identify children in 3rd Grade 2010-11 school year

with birth dates between Sept. 1, 2001 and Aug. 31, 2002

• At the state level, student identifiers entered in student data tracking system to obtain student service level (Gen Ed or Sp Ed), student outcomes, and demographic info

• Deidentified data provided to researchers


Studies and RQ for this Presentation1. Impact of Early Intervention on K Readiness– Who was served by EIS?– Does EIS impact later K Readiness Scores?

2. Tracking Children Receiving Early Intervention Services (Part C Services Birth to 3) into Elementary School to Grade 3– What is the educational placement at Grade 3 for

children who received EIS?– What is the SAA performance at Grade 3 for

children who received EIS?


Different Data Set

3. K Readiness and Grade 3 SAA Performance– Is there a relationship between K Readiness and

Grade 3 RSAA and MSAA scoring?– What subscales of the WSS predict RSAA and

MSAA scoring?


Study 1: Impact of Early Intervention Services on K Readiness

• Does level of service provided to children (Birth – 3) enrolled in early intervention services (EIS) programs enhance their later performance on the Kindergarten Work Sampling System (WSS-K)?


K Readiness: Method• 2,245 children – Who were eligible and received EIS services in MD

linked with MD MMSR scores– Born between Sept 1, 2001 and Aug 31, 2002

• EIS services• WSS-K– Summary scaled composite score

• Hierarchical Linear Regression


Participants

Birth Cohort09/01/2001-08/31/2002

Yes EISKindergarten

2006-07


EIS Services & WSS-K Average ScoresService Description Children WSS-K WSS-K

N Mean SD

Audiology 474 73.2 13.1

Family Counseling/Training 375 71.7 13.2

Occupational Therapy 467 71.0 14.4

Physical Therapy 565 73.0 13.9

Special Instruction 1145 71.8 13.5

Speech/Language Therapy 1553 74.4 12.9

Other Services Than Above 335 73.1 13.6

At least 1 Service 2245 74.5 12.8

Max WSS-K score of 90


VariablesOutcome: WSS-KPredictors:• Demographics: FaRMs, Gender, Minority

• Earliest Age of Child that Part C Services Begin(-) receive services earlier better prepares child to enter K

• Time in Program (+) indicates longer time in program better prepares child to enter K

• Total Minutes Services (+) indicates longer time in program better prepares child to enter K


Hierarchical Regression ResultsVariables Model 1 Model 2

β β

FaRMs -.170** -.171**

Gender -.048* -.044*

Minority -.120** -.103**

Age Svc Began (months) -.069*

Time in Program (days) -.066*

Total Minutes Services -.138**

R2 .061 .086

R2 Change .025**


EIS – WSS Conclusions• Demographic Controls: WSS-K was higher for students not

economically disadvantaged, higher for girls, and for White students

• Age Svc Began (-): For every month earlier a child starts receiving services, he/she is expected to score .017 SD increase on the WSS-K– Supports previous findings

• Time in Program (-): inconclusive; possible that children who are in the program for longer times have more severe disabilities

• Total Minutes Services (-): inconclusive, possible that children with more severe disabilities will have more/longer services– Correlation Time in Program & Total Min Svc r = .37, p < .001


Study 2

Tracking Children Receiving Early Intervention Services (Birth to 3) into Elementary School by Service Level:

How do they Compare with their Peers?


Background• Part C Services (IFSP)• Part b/B Services (IEP)• Little research examining outcomes of children

receiving Part C services, mostly Part B services (Cole, Dale, Mills & Jenkins, 1993; Daley & Carlson, 2009; Peterson et al., 2004; Walker et al., 1988)

– Focused on developmental progress over short term; not longitudinal or growth trajectories

– Enrollment in special education changes for children as they move through elementary school

– Limitations (disability, small samples, covariates)


First Steps: Descriptive Analyses

• Children in 3rd grade (within birth cohort)– What is their Education Service (Gen Ed, Sp Ed)?– Who were the children receiving EIS?


Participants

Birth Cohort09/01/2001-08/31/2002

Yes EIS

Gen Ed Grade 3

Sp Ed Grade 3

No EIS

Gen Ed Grade 3

Sp Ed Grade 3


Total Sample, Gen Ed and HI Sp Ed

Characteristic N %Total Sample 52,584 100.0

General Education 47,928 91.1

Special Education (Part B) 4,656 8.9

Disability Codes 4, 6, 8, and 9 3,994 86.7

04-Speech or Language Impairments 1,898 40.7

06-Emotional Disturbance 224 4.8

08-Other Health Impairments 721 15.5

09-Specific Learning Disabilities 1,151 24.7

All Other Disability Codes* 665 14.3

*Autism, Deaf, Deaf-Blindness, Developmental Delay, Hearing Impaired, Intellectual Disability, Multiple Disabilities, Orthopedic Impairment, Traumatic Brain Injury, Visual Impairment


Gender N %

Males 26,449 50.3

Females 26,135 49.7

New Race Descriptions

American Indian/Alaskan 140 0.3

Asian 3,108 5.9

Black/African American 17,044 32.4

Hispanic 6,611 12.6

Multiple Races 2,258 4.3

Native Hawaiian/Pac Islander 33 0.1

White 23,390 44.5

Eligible for Free and Reduced Meals

No 29,892 56.8

Yes 22,692 43.2

Limited English Proficiency Identified

No 47,081 89.5

Yes 4,056 7.7

Exited 1,447 2.8


Students in Grade 3 Historically Tracked to EISCharacteristic N %

Third Grade Students Received EIS Part C 2,482 100.0 General Education Grade Three 1,628 65.6

Special Education Grade Three 854 34.4

Disability Codes 4, 6, 8, and 9 (SL, ED, OHI, SLD) 617 72.2

All Other Disability Codes 237 27.8

Gender

Males 1,646 66.3

Females 836 33.7

Limited English Proficiency Identified

No 2,353 94.8

Yes 97 3.9

Exited by Grade Three 32 1.3

Free and Reduced Meals in Grade Three

No 1,627 65.6

Yes 855 34.4

Minority Status

Yes, Minority 1,086 43.8

No, Minority 1,396 56.2

34

Students in Grade 3 Tracked to EIS, EIS Characteristics

Part C Eligibility of Grade 3 Students Born Sept. 1, 2001 – Aug. 31, 2002

25% Delay Atypical Development

High Probability Condition

Student Placement Grade Three 2011 (n = 1,674) (n = 400) (n = 408)

N % N % N %

General Education Grade Three 1,093 65.3 284 71.0 251 61.5

Special Education Grade Three 581 34.7 116 29.0 157 38.5

2013 MIS Conference

65.5%, n = 1,628 enrolled in general education at grade three


Average State Assessment Scores by Educational Service at K and Grade 3

2008 WSS-K 2011 RSAA 2011 MSAA

N M SD M SD M SD

General Ed Gr 3 47928 77.9 10.9 430.8 38.2 429.9 41.1

No EIS 46300 77.9 10.9 430.9 38.2 429.9 41.1

Yes EIS 1628 77.2 11.1 427.8 39.1 428.6 41.7

Special Ed Gr 3 3994 67.8 13.5 368.0 120.6 364.6 114.4

No EIS 3377 68.0 13.4 371.5 117.3 367.1 111.2

Yes EIS 617 67.2 13.9 349.2 135.9 350.9 129.8


Hierarchical Regression Analyses

Outcome: RSAA Grade 3, MSAA Grade 3Three Models:

Model 1, 2, 3Demographics: FaRMs, Gender, MinorityModel 2, 3Part C EISModel 3EIS x FaRMsEIS x GenderEIS x Minority

Hierarchical Regression Results: WSS-K

WSS-K General Ed Special Ed

Model 1 Model 2 Model 3 Effect Size Model 1 Model 2 Model 3 Effect Size

Variable B (SE) B (SE) B (SE) Partial ŋ2 B (SE) B (SE) B (SE) Partial ŋ2

Constant 81.57**(.10) 81.60**(.10) 81.60**(.10) .592 72.95** (.50) 73.20** (.52) 73.65** (.55) .248

FARMS -3.57**(.12) -3.58**(.12) -3.56**(.12) .003 -3.73** (.50) -3.83** (.50) -3.79** (.56) .002

Gender -2.46**(.11) -2.45**(.11) -2.49**(.11) .000 -1.89** (.51) -1.86** (.51) -2.54** (.55) .004

Minority -1.98**(.12) -1.99**(.12) -1.98**(.12) .001 -3.62** (.50) -3.61** (.50) -3.63** (.56) .009

EIS -.80** (.29) -1.16* (.56) .000 -1.28* (.62) -4.37** (1.37) .003

EIS * FARMS -.47 (.68) .000 -.35 (1.32) .000

EIS * Gender 1.09 (.60) .000 4.22** (1.37) .001

EIS * Minority -.40 (.66) .000 .53 (1.31) .000

R2.057 .057 .057 .056 .057 .060

R2 Change .0001** .000 .001* .003*


*p ≤ .05, **p ≤ .01


WSS-K ResultsGeneral Ed

• WSS-K higher for students not FaRMs, for girls, and for White students

• EIS students scored lower (.7 M diff)

• No interaction effects

Special Ed• WSS-K higher for students

not FaRMs, for girls, and for White students

• EIS students scored lower (.8 M diff)

• Sig interaction EIS x Gender– F > M NO EIS (2.9 M diff)– M > F EIS (1.7 M diff)

Hierarchical Regression Results: RSAA

RSAA Grade 3 General Ed Special Ed



Constant 451.50**(.29) 451.70**(.30) 451.76**(.30) .592 399.95**(4.10) 404.91**(4.19) 406.34**(4.44) .248

FARMS -22.31**(.35) -22.34**(.35) -22.31**(.36) .003 -19.31**(3.99) -21.19**(3.99) -20.24**(4.36) .002

Gender -10.15**(.32) -10.04**(.32) -10.14**(.32) .000 5.16 (4.06) 5.79 (4.05) 4.08 (4.39) .004

Minority -11.26**(.35) -11.35**(.35) -11.39**(.36) .001 -46.56** (4.00) -46.75** (3.99) -48.16** (4.37) .009

EIS -5.49**(.89) -7.64**(1.71) .000 -27.86**(5.16) -39.27**(11.48) .003

EIS * FARMS -1.00 (2.08) -5.64 (10.86)

EIS * Gender 3.08 (1.84) 12.72 (11.48)

EIS * Minority 1.30 (1.98) 9.38 (10.75)

R2.153 .154

.154 .054 .061 .060

R2 Change .001** .000 .007** .000


*p ≤ .05, **p ≤ .01


RSAA ResultsGeneral Ed

• RSAA higher for students not FaRMs, for girls, and for White students

• EIS students scored lower (3.1 M diff)


Special Ed• RSAA higher for students

not FaRMs and for White students



Hierarchical Regression Results: MSAA

MSAA Grade 3 General Ed Special Ed



Constant 448.58**(.32) 448.77**(.32) 448.81**(.33) .550 390.64**(3.84) 394.63**(3.92) 396.59**(4.16) .248

FARMS -22.64**(.38) -22.67**(.38)-22.57**(.39) .003 -18.99**(3.74) -20.50**(3.74) -20.10**(4.08) .002

Gender -2.24**(.35) -2.13**(.35) -2.28**(.35) .000 18.68**(3.81) 19.19**(3.79) 15.52**(4.11) .004

Minority -14.36**(.38) -14.45**(.38)-14.47**(.39) .001 -53.53**(3.75) -53.68**(3.74) -53.01**(4.09) .009

EIS -5.35**(.96) -7.68**(1.85) .000 -22.53**(4.84) -37.81**(10.76) .003

EIS * FARMS -3.46 (2.24) -3.77 (10.18)

EIS * Gender 4.85* (1.99) .000

25.01* (10.76) .001

EIS * Minority 1.02 (2.15) -2.25 (10.08)

R2 .142 .142

.143 .078 .083 .084

R2 Change .001** .000* .005** .000


*p ≤ .05, **p ≤ .01


MSAA ResultsGeneral Ed

• WSS-K higher for students not FaRMs, for girls, and for White students


• Sig interaction EIS x Gender– F > M No EIS (1.9 M diff)– M > F EIS (4.2 M diff)

Special Ed• WSS-K higher for students

not FaRMs, for girls, and for White students


• Sig interaction EIS x Gender– M > F No EIS (10.4 M diff)– M > F EIS (42.9 M diff)


General Ed EIS SAA Conclusions• Demographic Controls: WSS-K, RSAA, & MSAA

was higher for students not economically disadvantaged, higher for girls, and for White students

• EIS: WSS-K, RSAA, & MSAA when FaRMs, Gender, and Minority are controlled, students who received EIS scored lower than their Gen Ed peers.

• EIS Interactions: WSS-K & RSAA no significant interactions; MSAA EIS x Gender (F EIS)


Special Ed EIS SAA Conclusions• Demographic Controls: WSS-K, RSAA, & MSAA

was higher for students not economically disadvantaged and for White students; females scored higher on WSS-K & MSAA

• EIS: WSS-K, RSAA, & MSAA when FaRMs, Gender, and Minority are controlled, students who received EIS scored lower than their Sp Ed peers

• EIS Interactions: WSS-K & MSAA significant EIS x Gender (F EIS lower in K and more at Grade 3)


Youth in EIS have lower average scores than their peers in Gen Ed and Sp Ed

WSS K RSAA G3 MSAA G3 0

5

10

15

20

25

0.67

3.1

1.30.8

22.3

16.2

Gen EdSp Ed (HI)

Aver

age

Mea

n Di

ffere

nce

Betw

een

Non

EIS

and

EIS


Female EIS in Sp Ed (HI) Services have lower average scores than their peers

WSSK RSAA G3 MSAA G30

50

100

150

200

250

300

350

400

F EISM EISFM


Study 3: K Readiness and Grade 3 MSA Performance

Is performance on the WSS-K predictive of Grade 3 high stakes testing (Reading MSA and

Math MSA)?


Background

Maryland Model for School Readiness (MMSR) Pearson Work Sampling System Assessment

Assesses 7 Domains, scaled then scored at 3 levels (Proficient, In Process, Needs Development)• Personal and Social Development• Language and Literacy• Mathematical Thinking• Scientific Thinking• Social Studies• The Arts• Physical Development


Method• Participants/Procedure– General Ed Fall K Work Sampling System student scores

(2003, 2004, 2005) N = 152,105Matched to – Grade 3 Spring MSA Math, MSA Read student scores (2006,

2007, 2008) N = 100,958Match Rate 66% for Reading & Math

• Instruments– WSS is a 30 item instrument with scaled items coded 1 - 3– MSA is a high stakes test coded 1 - 3


Kindergarten MMSR Composite Score Distribution by Grade 3 MSA Performance Level

MSA N % WSS-K M WSS-K SD Math

Basic 7,876 17.71% 63.98 5.71 Proficient 55,777 55.25% 72.21 5.52 Advanced 27,304 27.05% 78.38 5.25

Total 100,957 100.00% 72.42 3.51

Reading Basic 17,695 17.52% 64.78 5.67 Proficient 64,907 64.28% 72.63 5.62 Advanced 18,376 18.20% 79.02 4.91

Total 100,978 100.00% 72.42 3.85


Matching Students K to Grade 3Kindergarten School Year

N WSS-K Scores

Grade 3 MSAA

Grade 3 MSAA

Grade 3 RSAA

Grade 3 RSAA

N Students Match Rate (%)

N Students Match Rate (%)

2002-2003 54,452 36,032 66.17% 36,026 66.16%

2003-2004 50,024 33,986 67.94% 34,008 67.98%

2004-2005 47,629 30,940 64.96% 30,944 64.97%

Total 152,105 100,958 66.37% 100,978 66.39%


Method

• Causal Comparative Descriptive Design• CART analysis– Binary decision trees representing a series of rules

that lead to membership in a class or value– Variables in the dataset are analyzed and split on a

value to best predict outcome – By following the splits in the decision tree to the

targeted outcome, observers can discern the patterns and combinations of variables that best predict either the presence or absence of the desired outcome

MIS DC 2013 Data Conference 53

Grade 3 Math Decision Tree

All Participants



IIA3: Begin Phonemic Aware




IIIB1: Number & Quantity




IIIC2:Recog/Dup/Extend Patterns

IIA3:Begin Phonemic Aware

IIA3:Begin Phonemic Aware







IIIC2: Recog/Dup/Extend Pattern

IIIA1: Strategy Use for Math



IIA3: Phonemic Aware


IIIB1: Number & Quantity IIIC2: Patterns

IIIA1: Strategy Use Math IIA1: Listening




IIIC2: Patterns



IIIC2: Patterns

IBI: Rules & Routines




IIIA1: Strategy Use for Math

IBI: Rules & Routines

IIIC2: Patterns




IIIA1: Strategy for Math

IBI: Rules & RoutinesIIIC2: Patterns




IIA1: ListeningIIIA1: Strategy for Math

IIIC2: Patterns


Math Decision Tree Nodes DescriptionsTerminal

NodeSegment: Predicted Outcome

# Students Average Math Score

Demonstrates beginning phonemic awareness

(IIA3)

Shows understanding of number and quantity (IIIB1)

Recognizes, duplicates, and

extends patterns (IIIC2)

Begins to use and explain strategies to

solve mathematical

problems (IIIA1)

Gains meaning by listening

(IIA1)

Follows classroom rules

and routines (IB1)

1 3 2,838 1.495 3 3 {1,2,3} {1,2,3} {1,2,3} {1,2,3}

2 3 4,833 1.709 1 {2,3} {1,2,3} {1,2,3} {1,2,3} {1,2,3}

3 3 2,173 1.723 {2,1} {1,2,3} {3,2} 1 {1,2,3} {1,2,3}

4 2 14,620 1.927 {2,1} {1,2,3} {3,2} {2,3} {1,2,3} {1,2,3}

5 2 4,647 2.027 {2,1} {1,2,3} 1 {1,2,3} {3,2} {1,2,3}

6 2 4,857 2.188 {2,1} {1,2,3} 1 {1,2,3} 1 {1,2,3}

7 2 4,091 2.143 1 {1,2,3} {1,3} {1,2,3} {1,2,3} {1,2,3}

8 1 3,908 2.246 1 {1,2,3} 1 {1,2,3} {1,2,3} {1,2}

9 1 4,164 2.324 1 {1,2,3} 1 {1,2,3} {1,2} 1

10 1 10,019 2.453 1 {1,2,3} 1 {1,2,3} 1 1


Ranked (Highest to Lowest) Significant WSS-K Indicators used in the Predictive Math Model

SAA MathDemonstrates beginning phonemic awareness (IIA3)Shows understanding of number and quantity (IIIB1)Recognizes, duplicates, and extends patterns (IIIC2)Begins to use and explain strategies to solve mathematical problems (IIIA1)Gains meaning by listening (IIA1)Follows classroom rules and routines (IB1)


Grade 3 Math Score Distribution: Comparing Training & In Time Validation

Predicted Outcome

Training Average

MSA Score

In Time Average

MSA Score

Actual Training Outcome: % Students

Per Segment

Actual In Time Validation Outcome:

% Students Per Segment

1 (Basic)

2 (Proficient)

3 (Advanced)

1 (Basic)

2 (Proficient)

3 (Advanced)

Advanced 435 434 6.16% 49.80% 44.04% 6.99% 48.34% 44.67%

Proficient 411 411 19.08% 59.88% 21.04% 19.14% 59.48% 21.38%

Basic 385 384 42.05% 50.82% 7.13% 43.63% 49.50% 6.88%

Total 414 414 18.95% 55.04% 26.01% 19.46% 54.17% 26.37%


Conclusion

• Fall WSS-K is a moderately successful predictor of later standardized Math test performance


Grade 3 Reading Decision Tree





IIIA1: Strategy f/Math





IIIA1: Strategy f/Math IIA1: Listening






IIIA1: Strategy f/Math






IIIA1: Strategy f/Math IIIB1: Number & Quantity




IB1: Rules & Routines





IIC2: Print Concepts





IIIC2: Patterns

IIB1: Effective Comm





IIIC2: Patterns

IIB1: Effective Comm



IIIA1: Strategy f/ Math IIIB1: Number & Quantity


Read Decision Tree Nodes DescriptionsTerminal

NodeSegment: Predicted Outcome

# Students Average Reading Score

Demonstrates beginning phonemic awareness

(IIA3)

Begins to use and explain strategies to

solve mathematic

problems (IIIA1)

Gains meaning by

listening (IIA1)

Shows understanding of number and quantity

(IIIB1)

Follows classroom rules and

routines (IB1)

Shows some understanding of concepts about print

(IIC2)

Speaks clearly and conveys

ideas effectively

(IIB1)

1 3 4,164 1.566 3 3 {1,2,3} {1,2,3} {1,2,3} {1,2,3} {1,2,3}

2 3 3,466 1.708 3 {2,1} {1,2,3} {1,2,3} {1,2,3} {1,2,3} {1,2,3}

3 3 2,176 1.732 2 3 {1,2} {1,2,3} {1,2,3} {1,2,3} {1,2,3}

4 2 15,345 1.875 2 {2,3} {1,2} {1,2,3} {1,2,3} {1,2,3} {1,2,3}

5 2 4,419 1.949 2 {1,2,3} 1 {1,3} {1,2,3} {1,2,3} {1,2,3}

6 2 4,274 2.084 2 {1,2,3} 1 1 {1,2,3} {1,2,3} {1,2,3}

7 2 5,384 2.083 1 {1,2,3} {1,2,3} {1,2,3} {1,2} {1,2,3} {1,2,3}

8 2 2,609 2.123 1 {1,2,3} {1,2,3} {1,2,3} 1 {1,2} {1,2,3}

9 2 1,601 2.156 1 {1,2,3} {1,2,3} {1,2,3} 1 1 {3,2}

10 1 12,545 2.307 1 {1,2,3} {1,2,3} {1,2,3} 1 1 1


Ranked (Highest to Lowest) Significant WSS-K Indicators used in the Predictive Reading Model

SAA ReadingDemonstrates beginning phonemic awareness (IIA3)*Gains meaning by listening (IIA1)*Begins to use and explain strategies to solve mathematical problems (IIIA1)*Shows understanding of number and quantity (IIIB1)*Follows classroom rules and routines (IB1)*Recognizes, duplicates, and extends patterns (IIIC2)*Speaks clearly and conveys ideas effectively (IIB1)

*also significant predictors in MSA math model


Grade 3 Reading Score Distribution: Comparing Training & In Time Validation

Predicted Outcome

Training Average

MSA Score

In Time Average

MSA Score

Actual Training Outcome: % Students

Per Segment

Actual In Time Validation Outcome:

% Students Per Segment

1 (Basic)

2 (Proficient)

3 (Advanced)

1 (Basic)

2 (Proficient)

3 (Advanced)

Advanced

444

444 5.35% 58.62% 36.03% 5.50% 58.44% 36.06%

Proficient

421

421 17.94% 66.41% 15.65% 17.87% 66.32% 15.81%

Basic

398

398 39.03% 56.66% 4.31% 38.27% 57.89% 3.84%

Total

422

422 18.81% 62.96% 18.23% 18.76% 63.08% 18.16%


Additional Analyses/Conclusions

• Sex– For math and reading, similar performance

• Ethnicity– For math and reading, Whites and Asian/Pac Islanders

realized predicted performance at higher levels; African-American, Hispanic, & Native American realized predicted performance at lower levels

• FaRMs– For math and reading, student receiving FaRMs realized

predicted performance at lower levels


Conclusions • Fall K WSS is a modestly successful predictor of later

standardized test performance• Math SAA performance was better predicted than

Reading SAA• Overidentification of middle group (Proficient)• Findings support the importance of early literacy

skills• Findings support the persistent ‘gap’ in student

achievement, evident from K


Summary QuestionsAre early intervention services effective in preparing

children to enter kindergarten?Value of EIS?Continue funding of Birth to five model?

Is level of K preparedness predictive of grade 3 high stakes testing performance?Value of high quality preK programs (QRIS)?Fund additional preK programs?

Are early intervention services effective in preparing children to be successful in school?Value of EIS?Further tracking studies looking at child outcomes.

deborah t. carran, jacqueline nunn, sara hooks johns hopkins university stacey n. dammann

Documents

mis conference22

mis conference4how

mis conference10results

unmatched participants

mis conference33 benefits

data mining

significant difference

c eligibility25