valid, reliable & efficient:

Valid, Reliable & Efficient:

A Psychometric Evaluation of

“Flash” Word Recognition and

“NSSI” Passage Reading Measures

Kathleen J. BrownMatthew K. FieldsUniversity of Utah

R. Darrell Morris Appalachian State University

Impetus for Current Study

Need for valid, reliable, efficient instruments to determine instructional reading level

Flaws with current instruments DRA – no rate & time consuming DIBELS – screen only IRIs – psychometric evaluation often weak or

missing

Impetus for Current Study

Growing use of “Flash” & selected graded passages a.k.a. “NSSI” (Virginia/ASU effect)

Initial psychometric evaluations positive (Frye, 2004; Frye & Trathen, 2004; Frye

Trathen, Olson, & Schlagal, 2002; Palmer, Trathen, Olson, & Schlagal, 2002)

Theoretical Framework

Instrument Reliability Validity

Flash

(WRI)

Alternate Form

(manual vs. computer)

Content (rep of grade level?)

Concurrent (GORT)

NSSI

(IRI)

Alternate Form

(A passages vs. B passages)

Content (rep of grade level?)

Concurrent (GORT)

(Anastasi, 1988; APA, 1985)

Methods

4 schools 2 = Title 1 1 = public, 1 = parochial 2 = non-Title 1 both = public & mixed SES

192 students in G2-G5 in March, 2006

Rank ordered DIBELS or QRI, then sampled 12 students per grade: 4 high, 4 average, 4 poor to achieve a representative distribution for testing

Methods

135 minutes of assessment in 3 sessions

Presentation order counterbalanced

Flash item selection counterbalanced

9 on data team; 4 hours protocol training

Manual flash interrater differences = n.s.

Alternate Form Reliability

measure of temporal stability for scores

measure of consistency of response for scores

Alternate Form Reliability

To what extent are NSSI A passage scores equivalent to NSSI B passage scores?

To what extent are computer “Flash” scores equivalent to manual “Flash” scores?

Results: Alternate Form Reliability

NSSI Form B

Flash Manual

Flash Comp

NSSI A .878** .761** .803**

NSSI B Add

n.s. on t-tests

.807** .859**

Flash Manual

.847**

**p < .01

Content Validity

provides evidence that items on test represent a specific domain

provides evidence that the format and response properties of the test represent the domain

Content Validity

To what extent do the NSSI passages reflect/measure expected grade level benchmarks?

Maybe look at separate means for accuracy, rate, & comp & report those to show

To what extent does the Flash measure reading instructional level?

NSSI Reading Level Criteria

Acc. (%)

Rate (wpm)

Comp (%)

Mid G1 95 30 60 End G1 95 40 60 Mid G2 95 65 60

End G2 95 or

97 90 60 or

75 End G3 95 90 60 End G4 95 110 60 End G5 95 120 60 End G6 95 130 60 End G7 95 150 60 End G8 95 150 60

Performance Levelfor NSSI by Grade

NSSI Form A

NSSI Form B

Grade 2 M (SD)

2.14 (1.42)

2.28

(1.24)

Grade 3 M

(SD) 3.52

(1.09) 3.49

(1.11)

Grade 4 M (SD)

4.04

(1.43) 3.87

(1.34)

Grade 5 M (SD)

5.06

(1.85) 4.90

(1.57)

Performance Levelfor Flash by Format & Criterion

Manual (70%)

Comp (70%)

Manual (85%)

Comp (85%)

Grade 2 M (SD)

3.26

(1.24) 3.21

(1.31) 2.58

(1.09) 2.48

(1.20)

Grade 3 M (SD)

4.82

(1.82) 4.59

(1.58) 3.50

(1.31) 3.80

(1.41)

Grade 4 M (SD)

5.81

(1.63) 5.91

(1.88) 4.28

(1.63) 4.57

(1.79)

Grade 5 M (SD)

6.99

(1.49) 6.97

(1.93) 5.46

(1.76) 5.66

(1.94)

Concurrent Validity

To what extent are Flash scores and NSSI scores consistent with scores achieved on a “flagship” standardized reading measure (i.e., the GORT)?

Results: Concurrent Validity

NSSI Form B GORT

Flash Manual

Flash Comp

NSSI A .878** .823** .761** .803**

NSSI B .842** .807** .859**

GORT .808** .826**

Flash Manual

.847**

**p < .01

Average Performance Levelfor NSSI & GORT by Grade Level

NSSI Form A

NSSI Form B GORT

Grade 2 M (SD)

2.14

(1.42) 2.28

(1.24) 3.92

(1.72)

Grade 3 M (SD)

3.52

(1.09) 3.49

(1.11) 5.63

(1.55)

Grade 4 M (SD)

4.04

(1.43) 3.87

(1.34) 6.34

(1.53)

Grade 5 M (SD)

5.06

(1.85) 4.90

(1.57) 6.78

(1.22)

Conclusions: For G2-G5

NSSI A and NSSI B seem to have high validity for identifying students’ instructional reading levels

NSSI A and NSSI B can be considered equivalent forms

Conclusions: For G2-G5

Manual Flash and Computer Flash seem to have high validity for identifying students’ instructional reading levels when the criterion is set at 85%

Manual Flash and Computer Flash seem to be equivalent forms

Conclusions: For G2-G5

The GORT does not seem to have high validity for identifying students’ instructional levels—at any grade level.

The GORT over-predicts instructional level—by approx. 2 years.

Note: most GORT comp questions are passage independent (Keenan & Betjemann, 2006)