Decision-making framework

From screening to action

Procrustes

Objectives• 1.5 hours (one before lunch & half after)• Understand and reflect on the purpose of measures and

key assessment vocabulary– Screening and Benchmarking– Sharpening Instructional Targets– Progress monitoring

• Use problem-solving model to – Identify a reading problem– Analyze data for intervention planning

• Generate instructional targets• Create instructional groups

Problem-Solving ProcessWhat is the problem?

What should be done?

Why is it happening?Did it work?

Problem-Solving ProcessWhat is the problem?

What should be done?

Why is it happening?Did it work?

= Expectation – Level of Performance

• Is there a discrepancy between what is expected and what is occurring?– Expectation = > 140 WRC per

minute with 95% accuracy– Current performance = 95 WRC

with 80% accuracy– Is there a problem?– What do the data tell us?– Any other information you would

want to know?

Case Study: Katie

Problem-Solving ProcessWhat is the problem?

What should be done?

Why is it happening?

Did it work?

What alterable factors underlie or maintain problem?

ICEL/RIOT Matrix: Gives us clues as to why

Instruction Curriculum Environment Learner

Review

Interview

Observe

Test

Focus on educationally relevant info

• Goal: To formulate hypothesis about why the problem is occurring. I.E. Find out why she is reading so slow and with such poor accuracy.– Can she see well enough? (Review, Interview, Test)– What instruction has she been given? (Review, Interview)– Could she do it in the right environment?

Disengaged/Unmotivated/Distracted? (Observe, Interview, Test)– Does she have robust decoding skills? (Review, Interview, Test)– Is she fluent with decoding strategies? (Observe, Test)– Does she think about meaning as she reads? (Interview, Test)– Does she have appropriate background knowledge? Vocabulary?

English? (Review, Observe, Interview, Test)

Case Study: Katie

Problem-Solving ProcessWhat is the problem?

What should be done?

Why is it happening?Did it work?

How can we most effectively change those alterable factors that underlie or maintain problem?

What targeted interventions have the highest probability of success (EB)? What resources do we have to hit instructional targets?What is the goal?How will progress be monitored?

Case Study: Katie• Ease back on Fluency Training Program

– Practicing too many errors• Increase time in word study and multi-syllabic words.

Implement Rewards.• Increase intervention time small group instruction.• Monitor Progress weekly with a goal of increasing accuracy to

97% on fourth grade-level text by end of term.

Problem-Solving ProcessWhat is the problem?

What should be done?

Why is it happening?Did it work?

How can we most effectively change those alterable factors that underlie or maintain problem?

Did Katie meet our goal?If not, revisit problem-solving process.If so but she is not at grade-level, can we be more ambitious?If so and she is at grade level, can we reduce supports?

Activity: Think, Write, Share

1. Write (5 minutes): Without looking at notes, label the flowing

problem-solving diagram In your own words, give a quick description of

each step. Check your work with your notes and adjust.

2. Share (5 minutes) Ones share descriptions of each step with 2s. Twos offer constructive feedback.

You have

10minutes left.

You have

9minutes left.

You have

8minutes left.

You have

7minutes left.

You have

6minutes left.

You have

5minutes left.

You have

4minutes left.

You have

3minutes left.

You have

2minutes left.

You have

1minute left.

Attention Please

Time’s up!

Problem-Solving Process_________________________

______________________________

____________________________

Problem-Solving/Learning Theory

YesBenchmark

Met?No

Continue general

curriculum: Provide challenging

work and extensions to generalize skills

Can do/won’t

doSubstantial

Improvement with motivation?

No

YesPlan motivational

enhancement

Validate Skill Deficit

Screening Measure

Accuracy in skills?

No Teach skills &

practice to high accuracy

Practice skills to fluency/automaticity

Application and Generalization:

use in context.

Yes

Labeling Activity (5 Min)

Where do each of the objects on the previous flow-chart fit in the problem-solving model?

1. What is the problem?2. Why is it occurring?3. What can we do about it?4. Did it work?5. Not tied in to PSM

Reading Measures: Table of Specifications

Decoding/Word Study

Fluency Vocabulary Literal Cmprhnsn

InferenceCmprhnsn

LA-content

ELA-CRT X X X X X X

SRI X X X X X

Maze X X X X (X)

RCBM-WRC

X X (X) (X) (X)

RCBM- Errors

X (X) (X)

Decoding Inventory

X

Vocabulary Inventory

X

SRI & Lexiles

• Strengths:• Gives proficiency levels for more advanced students• Computer Scored• Repeated Measure• Tests both inferential and literal comprehension• Good overall screening tool for secondary students

• Weaknesses:• Reliability: computer issues, motivation…• Doesn’t give many clues concerning what to target

instructionally• Not appropriate for frequent progress monitoring

SRI Pointers

• Let students know their scores and set goals• Consider using as an initial screening measure

to determine who needs remedial reading instruction

• Use scores to help content area teachers scaffold

• Have a rational for score decision points

General Guidelines for Lexile Scores

Text Level

Magic Schl Bus

Charolette’s Web

Johnny Tremain

Animal Farm

Great Expectations

Example: Middle School Reading

YesBenchmark

Met?No

Continue general

curriculum: Provide challenging

work and extensions to generalize skills

CBM: Can do/won’t do

Substantial Improvement with

motivation?

No

YesPlan motivational

enhancement

R-CBM

Scholastic Reading Inventory (SRI)

> 95% accuracy

No Teach skills &

practice to high accuracy

Practice skills to fluency/automaticity

Application and Generalization:

use in context.

Yes

R-CBM

CVMS Reading Classes

• Can do/ won’t do: 83% (39/47) improved 68% significantly, 26% moved up to benchmark

• Inaccurate: 15% (<95% accuracy on RCBM)• Dysfluent: 44% (<25%ile on RCBM)• Poor literal comprehension: 26% (<25%ile

Maze)• Poor vocabulary and/or knowledge: ? %

Implications

• Lots of students in reading classes who don’t appear to need reading intervention

• Motivation a significant problem– many ersatz reading problems are really motivation problems and many real reading problems are compounded by motivation issues. (Pocket of low motivation)

• Clear skill deficits that we can measure and remediate: phonics, fluency, vocabulary

Incoming Students

What are we going to do with them?

Students in Reading Classes:Who will need what?

1. What kinds of data do we currently have?2. What does it tell us about who will need

help?3. What sort of help will they need?4. What sort of structures do we need to have

in place to make sure these needs are met?

Maze (multiple-choice close fluency)

• Can student read silently and understand text accurately and at a good pace?

• Hint: look at rate (better than 17 correct replacements)…but also look at errors (better than 90% accuracy).

Oral Reading Fluency:Four Quadrant Instructional Support

Automatic: Rate ≥ 140 WRCPM

Slow: Rate < 140 WRCPM

Accurate: 95% or higher

Quadrant 1 Accurate

And Good Pace

Quadrant 2 Accurate

And Labored

Inaccurate: 95% or lower

Quadrant 4 Inaccurate

And Good Pace

Quadrant 3 Inaccurate

And Labored

Who needs decoding support?

• 104 Students (5.9%)—minimum.• Measure: 95% accuracy—RCBM

Who needs fluency support?

• 275 Students (16%)• Needing support defined as <114 WRC…, <25th%ile

Relationship between ORF and passing a state high-stakes test (McGlinchey & Hixson, 2004)

Who needs comprehension, vocabulary and background knowledge support?

• Everyone…that is what content courses are about…• But some are more behind

– Deliberate direct instruction of important academic vocabulary

• How do we identify the most behind?– Non-readers– Common formative assessments– SRI

Who needs motivational support

• Unknown from data at hand…but we extrapolate that there are many and that there will be many who slough.

• Success is the best motivation (self perpetuating & natural)

• Perky pace at a level that is neither too difficult nor too easy is motivational

• Social interaction is motivational• School stores and silly prizes are motivating

BL

Significance

Behavior Instruction

Reading Instruction

Reading and Behavior Instruction

R BR BR B

The Need for Academic & Behavioral Integration

Source: Shepard Kellam, Ph.D, Senior Research Fellow, American Institutes for Research (AIR)

Discussion Question

• Would we want to group students by motivation?

Reading Comprehension: Literal and Inferential (SRI)

• 16 % (329 students)

Color Key

SRI, RCBM, & Maze CRT Level of Mastery< 10%ile   1   Minimal10th to 25%ile   2   Partial25th to 50%ile   3   Sufficient> 50%ile   4   Substantial

SRI Only50 to 75%ile  75 to 100%ile  

Individual Student Example

LEXILE Prc_Rnk WRC Er_CBM Maze Er_MzLACRT_

PC LACRTSCCRT_P

CSCCRT_UPA

SS_5

231 91 99% 8 73% 37 1a 63 2b

176 100% 36 100% 92 4 83 4

300 80 80% 48 1b 74 3

1110 86 176 100% 26 96% 90 4 76 3

946 61 172 98% 27 96% 80 3 69 3

1105 85 143 99% 24 100% 86 4 81 3

400 5 150 85% 80 3 76 3

983 67 164 99% 30 100% 82 3 87 4

0 1 75 96% 14 88% 49 1b 53 2a

932 58 112 100% 19 95% 80 3 83 4

Rough Guideline for identifying readers for skills based intervention classes

• Below 140 words read correctly per minute or below 95% accuracy on R-CBM.

AND• Below 17 correct replacements on Maze or below 25th

percentile on SRI

When in doubt:1. look at LA-CRT scores. 2. Ideally you should do some additional testing.

Your Turn

• Find practice materials at the back of your “Tools for Assessment”… handout.

• Get in groups and look at your data and complete the question sheet with your group

• Discuss– Student need– How school might meet that need– Grouping for reading class (using quadrant & SRI/Maze)– What additional data might be needed