avoiding paralysis by analysis: part 2 progress · pdf fileavoiding paralysis by analysis:...

Avoiding Paralysis by Analysis: Part 2 Progress Monitoring

ND RTI Conference June 18-20, 2012

Dr. Lisa H. Stewart stewart@mnstate.edu

“Real” Data thanks to several local RTI coaches and teams – Thank You!

Is this progress monitoring? Why or why not?

How can we impact learning for all students?

• Visible Learning: John Hattie • What works in education? Almost everything,

but some things work better and work worse compared to many alternatives.

How do we know what works BEST?

• Meta-Analysis: Compares results of many different research studies and outcomes

• Effect sizes: Converts outcomes of studies to a single scale we can compare outcomes across studies. – .2 small – .5 moderate – .8 large

• Effect size of 1 indicates an increase of one standard deviation on the outcome (achievement). – Advanced achievement by 2-3 years – Student who got “xyz” exceeded 84% of kids who didn’t

get “xyz.”

Formative Evaluation

Variables Impacting Student Achievement: Some of the Biggies!

• Formative Evaluation .90 • Teacher Clarity .75 • Reciprocal Teaching .74 • Feedback .73 • Teacher-Student Relationship .72 • Spaced vs. mass practice .71 • Metacognitive strategies .69

Why Monitor Progress? • When teachers USE progress monitoring

– Students learn more! – Teachers design better instructional

programs – Teacher decision

making improves – Students become more

aware of their performance Safer & Fleishman, 2005

Sometimes it is nice to have data… was changing to RR a good idea?

What is Progress Monitoring?

• Standardized measures – Reliable – Valid (sensitive, treatment validity)

• Tied to important educational outcomes

• Given frequently (e.g., 1 or 2x weekly) – Simple, brief, efficient, and cheap

www.rti4success.org Resources Chart: Progress Monitoring Tools

Some Common CBM Tasks Area Time Procedure Scoring Early Literacy

- Letter Naming

- Letter Sounds

- Phonemic Segmentation - Nonsense Word

1 minute

Individual

Letter Names Correct Letter Sounds Correct Segments Correct Letter Sounds Correct Words Correct

Reading - Oral Reading Fluency

- Maze

1 minute 3-5 minutes

Individual Group or Individual

Words Read Correct/ Errors Words Correct

Math - Computation

- Concepts

2-5 minutes 8-10 minutes

Group or Individual Group or Individual

Correct Digits Correct Problems Points Correct

Written Expression

3-5 minutes Group or Individual Total Words Written Correct Words Sequence Words Spelled Correctly

What about….

• Using MAP or STAR ? • Using Curriculum tests? • Using Common Core Assessments?

– Skwira, 2011 r= .31 to .62, most <.50

Short Term (Mastery) and Long Term Progress Monitoring

BOTH are IMPORTANT!

Short Term Mastery Monitoring

Test subskill mastery and individual lesson effectiveness

Ex: Q&A, worksheets following directions

unit tests, “hot” reads accuracy, skills “checks”

CBE, cba

Long Term General Outcome Measures

Test retention, generalization and progress

toward overall general outcome (reading)

Ex: CBM, DIBELS

How Often Do We Monitor Progress?

• For CBM Early Literacy (Letter Sound Fluency, Nonsense Word Fluency) and Oral Reading Fluency – Weekly with 1 “probe” (this is most common) – 2 or 3x per week (1 probe each time) – Every 3 weeks with 3 probes

• For CBM Maze, Math or WE – Weekly (?) or Every two weeks

Who does it? When? Where?

• Anyone trained in the procedures can collect progress monitoring data – Be creative but careful

• When and Where? – At a time and place that will provide valid information – Use common sense

Using the Data

• Just having progress monitoring data is not enough. You need to USE it.

– Scheduled graph review dates

– Grade level or problem solving meetings

Avoid Common Mistakes • Use “Good” data collected with fidelity • Have an organized system in place

– Progress monitoring schedule for students – Preprinted passages/probes in a binder – An easy way to graph and look at the data

• Internet-based data management and reporting system (AIMSweb, EasyCBM, DIBELS, etc)

– Share/look at the data

What Level of Materials? • What materials to use for progress

monitoring? What grade level? – Rule of Thumb – Use Grade Level materials – if you monitor out of grade level, keep in mind

catching up to peers when setting goal and monitor monthly in grade level material

– Switch to grade level monitoring weekly when intervention is successful and level of intervention intensity is decreased (e.g., moving from Tier 3 to Tier 2 or decreasing time in Tier 2)

Progress Monitoring Steps • Establish solid initial data point, set up graph,

set goal, draw aimline

• Measure and plot student performance over time

• Analyze student performance – check data points below the aimline, “go upness”, trendline, etc

• Make Instructional changes if indicated

• Continue to measure and monitor student performance

Develop goals and graphs with the student and have them graph their progress

Aimline

• Shows general trajectory needed for student to reach his/her goal

– Typically set so student gets back “on target” or “on grade level” within a set amount of time (e.g., by the end of the year) if possible

– Simply draw a straight line from the student’s first data point on the graph to the date and score representing his target or goal

– Use SMART goals: specific, measurable, attainable, realistic, with a clear timeframe

SMART Goals…

• Specific • Measurable • Attainable • Realistic • Timely

Not so SMART goals: “sarah will improve in reading…” “Adam will comprehend at a 6th grade level…” “Suzanne will follow classroom rules…” SMART goal: By May 30, 2012 when given a 3rd grade AIMSweb reading passage, Sarah will read aloud 109 wrc per min. with 5 or fewer errors.

For setting CBM goals – Can use local norms or benchmark targets set by your district

or based on national datasets and research (e.g., DIBELS targets, AIMSweb targets)

– Can use information on the amount of progress students who were successful have made in the past in this intervention or curriculum (e.g., what was the slope of progress in the research?)

– Good rule of thumb: For Oral Reading Fluency 1.5- 3.0 words

per week growth • You are starting a progress monitoring graph for Finnegan, Gr 2 ORF =

48 in the end of January, what would his end of year goal be?

Example Gr 1-5 “Targets” for Aimline Based on the St. Croix River Education District 10-11 Targets linked to success on Minnesota Comprehensive Assessment – II

Grade Measure Target

1 Nonsense Word Fluency January = 52 letter sounds correct/min

1 CBM Grade Level Oral Reading Fluency (ORF)

Spring = 52 words correct/min

2 CBM ORF Spring = 90 words correct/min

Early CBM national “norms” and growth rates in oral reading (words correct per min.)

Grade Percentile Fall ORF Winter ORF Spring ORF Weekly Progress

1 50 27 54 1.80

2 75 50 25

82 53 23

106 78 46

124 94 65

3 75 50 25

107 79 65

123 93 70

142 114 87

4 75 50 25

125 99 72

133 112 89

143 118 92

5 75 50 25

126 105 77

143 118 93

151 128 100

6+ 50 125-150 125-150 125-150 0.66

Hasbruck & Tindal 1992, Teaching Exc. Children, Deno et al, 2001 School Psych Review

www.studentprogress.org Guidelines for slope of progress and level of skill: Math

Note: These figures may change pending additional RTI research.

Grade Mixed Computation Concepts & Applications

Slope Slope

Grade 1 0.50 0.40

Grade 2 0.40 0.40

Grade 3 0.40 0.70

Grade 4 0.70 0.70

Grade 5 0.70 0.70

Grade 6 0.70 0.70

Looking at the Graphs

• Is there “go upness”????

• Is there ENOUGH “go upness”???? Dr. Roland Good

Using Progress Monitoring Data: Is this intervention working?

Adam Gr 4

0102030405060708090

100110120130140

errors

And Now? Adam Gr 4

1/13/2

1/20/2

1/27/2

2/3/20

2/10/2

2/17/2

2/24/2

3/3/20

3/10/2

3/17/2

3/24/2

3/31/2

4/7/20

4/14/2

4/21/2

4/28/2

5/5/20

5/12/2

5/19/2

5/26/2

wrcerr

Repeated Reading 1:1, 20 min day Repeated Reading 1:1, 10 min 2xday

Data Decision Guidelines

• If the student has some data points above and some below the aimline (doing the “aimline hug”), keep doing what you are doing!

• If the student has 3 to 5 consecutive data points above the aimline, consider moving the student to less intervention (e.g., decreasing minutes, or moving from Tier 2 to Tier 1 or Tier 3 to Tier 2)

– Also use other pieces of information like Mastery Monitoring Data!

– Continue to progress monitor

Data Decision Guidelines (Cont’d) • If the student has 3 to 5 consecutive data points below the

aimline, ASK THE FOLLOWING QUESTIONS (and continue to progress monitor):

– What does the “other” evidence available suggest about the student’s progress?

• Error rates? Behavior during the intervention?

– What is the general “trend” of the data? Is the student likely to get where we want if this continues?

• Use visual analysis and other evidence • Use “trendlines” and “aimlines”

– Is the intervention being implemented with fidelity?

Finn Gr 2 CBM-ORF Finnegan Grade 2

Aimline Reading Links 1:5 for 15 min.

Finn Gr 2 CBM-ORF (Cont’d) Finnegan Grade 2

Aimline Added distributed practice and preteaching Reading Links 1:5

for 15 min.

Justan Gr 1 NWF Is Justan Making Progress?

And Now?

And Now????

On Track…

Abdi Gr 2 What Decision Would You Make?

And Now?

Is There “Enough” Go Upness?

What about with STAR data?

A math example…. Is this student making progress?

Another math example…

InIn Math, don’t always see growth like this! But it shows what is possible!

Benjamin, gr K • In Fall, was flagged as “at risk” on AIMSweb screening (had 3

corrects on LNF, 6 corrects on LSF but high errors/guessing) – monitored weekly in LNF, received Leveled Literacy

Intervention from GE teacher

• In January, winter benchmark ok in LNF, but very low in sounds (LSF=4) and phoneme segmentation (PSF=12), went to RTI team – Reviewed screening and graph, teacher comments about

immature speech and actions – moved from LLI to small group working on Phon

Awareness (from curriculum materials) with GE teacher and classroom para support, and RTI support 3-4 days per week working on letter recognition and sounds/phonics, progress monitored LNF and LSF

How is Benjamin doing?

Benjamin, cont’d

• In May, team met again, he had shown growth

in LNF, but not LSF. He also has other concerns with social skills, some speech patterns (Ks and Gs) and math. Team recommends summer school and review of data and plan in September.

Jared, Gr 1

• In Fall, Screening data very low on AIMSweb NWF, also low NWEA

• Other: behavior concerns, in group of struggling kids without strong K curriculum (maybe not much exposure to good Curriculum and Instr?)

• RTI team worked with teacher regularly throughout the year – Variety of interventions in phonics – Behavioral support/plan as well – Monitored weekly in NWF and PSF (later in RCBM)

Jared PSF

Was this a concern? Did they need to monitor progress?

Jared NWF

Jared, started RCBM in Spring

End of Year

• Went from among lowest in grade in Winter on RCBM (23) to in “average” range, almost in the middle of his class, on AIMSweb screening (55)- a gain of 32 words in 15 weeks or ? 2 words per week gain, but still

• Will he still need support?

Holly, Gr 1 and “The case of different points of view”

Holly PSF

Holly again NWF

How is this kid doing? I call this one the “kitchen sink kid”

Documentation for Special Education Eligibility…

Using a Trendline • Shows the general “trend” or trajectory of the student’s

data so far – Web-based programs typically use an Ordinary Least Squares

regression line • AIMSweb, DIBELS data system, Excel

– Need approx. 10-14 data points, the more the better – Trendlines on few data points or on highly variable data are NOT

reliable!!! • Christ, T. (2006). Short term estimates of growth using CBM ORF:

Estimating Standard Error of Slope to construct confidence intervals. School Psychology Review, 35(1) 128-133.

Trendline and Aimline

y = 0.7995x + 52.547 R² = 0.0731

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31

Moira, Grade 3

AIMLINE

TRENDLINE

What Can You Do About “Bounce” in the Data?

Dealing With Bounce • Is there a “measurement” problem?

– Fidelity of administration and scoring

– Materials aren’t well designed or are too difficult

– Who, where, and when measurement takes place can matter (esp. for some kids)

– Motivation issues (can’t do vs. won’t do)

Dealing with Bounce (Cont’d)

• Other ways to minimize bounce or make decisions despite bounce – Look at data trend over a longer period of time – Do more probes at one time and take median or average

score – Do more frequent measurement (e.g., weekly or 2x week)

• Look at trend over time with many data points

– Look at ALL data together (errors, mastery data, etc.) – Use the least dangerous assumption… it isn’t working.

What if There isn’t Adequate Progress?

If you keep doing what you’ve been doing then you will keep

getting what you’ve got.

Insanity is doing the same thing over and over and expecting different results.

Back to Problem Solving

RIOT and ICEL Instruction Curriculum Environment Learner

Review

Interview

Observe

Work samples (best if conditions are known)

Dynamic Teaching, Diagnostic Testing

• Vary the ICE! • Can’t do/Won’t do assessment • Brief Experimental Analysis… try stuff! (and

collect good/consistent data) • Collect more data: Quick Phonics Screener,

CTOPP, etc etc • Think about instructional hierarchy and

instructional “match”

Tiered Interventions: Need to “fit” • PALS

• HOSTS

• Read Naturally

• Rewards

• Reading Rockets

• Etc., etc., etc.

Phonemic Awareness

Phonics

Fluency

Vocabulary and Comprehension

Credit: Matt Burns

What if There isn’t Adequate Progress?

– Is the intervention being done with fidelity?

• Have fidelity checks been done?

• Is the student in the right level of materials?

• Has the student been in school? Are they getting enough minutes of intervention per week? Do they have good academic engaged time during the intervention?

What if There isn’t Adequate Progress? (Cont’d)

– Should the intervention be “tweaked”? Changed? Is there an intervention better “matched” to this student’s needs?

• Changes could include trying a different intervention or just “tweaking” the current intervention such as adding a 5th repeat to a repeated reading or a sticker incentive for accurate reading.

• Grade level or problem solving team members work together to discuss the data, the student, and what intervention changes would have the best chance of success.

What Could We Change?

• Focus or skill

• Teaching strategies: More explicit, more modeling, more practice, more previewing, better matched with core

• Materials: Easier, better matched (cultural, interests, etc.)

• Arrangements: Size group, location, who is teaching?

• Time: Amount of time, days per week, time of day

• Motivation: Interests, goals, rewards, home/school

LHS sample intervention inventory “net”: Big Five for Reading

Example Program Phon Aware

Phonics Fluency Vocab Comprehension

Early Reading Intervention X X X X x

PALS (Vanderbilt) X (K,1) X (K,1) X (2-6) X (2-6)

Reading Mastery (or other SBRC)

X X X X x

Great Leaps x x x

Sound Partners x x

Say it Move It X (fluency)

REWARDS X (gr 4+) X X (Sc, SS) X (SC, SS)

Partner Reading (Archer) x

Rep. Rdg, Rep. Phrases x

Incr. Rehearsal (letter sounds) X (fluency) (preview, vocab?)

Vocab Strategies (Archer) x

Keyword Strategy x

Kansas SIM x

Vocab Strategy (Archer) x

Web Resources

• Research Institute on Progress Monitoring – http://progressmonitoring.net/ – Includes…

• A Study Group Content Module with 15 sections on CBM including activities – http://progressmonitoring.org/pdf/cbmMOD1.pdf

• Progress Monitoring Leadership Team Content Module with 6 sections (e.g. measureable goals, decision making) including activities

– http://progressmonitoring.org/pdf/cbmMODldrshp.pdf • Handouts, videos, and power point presentations • Technical reports of CBM measures

• Research Institute on Problem Solving • http://www.cehd.umn.edu/EdPsych/RIPS/

Web Resources, Cont’d

• www.aimsweb.com, www.edcheckup.com, dibels.uoregon.edu, easyCBM.com – Look for information about progress monitoring as well as

access to materials and graphing for progress monitoring • www.interventioncentral.org

– look for information on CBM, graphing, etc. • www.studentprogress.org

– Growth rates, use in RTI model, etc (not updated)

• http://www.rti4success.org – click on Progress monitoring on right side

avoiding paralysis by analysis: part 2 progress · pdf fileavoiding paralysis by analysis:...

Documents

cyclic oculomotor paralysis

vocal cord paralysis

analysis paralysis

facial paralysis reconstruction

facial nerve paralysis

irritate. paralysis paralysis caused by speeding drivers

acute flaccid paralysis

klumpke’s paralysis

klumpke paralysis

sleep paralysis research

sedation , analgesia & paralysis

motor paralysis clinical

sleep paralysis

vocal cord paralysis

sleep paralysis -presentation

laryngeal paralysis 3

fear paralysis reflex

leg paralysis

hypokalemic periodic paralysis

escape analysis paralysis