progress monitoring pennsylvania’s seven-step process

Progress Monitoring Pennsylvania’s

Seven-Step Process

Summer 2011Capital Area Intermediate Unit

Tanya Morret, Educational Consultant, CAIU

Essential Questions

1.What are the steps to monitoring student progress?2.How do we use the Assessment Tools?

Early Literacy Oral Reading Fluency Comprehension Writing Early Numeracy Computation Concepts and Applications

3.How and when do we make instructional changes? Graphing Rate of Improvement

Assessment – It Begins Here

Four Types of Assessments:SummativeBenchmarkDiagnosticFormative

Summative

PSSA/PASA, Terra Nova, CATDetermines the amount of yearly progress the

student is making within their current grade level standards/essential learning in comparison to a normative group

Administered annually Research has shown that, despite learning

discrepancies, students can make as much as a 20+ point difference on these tests if they are provided with effective instruction

Benchmark

4Sight, DIBELS, Math Benchmarks, District or Regional Assessments Is a predictor of how well students are likely to

perform on the annual, state-wide assessment

Provides the school/teacher with information regarding deficit skills/areas to inform instructional planning and decision-making

Administered 3-4 times per year

Diagnostic

Woodcock-Johnson, DRA, Etc.Determines a student’s current functioning

level and where instruction should begin.May identify deficit skill areas for remedial

instructionProvides direction as to what

level/skills/curriculum is most appropriateAdministered 1 time per year, usually

immediately prior to a student’s IEP/ER or at the beginning of the school year

Formative

Progress Monitoring Assessments, Checklists, Curriculum Based Assessments, etc. Quick assessment of current skills

Provides the teacher with the opportunity to make on-the-spot adjustments

Administered on a regularly scheduled basis throughout the school year (daily, weekly, bimonthly)

Essential Questions




The Seven-Step Process

6. Instructional Adjustments

2. Data Collection Decisions

3. Data Collection Tools and Schedule

4. Representing the Data

5. Evaluation of Data

7. Communicating Progress

1. Measurable Annual Goals

Progress Monitoring Cycle

Initial Assessment: • Direct Intervention• Historical Data• Standardized Assessments• Diagnostic Assessment• Curriculum-Based Assessment• Parent Input

Design Instruction:• Develop Goals (& Objectives)• Identify SDI• Alignment to General Curriculum• Instructional Grouping & Scheduling• Identify Progress Monitoring

Deliver Instruction:• Deliver Instruction According to Goals

(& Objectives) Using SDI• Collect Data on Progress• Monitor Student Response & Feedback

Ongoing Evaluation: • Evaluate Effectiveness of Instruction• Use Data to Make Decisions & Assess Progress• Adjust Goals (& Objectives)• Adjust SDI & Instruction As Needed• Report to Parents

Measurable Annual Goals

Components of Measurable Annual Goals:

Condition

Student’s Name

Clearly defined Behavior or Skill

Performance Criteria

Remember, CNBC !

Step 1:

Goals in General Education?

Do we write formal goals in general education? Should we?

What is the goal in general education?How does a Universal Screening/

Benchmark Assessment help us identify students at risk (and therefore in need of closer monitoring)?

Research tells us that…

Monitoring student progress alone

can (and does)

Increase student achievement!

Step 2:

Data Collection Decisions Type of data to be collected

Frequency or rate Percentage or accuracy Fluency Duration

Setting(s) and/or Situation(s) where data will be collected

Person(s) responsible for data collection

• Latency• Quality• Level of assistance

If these decisions are not made at the beginning, the likelihood of data collection is greatly reduced.

Step 3: Data Collection Tools and Schedule

What tools will best measure the skill? Curriculum based measures Checklists/Rubrics/Rating Scales Teacher-made tests (standardized)

How often will the data be collected? Weekly Every other week/Bimonthly Monthly Two times per week Daily*

Step 3: Data Collection Tools and Schedule

How often will data be evaluated and with what rules? 4 out of 6 points above or below expectation (aimline) 3 consecutive points above or below expectations

“Without the data, you’re just another person with an opinion.”

A. Sunderville, 2003

Step 4:

Representing the Data

Visual representation is ideal Graphs (include aimline, data and trend line)

Bar graphs compare sets of valuesLine graphs reveal trends over time

Tables Charts

The data should stand alone The data should be understandable to all

Graphing is Best

16

11162126313641465156616671768186

BL 1 2 3 4 5 6 7 8 9 10 11 12 13 14 16 17 18 19

WC

PM

BL Goal Line WCPM Errors/M Trendline (WCPM)

WC

PM Student Data

AimlineTrendline

Errors (optional)

Step 5: Evaluation of Data

Summarize student performance Level of performance

What changes occur following intervention? Slope of performance (Trend)

What is the rate of change? Variability of performance

Up-and-down movement on the graph

Apply Decision Rules (keep rules constant) Examples: 4 out of 6, 3 consecutive

Step 6: Instructional Adjustments

Look for simple changes first before moving to more complex

Do not abandon a goal before trying to change the components of instruction!

Consider changes in intervention strategies Reinforcement type or schedule Different methods Different instructional groups Frequency, intensity, duration Students graphing their own data

Step 7: Communicating Progress

Communication is a two-way street Home issues affect performance Classroom and instructional issues affect performance

Communication with parents and team members should be ongoing Weekly (regular) phone calls, team meetings Communication books, data logs, email Progress reports (including graphs) Parent-teacher conferences

Summarizer

3-2-1First Three Steps of Progress Monitoring

Two data rules.

One Question you still have

Questions?

Essential Questions




Practice with

the tools

Measuring

Literacy

Early Literacy Tools

Assessments available:Phoneme Segmentation Fluency (PSF)

Phonemic AwarenessNonsense Word Fluency (NWF)

Phonics

Phoneme Segmentation Fluency

Words orally presented to student. Student segments words into parts. Record each different, correct part of the

word. Timed for 1 minute. Established at 35 segments per minute.

Recommended Schedule:

One time per week

Tips for PSF

Pacing is everythingKeep your talk to a minimum

Do not add phrases such as… “Good!” “The next word is…”

Underline each different, correct part Whole words = 0 points

PSF Example

Target word Student Response Scoring

roof /r/ /oo/ /f/ /r/ /oo/ /f/

star /star/ /s/ /t/ /ar/

arm /ar/ /ar/ /m/

cat /ka/ /a/ /at/ /k/ /a/ /t/

3 points

0 points

1 point

3 points

Score Along

Listen to the presenter. Score using the form in your packet

Nonsense Word Fluency

Non-words presented on a page for student. Student may read word as a whole or read each

sound in the word. Underline each correct letter sound. Record errors. Timed for one minute. Established at 50 letter sounds (CLS) per

minute and 15 words read correctly (WRC).

Recommended Schedule:One time per week

Tips for NWF

Model segmentation of sounds and word reading before beginning assessment.

All vowel sounds are short.The goal is word reading. If the student

can decode the entire word, encourage him/her to do that.

Underline each letter the student decodes accurately.

NWF Example

Target word Student Response Scoring

lin /l/ /in/ l i n

pok /p/ /ō/ /k/ p o k

mus /mus/ m u s

bav /p/ /ə/ /v/ b a v

3 CLS

ō2 CLS

3 CLS 1WRC

p ə1 CLS

Score Along

Listen to the presenter. Score using the form in your packet

Monitoring Reading and Writing

Assessments available:Oral Reading Fluency (ORF)

Fluency of DecodingMaze

ComprehensionAIMSWeb Comprehension Questions

ComprehensionPassage or Story Retell

ComprehensionWriting Sample Writing Fluency, CWS, Domain Scoring Guide

For most students:

Oral reading and fluency go

hand-in-hand

Progress in comprehension can

be monitored using Oral

Reading Fluency Measures

Fluency Comprehension

Measuring Oral Reading Fluency


Easy to administerQuick although administered individuallyCan be done by different examiners

(if standard procedures followed)

Recommended Schedule:

One time per week


Provides formative miscue analysisAllows teachers to assess qualitative featuresAllows for assessment of accuracy and fluencyIs predictive of future achievement

First grade outcomes are strongly related to third grade outcomes

Timed for one minute

Tips for ORF

Monitor at instructional level (90-94% accuracy) You may read the title of the passage Slash errors - add notes for errors Remember to bracket ( ] ) after the last word

read To establish baseline, the student reads three

passages Record the median score.

Steps to Monitor ORF

1. Select a passage at the student’s instructional level.

2. Deliver a standard set of directions.3. Set countdown timer for 1 minute.4. Begin timing when student begins

reading5. Record errors as the student reads.6. Bracket (]) after the last word read

within the time limit.

ORF Example

My favorite part of family camp is the campfire program at the lake. We

wait until the stars and moon are out. We walk down to the edge of the water

where a big bonfire is going. We all snuggle together watching the flames

because it’s cold after dark.

The camp director leads everyone in songs. He plays the banjo. We

sing “You are My Sunshine,” and “She’ll be Coming Around the Mountain.”

Then he divides the group in two and we sing rounds like “Row, Row, Row

Your Boat” and “Are You Sleeping?”

Then we have stories and skits. The stories are usually tales about

campers at family camp in previous years, and a funny or brave thing that

they did. One story was about my older brother rescuing a cat. The skits

are always silly.

14

30

43

49

61

73

88

94

106

120

134

137

DIBELS 3RD Grade Passage

Going to Family Camp

]

flavor pragram

+ed

leds

ed gee

bancho

divide is

83 words read - 7 errors = 76 wcpm

76 wcpm ÷ 83 total words = 92% accuracy

Characterize the Errors

Multi-syllabic words – favorite, program, waitedEndings – waited, dividesVowels – program, leadsConsonant digraphs/trigraphs – edge, banjo

Qualitative Features

Dysfluent: Student is considered “some risk” for grade level (fall, winter benchmarks)

Word attack: Initial sound correct in all miscued words read (errors), self corrected some words

Pacing: Slowed down to read difficult or unfamiliar words

Prosody: Little expression, inflection; did not pause for end punctuation with consistency

ORF Example

The ancient Egyptians believed that once they died, they would go to a

place called the New World. This New World was thought to be a wonderful

place where people could live forever. Arriving in the New World was so

important to the Egyptian people that they make great plans for this afterlife

journey. One important plan was to build pyramids to be used as tombs.

Another important plan was to preserve the body. The Egyptians believed

that if the dead body decayed, the person’s spirit would die and be unable to

reach the New World. They developed a process called embalming. This

process turned dead bodies into mummies. Usually only the wealthy

Egyptians were able to have their bodies mummified.

The Egyptians believed that travel to the New World was difficult. They

thought that the dead needed to have food and drink to make the journey

13

27

40

53

66

77

92

103

113

121

133

147

Six Minute Solution 7th Grade Passage

Ancient Egyptians Traveled to the New World

]

ainsent (-ed) omit

made

ĭ

sound out

tah mbs

103 words read - 11 errors = 92 wcpm

92 wcpm ÷ 103 total words = 89% accuracy

sound out

pie ra mid

pre serve

sprit

im bal ming

sound out

sound out

Jow er ney SC

sound out

Characterize the Errors

Multi-syllabic words – ancient, believed, wonderful, arriving, important, afterlife, etc.

Endings – believed, arrivingVowels – journey, pyramid, tombsPrefix - embalming

Qualitative Features

Dysfluent: Rate is below basic for grade levelWord attack: Sounded out by syllablesPacing: Slowed down to read difficult or

unfamiliar words Prosody: Little expression, inflection; paused

for end punctuationAutomatically rereads words: Only becomes

automatic after several exposures

Questions about Reading Fluency?

Foundation skills (LNF, PSF, NWF)Oral Reading Fluency

Determining instructional levelFrequency of monitoringError analysisReporting present levels

Writing measurable goals

MeasuringReading Comprehension

Measuring Reading Comprehension

Comprehension RetellingAllows examiner to consider the

student’s ability to independentlyIdentify key components of a passage

main idea, plot, sequence, etc.Organize information from reading

Having a student retell or summarize what s/he has read allows us to peer into his/her brain and see how

information is being processed.


Narrative TextSettingsCharactersProblemMajor EventsResolution

Expository TextText-Based FactsKey VocabularyLevel of

Understanding


Comprehension Retelling

Use standard passages DIBELS, The 6-Minute Solution, etc.

Allow students to read orally or silentlyCould assess both to see if reading orally interferes with or

enhances comprehension

Could provide student with a graphic organizer to assist with retelling


Comprehension RetellingTakes more time to assess each studentMust be administered individuallyScoring with a rubric is more subjective

Recommended schedule: One time per monthMid and end marking periods

Practice with Retelling

Join another partnership to form a group of four. One person is the “student” reteller.

Read the 6th grade passage provided. Listen to your partner retell the passage. Record the information on the record sheet

provided. Score the retelling. Share/discuss your score with the group.

Practice with Retelling

Secondary Example: Listen to the presenter retell the passage. Score the retelling. Discuss with your group. Share with large group.

Elementary Example: (optional practice)


Comprehension Maze AssessmentPaper/Pencil technique

Leveled reading passages with every seventh word removed

Three choices; correct response, closely related incorrect response, and distracter


Maze Comprehension Passages Takes less time to assess each student Can be administered as a group Scoring based on correct responses in allotted

time, can be proratedAfter 2(or 3) consecutive errors, stop

scoringSchedule:

Can be given every week, but incremental growth is much slower


Comprehension Questions

Use graded passagesNeed to have standardized questions for

all passages (literal and inferential)Question matrix (example: 2 vocab, 2 prediction, 1

main idea, etc.)

Determine if students respond orally or in writing


Comprehension Questions Easy to administer and score

BUT May be an assessment of recall rather than

comprehension Oral reading is a stronger predictor of

comprehension Not as sensitive to growth

Recommended Schedule:Two times per month. Again harder to see incremental growth.

MeasuringWriting Fluency

Measuring Writing Fluency

Writing SampleGive Students a writing prompt and two

minutes to “pre-write’Allow three minutes to writeMeasure Fluency, CWS, and Spelling, and

performance against PSSA scoring Guide

Tools to Assess Writing

Writing Fluency is the rate of word production and/or correct word sequences per minuteAdvantages:

ObjectiveReliableHighly correlated with norm-referenced

achievement tests and teacher judgment of quality at the elementary level

Sensitive to student growth in written expression across 10 and 16 week periods


Written Expression Tools:Fluency

Count the total number of words written in three-minute time (including the title). Divide the total number of words by the time (3 min.) to determine the fluency rate in words per minute.

Misspelled words count if they are separated from other words in the printed material

Numerals not spelled out are not counted If the student copies the story starter or prompt,

include these words in the count.

Sample

On Christmas vacation I played with my German

Shepard Brandy. I sprade water at her. I

wrestled her and I played soccer with her. We

set down together, we telled jokes to each other,

we played and played intell I had to go into the

house and eat lunch. Fluency - 49 words/3 min.

16.3 wpm


Written Expression Tools: Spelling

Count the total number of words in the sampleCount the number of incorrectly spelled wordsSubtract the incorrectly spelled words from the total

wordsDivide the number of correctly spelled words by the

total words to yield an accuracy rate

Sample

On Christmas vacation I played with my German

Shepard Brandy. I sprade water at her. I

wrestled her and I played soccer with her. We

set down together, we telled jokes to each other,

we played and played intell I had to go into the

house and eat lunch.

Spelling: 44/49 = 90% accuracy


Written Expression Tools – Correct Word Sequences

Considers units of writing and their relation to one another

Sequences must be correctly spelled and grammatically correct including punctuation

Sample Writing

On Christmas vacation I played with my

German Shepard Brandy. I sprade water at

her. I wrestled her and I played soccer with

her. We set down together, we telled jokes

to each other, we played and played intell I

had to go into the house and eat lunch.

^ ^ ^ ^ ^ ^

^ ^^° ° ° °

^

^

^ ^^^ ^ ^ ^^ ^ ^

^ ^^ ^ ^° ° ° °^ ^ ^ ^ ^ ^ ^ ° °

^ ^ ^ ^ ^ ^ ^ ^ ^ ^

CWS = 43

WRITING RUBRIC FOR STUDENTS

FOCUS CONTENT ORGANIZATION STYLE CONVENTIONS

The single controlling point made with an awareness of task (mode) about a specific topic

The presence of ideas developed through facts, examples, anecdotes, details, opinions, statistics, reasons, and/or explanations

The order developed and sustained within and across paragraphing using transitional devices and including introduction and conclusion

The choice, use and arrangement of words and sentence structures that create tone and voice

Grammar, mechanics, spelling, usage and sentence formation

FOCUS CONTENT ORGANIZATION STYLE CONVENTIONS 4

Sharp, distinct controlling point made about a single topic with evident awareness of task (mode)

Substantial, specific, and/or illustrative content demonstrating strong development and sophisticated ideas

Sophisticated arrangement of content with evident and/or subtle transitions

Precise, illustrative use of a variety of words and sentence structures to create consistent writer’s voice and tone appropriate to audience

Evident control of grammar, mechanics, spelling, usage and sentence formation

3

Apparent point made about a single topic with suffi cient awareness of task (mode)

Suffi ciently developed content with adequate elaboration or explanation

Functional arrangement of content that sustains a logical order with some evidence of transitions

Generic use of a variety of words and sentence structures that may or may not create writer’s voice and tone appropriate to audience

Suffi cient control of grammar, mechanics, spelling, usage and sentence structure

2

No apparent point but evidence of a specific topic

Limited content with inadequate elaboration or explanation

Confused or inconsistent arrangement of content with or without attempts at transition

Limited word choice and control of sentence structures that inhibit voice and tone

Limited control of grammar, mechanics, spelling, usage and sentence formation

1

Minimal evidence of a topic

Superficial and/or minimal content

Minimal control of content arrangement

Minimal variety in word choice and minimal control of sentence structures

Minimal control of grammar, mechanics, spelling, usage and sentence formation

NON- SCOREABLE OFF- PROMPT I s illegible; i.e., includes so many indecipherable words that no sense can be made of the response

I s incoherent; i.e., words are legible but syntax is so garbled that response makes no sense

I s insuffi cient; i.e., does not include enough [detail] to access domains adequately

I s a blank paper

I s readable but did not respond to prompt

Essential Questions




Remember…The Seven-Step Process

6. Instructional Adjustments

2. Data Collection Decisions

3. Data Collection Tools and Schedule4. Representing the Data

5. Evaluation of Data

7. Communicating Progress

1. Measurable Annual Goals

Annie is a 6th grader. Her current instructional level is 3rd grade. She reads at a rate of 65 words correct per minute with 4 errors (94 % accuracy).

Her goal is to read at a 4th grade level at a rate of 115 words correct per minute and 90% accuracy.

(interim goal 91wcpm at 3rd grade level)

Consider Annie, a Struggling Reader

Data from Annie’s weekly assessments:

Is Annie making progress?

Is her progress enough?

Will she achieve her goal?

Date WCPM Errors

9/8 65 4

9/15 60 2

9/22 64 3

10/6 70 1

10/13 76 2

10/20 75 4

10/27 72 2

11/3 77 3

11/10 69 6

11/17 73 3

11/24 82 3

11/29 78 2

Consider Annie, a Struggling Reader

Annie's Oral Reading Fluency

50525456586062646668707274767880828486889092949698

100

Date

Wo

rds

Co

rrec

t p

er M

inu

te Does the graph help?



Will Annie achieve her goal?


50525456586062646668707274767880828486889092949698

100

Date

Wo

rds

Co

rrec

t p

er M

inu

te

Does the aimline help?





50525456586062646668707274767880828486889092949698

100

8/29/05

9/5/05

9/12/05

9/19/05

9/26/05

10/3/05

10/10/0

5

10/17/0

5

10/24/0

5

10/31/0

5

11/7/05

11/14/0

5

11/21/0

5

11/28/0

5

12/5/05

12/12/0

5

12/19/0

5

12/26/0

51/2

/061/9

/06

1/16/06

1/23/06

1/30/06

2/6/06

2/13/06

Date

Wo

rds

Co

rrec

t p

er M

inu

te

Does a trendline help?




Step 5: Evaluation of Data

Summarize student performance Level of performance

What changes occur following intervention? Slope of performance (Trend)

What is the rate of change? Variability of performance

Up-and-down movement on the graph

Apply Decision Rules (keep rules constant) Examples: 4 out of 6, 3 consecutive

Rate of Improvement

Important calculationAids goal settingAids decision makingClarifies progress reporting

Method of calculationSubtract baseline from end scoreDivide difference by the number of weeks This is the student’s actual rate of improvement

Annie: Baseline = 65 End score = 78 Weeks = 11

Calculate: 78 – 65 = 13 13 ÷ 11 = 1.2

Annie’s ROI is 1.2 words per week

Rate of Improvement

Date WCPM Errors

9/8 65 4

9/15 60 2

9/22 64 3

10/6 70 1

10/13 76 2

10/20 75 4

10/27 72 2

11/3 77 3

11/10 69 6

11/17 73 3

11/24 82 3

11/29 78 2

Your turn:Ryan is a 3rd

grader reading at a 1st grade level

His goal is to reach middle 3rd grade at 92 wcpm

Calculate Ryan’s ROI

Rate of Improvement

Date WCPM Errors

9/8 32 2

9/15 36 3

9/22 38 1

10/6 43 4

10/13 40 2

10/20 44 2

10/27 47 3

11/3 45 4

11/10 41 3

11/17 43 1

11/24 46 2

11/29 48 2

Ryan’s data: Baseline = 32 Last score = 48 Weeks = 11

Calculate: 48 – 32 = 16 16 ÷ 11 = 1.45 wcpm

Rate of Improvement

Ryan’s target: Baseline = 32 Goal = 92 Weeks = 32

Calculate: 92 – 32 = 60 60 ÷ 32 = 1.8 wcpm

Is this enough?

Step 6: Instructional Adjustments

Look for simple changes first before moving to more complex

Do not abandon a goal before trying to change the components of instruction!

Consider changes in intervention strategies Reinforcement type or schedule Different methods Different instructional groups Frequency, intensity, duration Students graphing their own data

Step 7: Communicating Progress

Communication is a two-way street Home issues affect performance Classroom and instructional issues affect performance

Communication with parents and team members should be ongoing Weekly (regular) phone calls, team meetings Communication books, data logs, email Progress reports (including graphs) Parent-teacher conferences

Imagine the Possibilities!

707274767880828486889092949698

100102104106108110112114116118120122124126128130132134136

8/29

/05

9/12

/05

9/26

/05

10/1

0/05

10/2

4/05

11/7

/05

11/2

1/05

12/5

/05

12/1

9/05

1/2/

06

1/16

/06

1/30

/06

2/13

/06

2/27

/06

3/13

/06

3/27

/06

4/10

/06

4/24

/06

5/8/

06

5/22

/06

6/5/

06

Week of

Wo

rds

Co

rrec

t p

er M

inu

te

Positive trend - continue program

am bus incident

Begin Graphing ow n data

Daily repeated reading 3x in school and 3x at home w /graphing

Begin direct vocabulary instruction

continue w ith vocab program

M & M Activity

Share Your Thoughts… Red – The most important thing you

learned.

Yellow – Something important you found in your manual.

Green – Share your first next step as a

result of this training.

Blue – Something you agree with.

Orange – Something you disagree with.

Brown – Something you will try.

Red – The most important thing you

learned.

Yellow – Something important you found in your manual.

Green – Share your first next step as a

result of this training.

Blue – Something you agree with.

Orange – Something you disagree with.

Brown – Something you will try.

progress monitoring pennsylvania’s seven-step process

Documents

assessment types

assessment tools

type of assessment

student progress

monitoring growth

students ieper

teacher of students

instructional changes