using the pmrn and eci and ei indices to make effective decisions

Using the PMRN and ECI and EI Indices to Make Effective

Decisions

Elizabeth Crawford, MS, CCC-SLPDirector of Interventions

The Florida Center for Reading Researchwww.fcrr.org

2

Agenda General Uses of Data PMRN 101 Highlights of DIBELS Use Data to:

manage school level intervention resources monitor progress of specific groups of students run an effective class/student level data meeting interpret ECI and EI tables

Concluding Thoughts and Resources

3

General Uses of Data

4

Barriers to Using Data to Inform Instruction Grunwald Associates (out of

Maryland) surveyed 353 small, medium and large school districts asking them “What are the barriers to you using data-driven decision making more effectively?”

The next slides list the results

5

Grunwald Associates Page 0

Barriers to More Effective Data-Driven Decision MakingLack of training and interoperability are the main barriers to more effective data-driven decisions. A lack of understanding in what to do with the data is a barrier that’s particular to large and medium districts.

Barriers to Data-Driven Decision MakingTo what degree does your district consider the following to be barriers to

a more effective data-driven decision making process? (N=353)(Percent Responding 8, 9, or 10 on 10-point scale) Large (A)

School District Size

Significantly higher

Medium (B) Small (C)

50%

42%

39%

36%

35%

31%

24%

24%

22%

Lack of training

Interoperability - systems that are unable toshare/exchange data

Lack of understanding in what to do with thedata

Absence of clear priorities on what datashould be collected

Failure to collect data in a uniform manner

Outdated technology / legacy systems

Low quality data - inaccurate or incomplete

Timing of data collection

User interface is too complicated tounderstand reports

47%

44%

41%

36%

38%

34%

24%

23%

19%

55%

42%

44%

37%

35%

30%

25%

25%

24%

42%

36%

17%

28%

28%

22%

20%

23%

21%

C

C

6

Barriers to more effective data-driven decision making

Lack of training 50%

Interoperability – systems that can’t share data 42%

Lack of understanding about what to do with the data 39%

Absence of clear priorities 36%

Failure to collect data uniformly 35%

Outdated technology 31%

Incomplete or incorrect data 24%

Timing of data collection 24%

User interface is too complicated to understand reports

22%

7

PMRN 101

8

What is the PMRN?(Progress Monitoring and Reporting Network)

A Web-based data management system that provides:

a convenient place for entering and organizing the results of student assessments

a secure, centralized, easily accessible location for the storage of student information

a tool for timely and helpful reports so that educators can effectively analyze data, plan instruction, and communicate student progress

9

The PMRN is located at https://pmrn.fcrr.org.To Sign In to the PMRN, you will need the User Name and Password that were E-mailed to you, as well as your Identifier which was received in a letter from the FCRR.

Signing In to the PMRN

Please note the “S” in https. The “S” indicates a secure network and is required for your web browser to find the PMRN.

https://pmrn.fcrr.org/

11

Active Header Menu

Availability of information is limited by the Users access level. The level of access determines if the User can select all or specific grades, teachers, students, and/or tests.

12

Types of Reports and how data is reported Status - Table Progress – Box and Whiskers and Pie chart (except at the

student level which only has Box and Whiskers) Class Progress Tracking – Bar graph Historical - Box and Whiskers Risk Level - Table Recommended Level of instruction – Table and/or Pie

chart Student Recommended Instructional Level - Table Grade Summary – Box and Whiskers Comparison - Box and Whiskers Demographics – Table and Bar graph Cumulative - Table Ongoing Progress Monitoring – Line graph Year End Outcome – Pie chart and Table

13

Box & Whiskers Graph

The Box & Whiskers Graph is a type of distribution report. It is used to report how an individual or group compares to a larger group. The Box & Whiskers used in the PMRN is a modified version of the those that report quartiles.

14

Highlights of DIBELS

15

GRADE DIBELS MEASURE READING COMPONENTASSESSED

Kindergarten Initial Sounds, Phoneme Segmentation

Letter Naming, Nonsense Words

Phonemic Awareness

Phonics

First Phoneme Segmentation

Letter Naming, Nonsense Words

Oral Reading

Phonemic Awareness

Phonics

Reading Fluency

Second Nonsense Words

Oral Reading

Phonics

Reading FluencyThird Oral Reading Reading Fluency

16

Risk Level Key

18

SY0607 Florida DIBELS Recommended Level of Instruction Decision Rules

http://www.fcrr.org/assessment/pdf/ril_decisionrules_0607.pdf

19

Use Data to… Manage school level intervention

resources Monitor progress of specific groups

of students Run an effective class/student

level data meeting Interpret ECI and EI tables

20

Strong Leadership is critical to success The great leaders are like the

best conductors - they reach beyond the notes to reach the magic in the players. - Blaine Lee

21

Use Data to… manage school level intervention resources

22

School Level Reports “The Chart” Highlight:

School Status (“Home Base”) School Demographics

23

“The Chart” Use the chart as a “quick start”

guide to the PMRN for the School Level Reports

Refer to separate handout The PMRN User’s Guide is the

complete manual and should be read and referenced for specific questions and information

26

Using the School Status Report – “Home Base” Make a list of school wide resources Who needs extra support? (High Risk,

Moderate Risk, Low Risk) Which Grades? Which Teachers?

Which skills need to be emphasized in professional development?

28

School Status Report – “Home Base”

29


30


31

Navigating from the School Status Report – “Home Base”

School Grade Summary – Click on the words ‘grade summary’ under the grade want to see

Class Status – From the extended view, click on the teacher’s name

School Progress – From Grade Summary, click on ‘show progress report’ at bottom

Class Progress – From the School Progress, use active header menu to select teacher/class of interest

School Comparison – From School Progress, click on ‘show school comparison’ at bottom

** Any time you click on the number below a specific measure you will get a progress report and depending on the level of the report you started from will determine which level of progress report you get

32


33

School Grade Summary Report

34








35


36

Class Status Report

37








38

School Grade Summary Report

39

School Progress Report

40








41


42

Class Progress Report

43








44


45

School Comparison Group

46








47


48


49

Use Data to… monitor progress of specific groups of students

51

School Demographic Reports

Demographic data is available, by assessment period, at the school and district levels for individual measures and the Recommended Level of Instruction. The data file allows all of the data to be downloaded so that Users can complete their own analyses.

54

School Demographic Reports

You can also display a bar graph for a specific demographic group from this report. The next slides shows a bar graph for kindergarten students at the third assessment by Ethnicity. The total number of students in each category is in ( ) next to the group name.For example Hispanic (17)

57

Use Data to…run an effective class/student level data meeting

58

Does the district provide the tools and training to interpret and query data?

Have data teams developed a process for identifying, recommending and implementing intervention based on data?

Do teachers have access to data in an easy-to-use format soon after assessment?

Using Data Checklist

59

Data Utilization and AnalysisKey Characteristics of Data Meetings: Held on a regular basis (bi-weekly or

monthly) Attendees were able to make school

level decisions and personnel changes (usually the principal)

Systems/worksheets used for structure Data Compiled from multiple sources Follow-up

60

Class/Student Level Data Meeting Some principals start with discussing:

Any students who made great gains (celebrations)

Surprises in results (or discrepancies between impressions and actual data – if so, might look at student answer sheet to get more specific info)

Any students of great concern (and any circumstances that would be of importance – for example, the student has been tardy for the past 20 days missing all whole group instruction)

If any ESE, social work or other services are involved

Remind staff to look at scores not just colors and to keep ‘next target’ at bottom of column in mind when looking at scores

61

Class Level Reports Two main class level reports can be used to

run the meeting (with worksheets) Class Status Reports

To determine groups and areas of focus You might also want to have the status report from

the previous assessment period to compare scores Class Recommended Level of Instruction

To determine effectiveness of instruction and target students

62

Class Status Report The Class Status report from the PMRN will

help you answer 3 main questions:

1. Who needs extra support?

2. How should groups be formed?

3. Which skills need to be emphasized? Refer to “Class Status Report” Worksheet

Handout

64

Class Status Report

• 1st Grade Class• Assessment

Period 2• Full year

students only• You can get here

from ‘School Status’ by clicking on teacher’s name when the grade is extended

65

Who Needs Extra Support?

• High Risk:5• Moderate Risk: 7• Low Risk: 4

• Look at the “Next Target” to

help guide grouping

66

How will small groups be formed?

• Group 1:

• Group 2: • Group 3:

• Group 4:

67

How will small groups be formed?

• Group 1: Students 2,4

• Group 2: Students 1,3,5,6,7

• Group 3: Students 8,9,11,12

• Group 4: Students 13,14,15,16,10 (10 to get peer modeling to improve fluency)

68

Which skills need to be emphasized?

• Group 1:• Group 2:• Group 3:• Group 4:

69

Which skills need to be emphasized?

• Group 1: PA and Phonics

• Group 2: Phonics and Fluency

• Group 3: Phonics and Fluency

• Group 4: Fluency and Comprehension

70

Calculate the Class Level ECI, EI-I, and EI-S How do you do it? AND…what does it all mean??

Refer to “DIBELS Data Analysis Conference” Worksheet Handout

72

Class Recommended Level of Instruction Report

73


This report provides a summary of the students’ overall progress. It can be used to get an overall sense of instructional/risk levels in the class and to calculate the Effectiveness of Core Instruction (ECI) index and the three Effectiveness of Intervention (EI) indices.

74


This is the same type of report with “all” students included. Students 5, 6, 7, and 19 would not be used to calculate the ECI and EI indices because they were not in the school for all three assessment periods. This is why we select the full year only function for calculation.

75


You can use the pie charts to help you fill out the top part of the DIBELS worksheet

Initial Level = Low riskStrategic Level = Moderate RiskIntensive Level = High Risk

Assessment #3% Initial = 44% Strategic = 50 % Intensive = 6

76


To calculate ECI – Effectiveness of Core Instruction for the year, use only those students that were at Initial Instruction during the 1st assessment and remained in the class throughout the year (that is why we select full year only).Divide the number of the students that finished at Initial Instruction for the 3rd assessment (5) by the number of students that were at Initial Instruction for the 1st assessment (9). The higher the percentage the better.

77


Effectiveness of Core Instruction

# stayed initial 5--------------------- = -------# started initial 9

5--- = 56% ECI = 56% 9

Target Students: Students 3, 6, 8, 9 (Initial to Strategic)

78


To calculate EI – Effectiveness of Intervention for the year, use only those students that were in the class for the first assessment and the last assessment (full year only).Take the number of students who finished the year at ‘initial’ or ‘strategic’ (if they started at intensive) level by the students who started the year at some level of risk (intensive or strategic)

The higher the percentage the better.

79


Effectiveness of Intervention

# moved to initial or strategic from Intensive 2--------------------- = -------# started strategic & intensive 7

2--- = 29% EI = 29% 7

Target Students: Students 4, 7, and 10 (strategic to intensive during assessment 2)

80


Effectiveness of Intervention-Strategic

# moved to initial 1--------------------- = -------# began strategic 5

1--- = 20% EI-S = 20% 5

Target Students: Students 4, 7, and 10 (strategic to intensive during assessment 2)

81


Effectiveness of Intervention - Intensive

# moved to strategic or initial 1--------------------- = -------# began intensive 2

1--- = 50% EI-I = 50% 2

Target Students: Student 5

82

What do all of these indices mean for my school and teachers? This particular teacher in 1st grade is struggling both with

maintaining children at grade level and reaching her struggling readers:

ECI = 56% EI = 29% EI-S = 20% EI-I = 50%

You may want to first determine if other first grade teachers have similar indices and if so, you want to examine your Core Reading Program and especially your Intervention Programs

You can look at the School Recommended Level of Instruction Report to see how different grade levels are doing by comparing the pie charts from one assessment period to the next

The goal is to see the green increase and the red decrease – to make the pie chart look more like an olive!!

83

School Level Indices Breaking down the tables and text

you receive in your ECI and EI letters

84

ECI Definition Effectiveness of Core Instruction

(ECI) — this index shows the percentage of students who began the year reading “at grade level” and are continuing to meet grade level expectations at the mid-year or end of year assessment. This index tells you whether the core instruction is sufficiently powerful to produce a year’s growth in reading skill for a year’s worth of instruction for all students who begin the year at grade level.

85

ECI – “Technical”

Percentage of students who started on grade level and remained on grade level

Number of children used to calculate the index score

Percentile Rank compared to other RF schools

86

ECI – “School”

91% of the 45 kindergarteners who started the year on grade level remained on grade level.91 x 45 = 41 children41 out of 45 children remained on grade level

.97 x 75 = 73 children73 out of 75 children remained on grade level, 2 children did not remain on grade level

In first grade, my school is doing equal to or better than 96% of the RF schools on keeping our 1st grade students on grade level.

87

EI Definition Effectiveness of Interventions (El)* —this

index shows the percentage of your students who began the year at some level of risk for reading difficulties (reading below grade level) but who have grown rapidly enough to advance to a lower level of risk by the mid year or end of year assessment (i.e. they move from intensive to strategic, or from strategic to grade level). This index, and the two explained below, provide information about how effectively intervention programs are working to accelerate the development of students who are lagging behind in reading growth.

88

EI – “Technical”

Percentage of children who progressed to a lower level of risk

Percentile rank when compared to the other Reading First schools in Florida

The number of children used to calculate the index

89

EI – “School”

T81% of the 42 kindergartners who started the year at some level of risk, have made gains and are now at a lower level of risk. .81 x 42 = 34 students

In first grade, my school is doing equal to or better than 44% of the RF schools in moving children to strategic or grade level.

.23 x 30 = 7 students7 out of 30 3rd graders who started at some level of risk have moved to a lower level of risk. 23 students are still at some risk.

90

EI-I Definition Effectiveness of Interventions

for Intensive Students (EI-I) — this index shows the percentage of students that moved from the “intensive” risk category to the “strategic” risk level or to grade level.

91

EI – I “Technical”

Percentage of students who started the year at “intensive” and made enough progress to be “strategic” or “grade level”

Percentile rank when index score is compared to other Reading First schools in Florida

Number of students used to calculate the index.

92

EI – I “School”

93% of our 14 kindergarten students who started the year at “intensive” made progress to a lower risk level..93 x 14 = 13 students made growth, 1 is still “intensive”

In second grade, we are doing an equal or better job than 47% of the RF schools at moving our children out of the “intensive” category.

In all grades, K-3, of the 46 students who started at “intensive” 50% (or 23) of them made growth to “strategic” or “grade level.”

93

EI-S Definition Effectiveness of Interventions

for Strategic Students (EI-S) — this index shows the percentage of students that moved from the “strategic” risk level to grade level.

94

EI – S “Technical”

Percentage of students who started the year at “strategic” moved to “grade level”

Percentile Rank of index when compared to all other RF schools in Florida

Number of students used to calculate the index

95

EI – S “School”

.56 x 27 kindergarten students who started the year at “strategic” moved to “grade level” = 15

27-15 = 12 are still “strategic”

In 1st grade, when compared to the other 583 RF schools in FL, We are doing equal to or better than 48% of them at moving first grade “strategic” students to “grade level”

In 3rd grade, 62% of the 16 children (10) who started at “strategic” are now at “grade level”

Worksheet Slides

97

ECI – “Worksheet”

___% of the ___ kindergarteners who started the year on grade level remained on grade level.___ x ___ = ___ children___ out of ___ children remained on grade level

.___ x ___ = ___ children___ out of ___ children remained on grade level, ___ children did not remain on grade level

In first grade, my school is doing equal to or better than ___% of the RF schools on keeping our 1st grade students on grade level.

98

EI – “Worksheet”

T___% of the ___ kindergartners who started the year at some level of risk, have made gains and are now at a lower level of risk. .___ x ___ = ___students

In first grade, my school is doing equal to or better than ___% of the RF schools in moving children to strategic or grade level.

.___ x ___ = ___ students___ out of ___ 3rd graders who started at some level of risk have moved to a lower level of risk. ___ students are still at some risk.

99

EI – I “Worksheet”

___% of our ___ kindergarten students who started the year at “intensive” made progress to a lower risk level..___ x ___ = ___ students made growth, ___ is/are still “intensive”

In second grade, we are doing an equal or better job than ___% of the RF schools at moving our children out of the “__________” category.

In all grades, K-3, of the ___ students who started at “intensive” ___% (or __) of them made growth to “strategic” or “grade level.”

100

EI – S “Worksheet”

.___ x ___ students who started the year at “strategic” moved to “grade level” = ___

___-___ = ___ are still “strategic”

In 1st grade, when compared to the other 583 RF schools in FL, We are doing equal to or ____ than ____% of them at moving first grade “strategic” students to “grade level”

In ___ grade, ___% of the ___ children (___) who started at “strategic” are now at “grade level”

101

ECI – “Worksheet”

___% of the ___ kindergarteners who started the year on grade level remained on grade level.___ x ___ = ___ children___ out of ___ children remained on grade level

.___ x ___ = ___ children___ out of ___ children remained on grade level, ___ children did not remain on grade level

In first grade, my school is doing equal to or better than ___% of the RF schools on keeping our 1st grade students on grade level.

102

EI – “Worksheet”

T___% of the ___ kindergartners who started the year at some level of risk, have made gains and are now at a lower level of risk. .___ x ___ = ___students

In first grade, my school is doing equal to or better than ___% of the RF schools in moving children to strategic or grade level.

.___ x ___ = ___ students___ out of ___ 3rd graders who started at some level of risk have moved to a lower level of risk. ___ students are still at some risk.

103

EI – I “Worksheet”

___% of our ___ kindergarten students who started the year at “intensive” made progress to a lower risk level..___ x ___ = ___ students made growth, ___ is/are still “intensive”

In second grade, we are doing an equal or better job than ___% of the RF schools at moving our children out of the “__________” category.

In all grades, K-3, of the ___ students who started at “intensive” ___% (or __) of them made growth to “strategic” or “grade level.”

104

EI – S “Worksheet”

.___ x ___ kindergarten students who started the year at “strategic” moved to “grade level” = ___

___-___ = ___ are still “strategic”

In 1st grade, when compared to the other 583 RF schools in FL, We are doing equal to or ____ than ____% of them at moving first grade “strategic” students to “grade level”

In ___ grade, ___% of the ___ children (___) who started at “strategic” are now at “grade level”

105

Key ‘follow-up’ sections on worksheet Items Discussed Roles and Responsibilities

Teacher will… Reading Coach will… Principal will…

106

Follow Up is Key

107

Concluding Thoughts and Resources

108

Concluding Thoughts Use PMRN Reports to make school,

classroom, and student level decisions: Monitor progress Determine which students need more

intensive instruction Offer instructional strategies to the teacher Determine types of professional development

by grade level or individual teacher By asking some of the following

questions…

109

Is the core reading program implemented with fidelity? Effectiveness?

Are the interventions being implemented with fidelity? Effectiveness?

Is there a need for more professional development for a particular area of reading?

Is there a need for more intensive coaching? Are materials allocated to maximize learning? Is personnel use maximized to focus on reading

instruction? Is the instructional day scheduled such that reading is

THE priority?

Important Considerations

110

Florida Center for Reading Research 227 North Bronough Street,

Suite 7250 Tallahassee, FL 32301(850) 644-9352 - Phone (850) 644-9085 (Fax)www.fcrr.org

PMRN Help Desk 850-644-0931 (Voice)1-866-471-5019 (Toll Free)850-645-1700 (Fax)[email protected]

User’s guidehttp://www.fcrr.org/pmrn/userguides.htm

http://www.fcrr.org/

mailto:[email protected]

http://www.fcrr.org/pmrn/userguides.htm

http://www.fcrr.org/pmrn/userguides.htm

111

PROCESS TO PLAN FOR FOLLOW-UPFOCUS ROLES

School-based Non-school Based

Principal Reading Coach

Teachers District RFPD CoordinatorGrade-level Leader Classroom

Teacher

Instructional Content

Explicit Delivery

Assessment

Professional Development

A) Resources 1.Human resources 2.Materials B) Time 1. Master schedule 2. Planning period C) Services 1.Workshops 2.Follow-up

QUESTIONS??THANK YOU!!

Liz Crawford, MS, CCC-SLPDirector of Interventions

[email protected]

using the pmrn and eci and ei indices to make effective decisions

Documents

level of access

users access level

effective datadriven

whiskerscomparison box

student progressthe

highlights of dibels

plan instruction

whiskersdemographics