fidelity of implementation a tool designed to provide descriptions of facets of a coherent whole...
DESCRIPTION
Developing an Evaluation Plan A PowerPoint program evaluation tool for Literacy Collaborative Schools (Revised 3/12)TRANSCRIPT
Fidelity of Implementation
A tool designed to provide descriptions of facets of a coherent whole school literacy initiative.
A tool designed to provide reflection opportunities that result in ongoing progress of student achievement over time.
A tool designed to help literacy teams establish and prioritize goals for the upcoming year.
Fidelity of Implementation Package
Fidelity of Implementation Tool Fidelity of Implementation Achievement Data Fidelity of Implementation Teacher Surveys
Developing an Evaluation Plan
A PowerPoint program evaluation tool for Literacy Collaborative Schools
(Revised 3/12)
Developing an Evaluation Plan
This presentation is intended to guide school-based literacy teams through the process of developing a school- or district-level evaluation plan. It is divided into three sections, which can be used separately or together.
Developing an Evaluation Plan
The three sections correspond to the three goals of the presentation:
Goal 1: To understand the purpose of the evaluation plan and the process for developing it
Goal 2: To formulate evaluation questions and to develop a plan to answer those questions
Goal 3: To put the plan into action
As a team...
Reflect on the following slides Share your thinking Ask questions Have a scribe take notes Make decisions Formulate evaluation questions and develop a
school- or district-level evaluation plan
Goal 1
To understand the purpose of the evaluation plan and the process for developing it.
Purpose of the Evaluation Plan
All Literacy Collaborative schools develop a school- or district-level evaluation plan to track student progress and evaluate the effectiveness of their Literacy Collaborative implementation over time.
How the process works...
Fidelity of Implementation Timeline – Getting Started
Literacy Team begins usingthe Fidelity of Implementation
Tool to describe Successes,Challenges, and Action Plans.
Mid- Year
Literacy Team develops an evaluation plan and
submits it to the training site by the end of September.
Spring/Summer
* This date may differ for district sites. Check with your trainer.
Fidelity of Implementation Future Timeline
Literacy team reviews evaluation questions & plan, modifies if necessary based on previous
year’s data results (beyond field year), and plans for current year’s
data collection Data are collected based
on the evaluation plan
Fall
During the year
Spring/Summer
Data are gathered, reviewed and analyzed
by the team and presented tothe university site.
•state the questions that have been identified by the team,•specify the assessments that the school will use for the next five years,•specify when the assessments will be administered,•include plans for analyzing and presenting results.
The evaluation plan will...
…be based on the information your school or district has to develop for other plans (Title 1 plans, School Improvement Plans) so you do not have to duplicate your efforts.
The evaluation plan can...
The evaluation plan can…
…include measures of teacher change as well as student-level measures of achievement. These could include teacher surveys or interviews that demonstrate positive changes in teacher practice or understanding over time.
Fidelity of Implementation: Teacher Surveys
Two Kinds of Teacher Survey:Implementation YearOngoing Implementation
Fidelity of Implementation: Teacher Surveys
Surveys are divided into three parts:ImplementationProfessional DevelopmentCoaching
..include results of some of the required classroom assessments listed in the Literacy Collaborative Classroom Assessment Charts (www.literacycollaborative.org). You may also use alternative assessments from the BAS-1.The information gathered from classroom assessments is designed to be used primarily to inform classroom instruction, but results from some of these assessments can be used to answer program evaluation questions as well.
The evaluation plan can...
For example, Benchmark Assessment System…
CAN BE USED TO EVALUATE PROGRAM EFFECTIVENESS if given in the fall and again at the end of the year to assess student progress in reading. Can be reported across an entire grade level to demonstrate changes in the number of students performing below, at, and above instructional level for each grade.
CAN BE USED TO INFORM INSTRUCTION if used at the beginning of the year to determine instructional level and placement in guided reading groups.
Example of an assessment used to inform classroom instruction
Running Records
Used twice a month to determine a student’s reading level, accuracy, use of sources of information, and reading behaviors. Teachers use this information to assist with group placement and individual instruction.
Make a List of Your Assessments
Make a list of all of the literacy assessments you currently administer.
Make sure you have listed all of the reading, writing, and word-study assessments at each grade level.
What information does each of these provide (e.g., area tested, type of score)?
Which of these assessments can be used to track change over a year, multiple years?
Identify any assessments that duplicate the kind of information you are looking for and choose which one to include.
Your list could include:
Literacy Collaborative Required Classroom Assessments– Reading– Writing– Phonics/Word-Study
Standardized Tests– State Mandated Grade Level Assessments– TerraNova, Gates-MacGinite, Stanford9, CAT5
District or Other Required Assessments– BAS -1
Other – student attitude surveys – teacher surveys
Goal 2
To formulate your evaluation questions and develop a plan to answer those questions.
•The examples on the following pages may help you think about the kinds of questions you might want to ask in the areas of reading, writing, and word-study.
•Read the examples and discuss as a team possible evaluation questions for your school. •As you consider possible questions, think of your stakeholders and what they might want or need to know.
Developing Questions Activity
Possible Evaluation Questions (cont’d)
What percentage of third grade students is scoring proficient or above on our state reading assessment each year? Are more students scoring proficient or above in reading as we implement Literacy Collaborative?
Possible Evaluation Questions (cont’d)
What is the average HRSIW score for first-grade at the beginning of each year? Across years?
What is the average HRSIW score for each class of incoming kindergartners in the fall?
Is the kindergarten teaching making a difference for first-graders as measured by changes in their average HRSIW scores in the fall over time?
Formulate your questions
• List the questions your team decides to pursue. Remember to make your questions clear, focused and measurable.
• Your questions may focus on one year only (fall to spring) or may focus on multiple years.
Revisit your list of assessments
Determine which assessments will allow you to answer your evaluation questions.
If you are not currently administering assessments that will allow you to answer your evaluation questions, identify assessments that will allow you to do so and add them to your list.
Write up your Evaluation Plan
Using all of the information that you have developed (your evaluation questions and your list of assessments), you can now write up your evaluation plan.
You can do it in chart, list, or table format.
Sample Evaluation Plans
Primary (K-2) Sample Evaluation Plan
1) Are K, 1st and 2nd grade students ma king progress in reading levels as measured by the Benchmark Text s? For each grade level, a. Are we being successful in moving students from below grade level to at and above grade level from fall to spring each year
(for K, fall = November)? b. How does the progress compare across years?
2) Is each g rade K cohort ma king progress over time in reading levels as measured by the Benchmark Texts from grade K to grade 2
(i.e., foll owing the same students over three years)? What percentage of students in each cohort is reading at or above grade level by grade 2?
3) Are K and 1st grade students mak ing progress in recognizing sounds as measured by HRSIW? For each gr ade level,
a. How does the average score in the fall compare to the average score in the spring each year? b. How does the progress compare across years?
4) Is each g rade K cohort ma king progress over time in recognizing sounds as measured by HRSIW from grade K to grade 1 (i.e., following the same students over two years)?
5) Are K and 1st grade students mak ing progress in identifying letters as measured by Letter ID? For each grade level,
a. How does the average score in the fall compare to the average score in the spring each year? b. How does the progress compare across years?
6) How does the percentage of grade 3 students performing at proficient levels on the state mandated reading and writing assessment (given in the fall of 3rd grade) compare across years?
a. How does the percentage of 3rd graders performing at proficient levels compare to district and state profic iency levels each year?
b. How do results compare when disaggregated by race/ethnicity and ELL status? 7) Are 1st and 2nd grade students improving in writing as measured by the district writing rubric? For each grade level,
a. Are we being successful in moving students out of the lower levels and into the higher levels from fall to spring each year? b. How does the progress compare across years?
8) Are K students mak ing progress in book handling skills as measured by Concepts About Print?
a. How does the average score in the fall compare to the average score in the spring each year? b. How does the progress compare across years?
9) Are K, 1st and 2nd grade students ma king progress in recording high frequency words as measured by Writing Voc abulary? For
each grade level, a. How does the average score in the fall compare to the average score in the spring each year (for K, fall = November or
December)? b. How does the progress compare across years?
Primary (K-2) Program Evaluation Plan Continued
Assessment Grades Administered
Purpose Frequency Presentation of Data
Lesley University Benchmark
Reading Assessment
K, 1, 2 To assess students’ reading levels
Fall & Spring of each year
Kindergarten will administer Nov., & Spring.
Fall-spring progress of each K, 1st, and 2nd grade cohort
Incoming and end of year levels across K, 1st, and 2nd grade cohorts over years
Longitudinal data of each K gr ade cohort over 3 years (same students over 3 years)
Writing Prompt 1st, 2nd To assess students’ ability to craf t a piece of writing. District rubric utilized f or scoring purposes.
Fall & Spring of each year Fall-spring progress of each 1st and 2nd grade cohort
Incoming and end of year levels across 1st and 2nd grade cohorts over years.
State Mandated Assessment
3rd To assess students’ levels in reading and writing compared to state
standards
Fall of each year
Fall scores across 3rd grade cohorts over years
3rd grade scores compared to district and state scores each year
Disaggregation of data by race/ ethnicity and ELL status
Concepts About Print
K To assess students’ book handling skills.
November & Spring Fall-spring progress of each Kindergarten student
Incoming and end of year levels across Kindergarten cohorts over years.
Letter Identification
K, 1 To assess students’ ability to identify letters in
isolation
Fall & Spring of each year Fall-spring progress of each K, and 1st grade cohort
Incoming and end of year levels across K, and 1st cohorts over years.
Hearing and Recording Sounds
in Words
K, 1 To assess students’ ability to hear and record sounds
in sequence.
Fall & Spring of each year Fall-spring progress of each K, and 1st grade cohort
Incoming and end of year levels across K, and 1st cohorts over years.
Longitudinal data of each K grade cohort over 2 years (same students over 2 years)
Writing Vocabulary
K, 1, 2 To assess ability to record high f requency words.
October, Spring Kindergarten administered in November or December for initial t esting.
Incoming and end of year levels across Kindergarten, first and second grade cohorts over years.
Making Sense of Quantitative Student Data
Thinking about numbers takes mental work, is not easy and is not an “objective” process.
Numbers have to be organized and summarized in some way in order to think about them.
Numbers cannot communicate on their own. Discussing and writing about numbers is a way to find meaning in numbers.
There are many ways of analyzing the same data, and the choices you make depend on your questions and interests. There is no “right” way and “wrong” way.
Which is why one person shouldn’t do it alone: analyzing data should involve multiple perspectives.
Changes in State Tests, 2003-2006
0
10
20
30
40
50
60
70
80
90
100
2003 2004 2005 2006
% o
f stu
dent
s Pro
ficie
nt o
r A
dvan
ced
3rd4th7th
Word Reading
0
5
10
15
Ms. Smith - KindergartenWord Reading 6.3 11.3 16.5
September J anuary J une
“Average word reading scores increased substantially over the year, from 6.3 in the fall to 11.3 in the winter to 16.5 in the spring.”
With all graphs, be mindful of scale. It’s usually best (though not always) to use the full scale of a measure when graphing it.
3rd Grade Scores, 2003-2006
0
5
10
15
20
25
30
35
40
45
50
2003 2004 2005 2006
% o
f Stu
dent
s Pro
ficie
nt o
r A
dvan
ced
3rd
3rd Grade Test Scores
0
10
20
30
40
50
60
70
80
90
100
2003 2004 2005 2006
% o
f Stu
dent
s Adv
ance
d or
Pro
ficie
nt
3rd
Summarizing results from your data analyses
Summaries should be short so that someone who isn’t going to read your whole report will be able to
quickly get an overview of student learning. (1-2 paragraphs)
Generalize across grade levels, across assessments, across the year, and across years (if you have multi-year data).
Include qualitative information about student learning as well as quantitative.
Summarizing Data Analyses: Example
“Overall, the K-2 students did well this year, with teachers reporting that students were enthusiastic about the literacy activities….etc. While only 40% of entering kindergarteners were reading on grade level according to the BAS, 70% were by the end of the year. Similarly, 1st and 2nd graders showed increases in the percentages of students on grade level, from roughly 50% in both grades in the fall to 75% of first graders and 60% of second graders in the spring. First grade scores were higher than last year’s first grade scores, but 2nd grade scores were slightly lower than last year’s 2nd grade scores. Most special needs students in all three grade levels made small gains in reading. In all three grades students showed particular strength in writing…”
Goal 3
To put your plan into action.
Your questionsHow you will answer them (which
assessments you will administer at different points during the year)
How you will analyze your data
Your evaluation plan consists of…
To put your plan into action, you will need to decide…
Who will be a part of the process? What will happen? When it will happen? What you will do with the information?
Make a Timeline
When will the assessments be administered?
Who will be responsible for administering, collecting, and analyzing the data?
When will the team gather to examine the data and talk about the results?
What needs to be done to facilitate this process?
Congratulations!
You have completed the first important steps toward tracking your students’ progress and evaluating the effectiveness of your Literacy Collaborative implementation over time.