is your school improving outcomes for students with disabilities: guesswork or science? presented by...
TRANSCRIPT
Is Your School Improving Outcomesfor Students with Disabilities:
Guesswork or Science?
Presented by
The Elementary & Middle Schools Technical Assistance Center
The American Institutes for Research
EMS CElementary & Middle SchoolsTechnical Ass is tance Center
www.emstac.org
TA
EMSTAC Model:Insider – Outsider Approach
TA Support
School District
TA Liaisonoutside the district
Linking Agentinside the district
Presentation Goals
• Provide an overview of the importance of evaluation efforts for school-based interventions involving students with disabilities
• Identify key steps in planning for and conducting an evaluation
• Show EMSTAC district “success stories,” demonstrating effective interventions
The Problem
• Most schools have multiple programs or interventions occurring; many are targeted exclusively or partly at students with disabilities.
• In many cases, however, the importance of a sound evaluation effort can be overlooked, underappreciated, and difficult to achieve.
Common Evaluation Challenges
• No “true” baseline data
• Not enough data have been collected
• Comparable data have not been collected
• Not enough time to collect desired data
• Longitudinal gaps (missing years)
• Separating the effects of one intervention from another
What Happens When You Don’t Evaluate?
• Inability to demonstrate that interventions are working
• Difficulty in obtaining sustained funding
• Problems with “scaling up” the intervention to other sites
• Dissemination of best practices to other districts & states is hindered
Evaluation is More Important Than Ever
• The “era of accountability:” Increased pressure from governing bodies to demonstrate results
• Too much focus on “just make things better”
• Budget difficulties at federal, state, & local levels
• IDEA ‘97: Students with disabilities must be included in accountability efforts
First Step: Getting a View of the Big Picture
• Program planning and evaluation go hand in hand
• Tailor to program purpose & goals
• Identify the purpose of the evaluation requirements
• Consider program reporting requirements
• Think ahead about available resources
• Be sensitive to the local context for the evaluation
• Identify program reporting
The Big Picture
Program Reporting
Requirements Evaluation
Evaluation Purpose
Local Context Time & Resources
Program Purpose & Goals
Second Step: Identifying the Evaluation Questions
• Questions related to program implementation– Program context
– Program delivery
– Access to the program
• Questions related to program impact– Impact on student performance
– Impact on teacher capacity
– Impact on moving research to practice
Third Step: Selecting a Design that Will Provide
the Data You Need• Experimental designs
• Quasi-experimental designs
• Simple before & after studies
• Time series designs
• Ethnographic research
Fourth Step: Identify & Develop Tools for
Collecting Data • Direct observation • Records & documents • Physical artifacts• Information from school
administrators, teachers, students, & parents
Special Considerations Time Reliability & validity Training data collectors Permission to collect data
& informed consent
Fifth Step: Data Analysis
• Storing data
• Quantitative analyses: Descriptive and univariate statistics
• Quantitative analyses: Inferential statistics
• Qualitative analysis
Sixth Step: Reporting the Findings
• Formal and informal reports
• Targeting the audience
• Keep it simple and straightforward
• Brief reports
• Website
• Public reports
• Support program improvement
EMSTAC “Success Stories”• Implementation looks different in each district
– various resources, time, staff, & experience with evaluation
• Examples of EMSTAC supported sites with positive student outcome results– Allegany County, MD
– Detroit, MI
– East Grand, CO
– Los Angeles, CA
Allegany County, Maryland
• Planning– Evaluation planned after program implementation
• Evaluation questions– Will the program and linked professional development
help general education teachers make instruction meaningful for all students?
– Are we going to see improvement in test scores?
• Design– Simple before and after design (Norm-referenced)
Allegany County, Maryland
• Data collection– Test results (i.e. CTBS, Curriculum Based Measures)
– Teacher surveys
• Data analysis– Analyze final scores
– Challenges with student attrition
• Reporting– Published data in department newsletter
– Plan scale up efforts
Allegany County, Maryland
Early Literacy Program EMSTAC assisted with needs
assessment, in 1998 Implemented portions of Early
Literacy Program (MSU) Training occurred in summer 1999 Data collected are based on results
of CTBS pre/post scores (1998/2000).
Language : 43 to 59.5 (28%) Lang. Mechanics : 43 to 54.0
(20%) Lang. Composition : 47 to
57.0 (18%)
0
10
20
30
40
50
60
Lng
uage
Lan
gM
ech
Lan
gC
omp
19982000
Detroit, Michigan Project ACHIEVE
Internal needs assessment, program selection & training process took a year
1999 began implementation of program in one middle school
by 2000-2001, school saw decrease in violations of code of conduct
• Class 1 referrals : 1,914 to 931 (51%)
• Class 2 referrals : 394 to 227 (42%)
• Class 3 referrals : 18-8 (56%).
0
400
800
1200
1600
2000
Class 1 Class 2 Class 3
19992001
East Grand, Colorado Literacy across curriculum
District conducted needs assessment
EMSTAC assisted with program selection via web
Implementation began in 1999 in one school
Results are based on pre-post test results (1999, 2001) from CO State Assessment Program
Reading : 5% (70-75) Math : 16% (50% to 66%) Writing unchanged
0%
10%
20%
30%
40%
50%
60%
70%
80%
Reading Math
19992000
Los Angeles, California
Peer Assisted Learning Strategies District conducted needs assessment PALS implementation began in 2000 Measures included CBM tools and probes (fall and spring) Probes indicated that during 2000, all grade levels using
program saw increases in WCPM (Words read correctly per minute)
Thus far, second grade classrooms have had highest gains of between 40% to 97.
Conclusion
• Evaluation is increasingly important.
• There are a set of key principles that guide sound evaluation efforts.
• You don’t have to be an expert to organize & conduct a sound evaluation.
• There are many useful resources for practitioners undertaking evaluation efforts (i.e. EMSTAC Evaluation Guide)
Presenters
• Jim Hamilton, EMSTAC Director
• Don Dailey
• Bradley Carl
• Suzanne Ritter
• e-mail us at www.emstac.org