Download - A School-Based Reading Program Evaluation Michael F. Lewis, Ph.D. Niagara Falls City School District
A School-Based Reading Program
Evaluation
Michael F. Lewis, Ph.D.
Niagara Falls City School District
Changing Education Funding
Current changes in educational funding– New federal grant opportunities– Private grant opportunities
This funding often requires increased accountability
Accountability
Landscape in the U.S. has shifted to mandated levels of accountability in many areas due to recent events– Deceptive accounting in business
ENRON
– Abuse of school district budgets Roslyn School District (New York)
Accountability
Recently, when funding is awarded accountability is achieved through program evaluation– program evaluation required by independent
evaluators Mandated compensation (7%)
– Strict accounting of expenditures The budget is fixed
What is Program Evaluation?
Definition:– a formalized approach to studying the goals,
processes, and impacts of projects, policies and programs
– can involve quantitative or qualitative methods of social research (or both)
– People who do program evaluation come from many different backgrounds:
sociology, psychology, economics, social work
Types of Program Evaluation
A needs assessment examines the nature of the problem that the program is meant to address
The program theory is the formal description of the program's concept and design
Process analysis evaluates how the program is being implemented
The impact evaluation determines the causal effects of the program
Cost-effectiveness analysis assesses the efficiency of a program.
Impact Evaluation
Impact evaluation is the most common form of program evaluation– Determines (as best as possible) the effects of a
particular program along some criteria DARE (on decreasing drug use) PBIS (on reducing negative behaviors in school) RTI (on reducing # of students identified as LD)
Evaluation as ResearchAs conceptualized by: Stephen Truscott, Psy.D. (Georgia State University)
Big ‘R’ research– University Run– For Publication– Large Scale
Little ‘r’ research– Individually Run– For Information– Small Scale
Program Evaluation can be both or either
The role of the School Psychologist
We do EVERYTHING!!!– SP has evolved:
We are now Jack (and mostly Jill) of all trades
– We were special education evaluators– We are now increasingly responsible for
many activities related to General EducationIncluding evaluation of programs
Program Evaluation In Action
Niagara Falls School District evaluation of: Fast ForWord (FFWD)
– Computer-based– Auditory Processing and Literacy Skills– Timed ‘protocols’ for going through program
50, 90, and 120 minute protocols
The Problem: FFWD has never appeared in peer-reviewed literature
The Problem (continued)
Fast ForWord– Proprietary program– No real scientific evaluation of program– How do we know it is really effective?
– We do a program evaluation on student’s who use FFWD…
FFWD Evaluation
The Subject– Administration of FFWD to entire 2nd grade– 50 minute protocol (run every day)
The Evaluation– Pre-test/Post-test design– Evaluate every 2nd grader in reading– Analyze findings for increase in reading scores
FFWD Evaluation
To evaluate you must have a measure The Measure: GRADE
– Group– Reading– Assessment (and)– Diagnostic– Evaluation
Standardized, norm-referenced
FFWD Evaluation
To evaluate you must have a measure The other Measure: DRA
– Diagnostic– Reading – Assessment
(This is a tool we use locally for Reading level)
FFWD Evaluation
The Procedure:
Every 2nd grader participated– Daily 50-minute FFWD protocol– Ran for 20 weeks– Took GRADE before and after FFWD
– Classroom instruction did not change
FFWD Evaluation
The Procedure (a summary)
1) GRADE pre-test
2) 20 weeks of FFWD (50 minute protocol)
3) GRADE post-test
4) Score and analyze GRADE results
5) Determine effects of FFWD on reading
FFWD Evaluation
Statistics:– Imported GRADE data into SPSS
SPSS: Statistical Package for Social Sciences
– Computed Paired Samples t-tests on all students with pre/post GRADE data
– Same computation for DRA levels This determined statistical significance
– Not the whole story…
FFWD Evaluation
Statistics:– Statistical Significance vs. Effect Size
Effect size determines a measurable degree of the statistical significance
– Effect size reported in standard deviation form
Evaluation can have statistical significance but a small Effect Size
FFWD Findings Paired Sample t-Test
Paired Samples Statistics
99.74 388 15.085 .766
101.84 388 14.367 .729
95.10 386 13.864 .706
99.23 386 15.094 .768
96.67 381 13.986 .717
100.64 381 15.367 .787
Vocabulary CompositeStandard Score-PreTest
Vocabulary CompositeStandard Score-PostTest
Pair1
ComprehensionComposite StandardScore-PreTest
ComprehensionComposite StandardScore-PostTest
Pair2
Total Test StandardScore-PreTest
Total Test StandardScore-PostTest
Pair3
Mean N Std. DeviationStd. Error
Mean
FFWD Findings Paired Sample t-Test
Paired Samples Test
-2.11 10.980 .557 -3.20 -1.01 -3.778 387 .000
-4.13 9.575 .487 -5.09 -3.17 -8.473 385 .000
-3.97 8.705 .446 -4.84 -3.09 -8.892 380 .000
Vocabulary CompositeStandard Score-PreTest -Vocabulary CompositeStandard Score-PostTest
Pair1
ComprehensionComposite StandardScore-PreTest -ComprehensionComposite StandardScore-PostTest
Pair2
Total Test StandardScore-PreTest - Total TestStandard Score-PostTest
Pair3
Mean Std. DeviationStd. Error
Mean Lower Upper
95% ConfidenceInterval of the
Difference
Paired Differences
t df Sig. (2-tailed)
FFWD Findings
Statistical Significance vs. Effect size
– With a large sample it is highly likely that even a small change will indicate a statistical significance
≈380 students is a large sample size
FFWD Findings
Effect Size takes more realistic look at actual increase of significant findings– Hedge’s G: one way to calculate effect size
g = t √(n1 + n2) / √(n1n2) or g = 2t / √ N
Effect Size findings:– Vocabulary: g = .27– Comprehension: g = .35– Total Test: g = .33
These are considered small effect sizes
Errors Inefficiency– No need for testing of all students
Maintain statistical meaning w/ smaller random sample – this was ignored by district administration
– Administration Testing all students reduces control of standard
administration– this was ignored by district administration
Evaluation Design– No Control Group
No way of determining if FFWD caused increased GRADE scores w/out control group
???Questions???
Program Evaluation Reference:
– Posavac, E. & Carey, R. (2006). Program Evaluation: Methods and Case Studies (7th Ed.). Prentice Hall, New York, NY.
Contact:– [email protected]