evaluating the impacts of classroom assessment initiatives: benefits and potential approaches third...
TRANSCRIPT
Evaluating the Impacts of Classroom Assessment Initiatives: Benefits and
Potential ApproachesThird Annual Black Sea Conference, September 2014
Batumi, Georgia
Mathematica Policy Research: Ira Nichols-Barrer Matt SloanMillennium Challenge Corporation: Ryan Moore Jenny Heintz
MCC & Education
MCC’s Approach to Evaluation
Context for the Current Evaluation
Impact Evaluations to Inform Education Policy
Evaluation Evidence Informing Assessment Programs and Policies
Next Steps and Points for Discussion
Agenda
2
MCC Portfolio
23
26 Total Compacts13 Active3 Signed8 Closed
2 Terminated
26
$454 million invested in Education
745 educational facilities constructed or rehabilitated
4,160 instructors trained or certified
228,847 students participants of MCC-supported educational activities
61,848 graduates of MCC-supported educational facilities
Results as of March 2014
MCC’s Results in Education
4
MCC is committed to assessing the impacts of its programs through rigorous impact evaluation, conducted by independent research consultants
MCC impact evaluations contribute to a better understanding of success and cost-effectiveness of programs
Many countries all over the world seek opportunities to evaluate their programs to support evidence-based decisions. Impact evaluation can help inform education policy in Georgia
Rationale for Impact Evaluation and MCC’s Approach
5
MCC has hired Mathematica to design and implement an impact evaluation for the Improving General Education Quality Project– Improved Learning Environment
Infrastructure– Training Teachers for Excellence– Education Assessment Support
Mathematica is currently in the early design phase of the study
Context for the current evaluation
6
Evaluations can systematically inform policy: – U.S. Dept. of Education’s What Works Clearinghouse
Value of Evaluation Evidence to Policymakers
7
Opportunity to develop a pilot program with a rigorous impact evaluation focused on learning outcomes– Impact evaluations can target highly specific research questions
related to assessments. – Goal would be to align program implementation schedule with
expected changes in student learning
Many potential ideas to explore, such as:– Support additional schools to use standardized exams– Train teachers to use more diagnostic assessments more regularly
and consistently– Inform students or parents about exam performance – Encourage community engagement in understanding exam results
and activities related to student learning
Potential for Evaluation Research to Inform Assessment-Related Policies
8
Example: does greater use of assessment data for instructional feedback improve student learning?– Yeh (2007) meta-analysis examined the impacts of requiring
rapid formative assessment of students (2-5 times per week) with principal feedback
• Study found that the intervention improved learning at low cost among disadvantaged students in the U.S.
– Similar results from line of research on “data-driven instruction” at U.S. charter schools
Potential study approach for this research question: random assignment of intervention, activity, or program to classrooms or schools
Potential Research Questions for Assessment Policy (1)
9
Example: does informing community stakeholders about local learning outcomes improve school performance?
Mixed results from past research:– Andrabi et al. (2009) RCT in Pakistan found that distributing school
report cards improved learning outcomes by 0.10sd, with greater benefits in low-performing schools
– Lieberman et al. (2012) and Banerjee et al. (2010) RCTs in Kenya and India found that publicizing reading test results did not change citizen engagement, oversight of schools, or learning outcomes.
Research approach: random assignment of program to communities
Potential Research Questions for Assessment Policy (2)
10
Example: does support for greater community engagement in learning improve student outcomes?
Some promising results from past research:– Banerjee et al. (2010) RCT in India found that disseminating
exam results combined with training volunteer tutors significantly improved learning.
– Barr et al. (2012) RCT in Uganda trained school management committees to develop their own indicators and objectives for school performance and monitor results—the intervention improved student test scores.
Research approach: random assignment of intervention or program to communities
Potential Research Questions for Assessment Policy (3)
11
Questions for discussion:– What type of intervention would be of greatest interest
for a pilot study?• Teacher use of classroom assessment data• Impacts of providing various types of individualized
assessment information to students and parents• Role of community engagement in improving learning
outcomes?• Other ideas?
Education Assessment Support – Discussion Questions and Evaluation Ideas
12
Mathematica® is a registered trademark of Mathematica Policy Research.
Please contact:
– Matt Sloan• [email protected]
– Ira Nichols-Barrer• [email protected]
– Ryan Moore• [email protected]
– Jenny Heintz• [email protected]
For More Information
13