promoting a culture of evidence through program evaluation patti bourexis, ph.d. principal...
TRANSCRIPT
Promoting a Culture of Evidence
Through Program Evaluation
Patti Bourexis, Ph.D.Principal ResearcherThe Study Group Inc.
OSEP Project Directors’ Conference July 21, 2009
Session purpose
Stimulate thinking and discussion about the important outcomes of evaluation for project directors.
Illustrate frameworks and processes that project directors can use to support both project implementation and accountability.
The Study Group Inc.
Session themes• Project directors can use evidence to assure sound
project planning, quality implementation and maximum results.
• Evaluation tools and processes can help project directors do the best job with project planning, implementation and accountability.
• Part D programs have their own priorities for evidence, which project directors need to support.
The Study Group Inc.
First theme
Project directors can use evidence to assure sound project planning, quality implementation and maximum results.
The Study Group Inc.
Culture of evidence
Shared value and belief about the importance of evidence in project planning, implementation, management and accountability.
The Study Group Inc.
Kinds of evidenceResearchPilot-testsModels
theoretical
experientialexperimental
Needs assessmentExpert opinionExperience and expertiseExternal review
The Study Group Inc.
Kinds of evidencePrior evaluationsStatutes/regulations/policiesStandardsDatabasesProducts/artifactsDocumentationAssessments
individual
programmaticorganizational
The Study Group Inc.
Characteristics of a culture of evidence
• Evidence is used throughout the project’s life cycle.• Numerous design and implementation changes occur during the projects’
life cycle.• Project outcomes are clearly stated in operational terms that are
understood by all staff.• Project directors make reliance on evidence a priority.• Processes for gathering and applying evidence are explicit and integrated
into project planning, implementation and management tasks.
The Study Group Inc.
Characteristics of a culture of evidence
• Staff are expected to use evidence to enhance the quality of their work.
• Staff have a common language and understanding about what evidence is, how to gather it, and how to apply it to improve project performance.
• There are opportunities for staff to expand their abilities to gather and apply evidence.
The Study Group Inc.
Two characteristics of culture of evidence
Evidence is used throughout the project’s life cycle.
Numerous design and implementation changes occur during the project’s life cycle.
The Study Group Inc.
Project life cycle
The Study Group Inc.
Project planning
Why evidence is important.Ensures the credibility of the project design: that it
reflects our knowledge about successful projects of this type.
Demonstrates relevance of the project to the target audience and situation.
Builds confidence that the project will succeed.
The Study Group Inc.
Project planning
The kinds of evidence that are important.ResearchModels
theoretical
experiential
experimental
DatabasesNeeds assessmentPrior evaluationsStandards
The Study Group Inc.
ED mission & principles of high-quality professional development
Focuses on teachers as central to student learning, yet includes all other members of the school community.
Focuses on individual, collegial and organizational improvement.
Is planned collaboratively by those who will participate in and facilitate that development.
Requires substantial time and other resources.
www.ed.gov/G2K/bridge.html
The Study Group Inc.
Project implementation
Why evidence is important.Show that project activities are implemented as
planned.Modify activities when performance falls short of
expectation or to take advantage of opportunities.Demonstrate that the outcomes are feasible and
reasonable.Engender confidence that the project is well-managed
and on track to realize its outcomes.
The Study Group Inc.
Project implementationThe kinds of evidence that are important.Pilot-testsDocumentationAssessments
individual
programmaticorganizational
Products/artifactsExternal reviewExperience and expertiseDatabases
The Study Group Inc.
Stability and sustainability
Why evidence is important.Confirm that the project’s implementation is stable.Show that the project outcomes are reliably
achieved and attributable to the project.Demonstrate the importance of the project to the
target audience and situation.Plan sustainability.
The Study Group Inc.
Stability and sustainabilityThe kinds of evidence that are important.Assessments
individual
programmaticorganizational
Products/artifactsDocumentationDatabasesStatutes/regulations/policiesExpert reviewExpert opinion
The Study Group Inc.
Third characteristic of a culture of evidence
Project outcomes are clearly stated in operational terms that are understood by all staff.
The Study Group Inc.
Capacity-building“Capacity building involves developing the collective ability---
dispositions, skills, knowledge, motivation and resources---to act together to bring about positive change…” (Fullen, 2005).
“Organizational capacity is its potential to perform---its ability to successfully apply its skills and resources to accomplish its goals and satisfy its stakeholders’ expectations” (Horton, et. al, 2003).
M. Fullen. (2005) Leadership and Sustainability: System Thinkers in Action. Corwin Press: Thousand Oaks, CA.D. Horton, et.al. (2003). Evaluating Capacity Development. International Service for National Agricultural Research: The Hague, The Netherlands.
The Study Group Inc.
Indicators of increased capacityIncreases in staff disposition, knowledge and skill.Increases in use of evidence-based interventions. Closer ties to networks for exchange of resources and expertise.Improvements in policies, processes and management practices.Increases in resources devoted to the particular issue or concern
(e.g., money, staff FTE).Changes in agency priorities.
The Study Group Inc.
SummaryCharacteristics of a culture of
evidence• Evidence is used throughout the project’s life cycle.• Numerous design and implementation changes occur during the projects’
life cycle.• Project outcomes are clearly stated in operational terms that are
understood by all staff.• Project directors make reliance on evidence a priority.• Processes for gathering and applying evidence are explicit and integrated
into project planning, implementation and management tasks.
The Study Group Inc.
SummaryCharacteristics of a culture of
evidence• Staff are expected to use evidence to enhance the quality of
their work.
• Staff have a common language and understanding about what evidence is, how to gather it, and how to apply it to improve project performance.
• There are opportunities for staff to expand their abilities to gather and apply evidence.
The Study Group Inc.
