Policy Evaluation in Action: Overview of Policy Evaluation

Christopher D. Jones, PhDHealth Scientist

2012 Joint Annual Meeting of the Safe StatesMay 3, 2012

National Center for Injury Prevention and Control

Division of Injury Response

OVERVIEW OF POLICY EVALUATION

Development of Policy Evaluation Guide

Environmental scanBest practices & challenges

Review of literature

Key informant interviewsKey informant interviews

Evaluation and policy expert consultation p y p

What is Policy?

Law, regulation, procedure, , g , p ,administrative action, incentive, or voluntary practice

Implemented by governments and other institutions

Frequently reflected in resource allocations

Types and Levels of PolicyOrganizational

Local education agencies and/or schools or school districts

Private hospital or other healthcare delivery sites

Community-based organizations

Governmental agencies

Business, industry, or corporations

Professional associations or accrediting organizations

RegulatoryRegulatoryState

Federal

LegislativeLocal

StateState

Federal

Policy Development Framework 

I. Problem Identification

 

 

II. Policy AnalysisV. Policy 

Implementation and Compliance 

 

 

 

   

 

Policy Evaluation

III. Education and Awareness

IV. Policy Development and 

Enactment

 

 

 

Stakeholder Engagement  and Participation

  

What is Policy Evaluation?

Application of evaluation principles and methods to pp p pexamine content, implementation, or impact of policy

Purpose:Informing implementation

Demonstrating impacts and value of policyDemonstrating impacts and value of policy

Informing evidence base and future policies

Accountability for resources invested

Policy Evaluation

Problem Education & Policy

Policy Evaluate effects Problem

Identification Policy Analysis Education & Awareness Development &

EnactmentImplementation

& Compliance

Evaluate effects & impact

Content Evaluation Implementation Impact Content Evaluation Implementation Evaluation

Impact Evaluation

Policy Evaluation

Content evaluationDoes content clearly articulate goals, implementation, and intended change?

Implementation evaluationPolicy implemented as intended?

Impact evaluationP li d i t d d t /i t? Policy produce intended outcomes/impact?

Policy Evaluation Guide

Planning the evaluationgEngage Stakeholders

Describe Policy

F i th l tiFocusing the evaluationSelect Questions, Design and Indicators

Implementing the evaluationp gIdentify and Collect Data

Analyze Data

C i ti ltCommunicating results

Encourage partners to conduct policy evaluation

Lessons Learned

Policy evaluation is a new topicy p

Methods similar to program evaluation

Important differences:Level of analysis required

Lack of control and clear “boundaries” of policy implementation

Difficulty identifying equivalent comparison communityDifficulty identifying equivalent comparison community

Requires data collection on larger scale & scope

Broad range of stakeholders

Next Steps

Translation of NCIPC guidegExternal document for State & Local use

Anticipated 2013 release

SuggestionsDuring Panel Q & A

Core Violence and Injury Prevention Program:Policy Evaluation Assessment Results

Acasia B. Olson, MPHPublic Health Prevention Service (PHPS) Fellow

2012 Joint Annual Meeting of the Safe StatesMay 3, 2012

National Center for Injury Prevention and Control

Division of Injury Response – Core VIPP Team

Agenda

Background g

Project purpose

Methods and Analysis

Discussion of results

Question and Answer

C VIPP Aff P li Core VIPP: Affect Policy

Evaluation Focus

This evaluation project served to assess Core VIPP funded p jstate health departments experience with policy evaluation while identifying best practices in technical assistance around policy evaluation assistance around policy evaluation

Key Questions: y QWhat past and current experience do states have in evaluating policies?

What are the best types of technical assistance to ensure successful What are the best types of technical assistance to ensure successful evaluation of policy activities?

CDC’s Evaluation Framework

Engage Stakeholder

Engage Stakeholder

Describe the program

Describe the program

Ensure use and share lessons

learned

Ensure use and share lessons

learned

S d dStandardsUtility

FeasibilityPropriety

Focus the evaluationFocus the evaluation

Justify conclusions

Justify conclusions

p yAccuracy

Gather credible evidence

Gather credible evidenceevidenceevidence

Methods

Selection Randomly selected one non-RNL state within each regional network and a site visit state

ParticipantsParticipantsHI, SC, MN, CO, RI, KY

InstrumentDeveloped and piloted a 9-item open-ended interview script

Data CollectionT l h i t i (N 5)Telephone interviews (N=5)

In-person policy evaluation discussion (N=1)

Quality Assurance/Control

Analysis

Used qualitative methods (coding)q ( g)

Feedback were reviewed and sorted by question

Developed a priori categories to identify and code themes throughout each interview

Coded using Word comment feature

Results (Policy Evaluation Experience)

One of two key questionsy q

States are familiar with evaluation

Most states reported little to no experience with evaluating policies

Policy Evaluation Experience- Case Example

CHALLENGEInjury and Violence Program (IVP) “A” desires to evaluate their GDL law

They need data from the department of transportation (DOT) on injuries and motor vehicle (MV) issues

DOT unable to share MV data due to liability concerns

RESOLUTIONRESOLUTIONThe state mandated IVP and DOT to evaluate GDL

DOT required to release their data

Policy Evaluation Experience- Case Example

Outcomes: IVP and DOT evaluate GDL annually

Solid relationship between two agencies

Passage of GDL bill to permanent law in 2010Passage of GDL bill to permanent law in 2010

Opportunities found: Continual collaborations between IVP and DOT

IVP trained DOT on public health perspective of injury and MV issues

DOT provided funding to:p g• Link DOT crash data to SHD records

• Link EMS data to hospital records

ResultsResults(Challenges)

Data (access issues, incomplete, baseline, collection)( , p , , )

Figuring out what to measure

Lack of clear definition or guidance

Gaps in how states and local health departments conduct evaluation

Results(Opportunities)

Solid partnershipsp p

Aligning priorities

Positive reputation

Gaining access to datasets

Results

Tools Practices and/or Resources

•Safe States Alliance•Children’s Safety NetworkSuicide Prevention Resource Center

Tools, Practices, and/or Resources

•Suicide Prevention Resource Center•American Evaluation Association’s Resources and Innovation Network•Annie E. Casey Foundation

•Shaping Policy for Health•Johns Hopkins Summer Institute Training Series • Access to quality data•Data collection and partnershipsData collection and partnerships•Trusted state health department and academic institution partners

Results (Technical Assistance)

“Provide a document that keeps states accountable, p ,(e.g. a reference guide)”

“Share key injury specific examples or policies that states have made progress in”

“Information on states with successful evaluation experience who can offer one-on-one TA or advice”

“More guidance on how to ask and frame analysis”

Discussion

Policy evaluation is an emerging cross cutting fieldy g g g

Resources, tools, and subject matter experts are available

States want to understand policy evaluation

Next stepsContinue with monthly TA calls DIR training sessions webinars and Continue with monthly TA calls, DIR training sessions, webinars and SME conversations

Provide ready-made tools, documents/manual on policy evaluation (e g RE-AIM framework)*evaluation (e.g. RE-AIM framework)

Injury Center Policy Office is in the process of developing a State Policy Evaluation Guide

E i i f ti h i d t k ll b tiEngage in information sharing and network collaborationBronson et al., (2009) “Government, Politics, and Law.” American Journal of Public Health, 99(9):1576-83

Q & A

Thank you

Colorado – Lindsey Myers, MSy y ,

Hawaii – Robin Argue, MPH , Dan Galanis, PhD

Kentucky – Terry Bunn, PhD

Minnesota – Mark Kinde, MPH

South Carolina – Nichole Spivey, MBA

h d l d i Rhode Island – Beatriz Perez, MPH

Core VIPP Team

www cdc gov/injury/anniversarywww.cdc.gov/injury/anniversary

Happy Anniversary

CDC’s Injury Center!CDC’s Injury Center!Celebrating 20 years of saving lives and Celebrating 20 years of saving lives and

protecting our nation from protecting our nation from

injuries and violence!

National Center for Injury Prevention and Control