From Logic Models to Evaluation Planning and Implementation

Workshop: July 19, 2011 Susan Grantham, Ph.D.

Naomi Clemmons, MPH

Nancy Kasen, MS

John Snow, Inc.

OBJECTIVES:Participants will: Understand the importance of evaluation for making informed

program decisions Understand the importance of evaluation for communicating to

other stakeholders Understand logic model leads into an evaluation plan Know and practice steps involved in developing an evaluation

plan Know how to proceed with the development of an evaluation plan Know how to work with an outside evaluator

Overview of Workshop Review logic model development Discuss technical review findings Transition from the logic model to evaluation Walk through steps of developing an evaluation

plan Conduct two small group sessions to practice

key components of developing your evaluation plan

What is a Logic Model?

Picture/graphic of how an initiative, project, program work

Systematic way to show the connections among parts of a project Underlying logic – reflects implicit assumptions about

how change occurs (IF – THEN logic: if we do X, then Y will happen)

Makes explicit the “theory of change” in local context

How Can You Use a Logic Model? For Project Planning

Understanding & specifying project elements Involving stakeholders and partners to reach agreement

For Project Management Monitoring & improving project implementation Communicating & building consensus

For Project Evaluation Showing assumptions in line of reasoning Suggesting measures needed for evaluation

For Communications Writing proposals for further funding → sustainability

Resources Activities Outputs Outcomes Impact

Certainresources are needed to operate your program

If you haveaccess tothese resources, then you can use them to accomplishyour plannedactivities

If youaccomplishyour plannedactivities, then you will deliverthe amount of product and/orservice that you intended

If youaccomplishyour plannedactivities to the extent youintended, then your participantswill benefit incertain ways

If thesebenefits toparticipants are achieved, then certain changes in organizations,communities,or systemsmight beexpected tooccur

Planned Work Intended Results

Resources

Activities Outputs Outcomes Impact

Resources are whatever the agency needs to deliver the program

Usually are nouns

Also called “inputs”

Examples: Staff

Families/Consumers

Partners

Grant funding

In-kind contributions

ResourcesActivities

Outputs Outcomes Impact

Activities are actions an organization takes to conduct its program or project

Usually expressed as verbs

How resources will be used

Examples: Conduct trainings

Develop statewide provider inventory

Conduct learning collaboratives

Engage consumers in Advisory Board

Develop website

Disseminate resources to providers

Disseminate resources to families/consumers

Resources ActivitiesOutputs

Outcomes Impact

Outputs are direct results of activities undertaken

Quantify

Think in terms of target audience & how many

Activities Outputs

Conduct trainings 2 webinars annually reaching 50 health providers

Engage consumers in Advisory Board

2 or more consumers attend each Advisory Board meeting

Develop website # of resources downloaded; 1st page of Google search

Conduct learning collaboratives 90% of invited participants attend each of 4 learning collaboratives

Resources Activities Outputs Outcomes Impact

Outcomes are how the target audience changes or benefits after participating in program’s activities. Short term: Immediate actions desired among participantsMeasures are usually expressed quantitatively

Increase the number of children, youth and families who report that they have a medical home

Increase the number of pediatric practices engaging in early and continuous screening opportunities to identify children with special health care needs.

Increase the number of families that indicate that they are included as partners at all levels of decision making in their child’s health care

Resources Activities Outputs Outcomes

Impact

Impact is organizational, community, or system changes expected over the long-term

Derived from MCHB’s six core outcomes

Tailor to what your program/project aims to achieve over life of grant

Family/professional partnership at all levels of decision-making; Access to coordinated ongoing comprehensive care within a med. home; Access to adequate financing and private and/or public insurance to pay for needed services; Early and continuous screening for special health care needs; Organization of community services for easy use; Youth transition to adult health care, work and independence.

Technical Review Key Findings

Inputs Legislation/policy

Point for leveraging/accelerating implementation

Diversity of inputs/resources Comprehensive? Strong?

Think about sustainability

4 legged stool: Medicaid, Title V, family and providers (e.g, AAP)

Technical Review Key Findings

Activities Macro/overall project Micro/specific project component

Key activities well defined

Outputs Outputs vs. outcomes

Review activities Think about intended results

Technical Review Key Findings

Outcomes SMART language

How will you know when you get there?

Think audience Who will benefit? Who will be interested?

Impact Six core performance outcomes

MOVING FROM LOGIC MODEL TO EVALUATION

Purpose of EvaluationFormative - “to improve”

Provides information to help team improve the program

Information/data must be provided and shared quickly and on a routine basis

Drawn from activities, outputs, and short-term outcomes column of logic model

Aka: process evaluation

Summative – “to prove”

Generates information that demonstrates the results of your program

Information/data collected throughout project but purpose is to prove results

Drawn from intermediate/ long-term outcomes and impact columns of logic model

Aka: outcome or impact evaluation

Resources Activities Outputs Outcomes Impact

Formative

Process

Evaluation

Summative

Outcomes

Evaluation

What Level of Evaluation

Bronze - Process evaluation (focused on activities and outputs columns of logic model)

Silver - Process evaluation and results of short-term outcomes

Gold - Process evaluation and summative evaluation

Evaluation Planning Steps Complete logic model with team Assess audiences for evaluation and

questions they would like addressed Develop measures (using SMART

language) Identify data needs to fulfill measures Identify who will collect data and

timeframe for doing so

Evaluation question

Anticipated Result

Measures Data Collection Method/ Source

Who will Collect the data?

Time frame

       

Sample Evaluation Planning Matrix

Evaluation question

Anticipated Result

Measure(s) Data Collection Method/ Source

       

COMPLETE LOGIC MODEL WITH TEAM

Step One:

ASSESS AUDIENCES FOR EVALUATION AND QUESTIONS THEY WOULD LIKE ADDRESSED

Step 2:

Who are audiences for evaluation?

Audience Typical Questions Evaluation UseProgram Management & Staff

Are we reaching our target population?Are our participants satisfied with our program?Is the program being run efficiently?How can we improve our program?

Programming decisions, day-to-day operations

Participants Did the program help me and people like me?What would improve the program next time?

Decisions about continuing participation

Community Members

Is the program suited to our community needs?What is the program really accomplishing?

Decisions about participation and support

Public Officials Who is the program serving?What difference has the program made?Is the program reaching its target population?Is the program worth the cost?

Decisions about commitment and support. Knowledge about the utility and feasibility of the program approach

Funders Is what was promised being achieved?Is the program working?Is the program worth the cost?

Accountability and improvement of future grantmaking efforts

From: Kellogg Fdtn

Exercise #1: Think about the MCHB core outcomes that your project

is working toward Brainstorm various audiences who would be interested

in the evaluation List questions that 3 different target audiences would

want to know about your project (three audiences must be Medicaid and state legislature funder and HRSA or other current funder!)

Identify column(s) of logic model where answers to questions are addressed

Identify whether formative or summative questions

DEVELOP MEASURES (USING SMART LANGUAGE)

Step 3:

SMART

S = SPECIFIC

M = MEASURABLE

A = ATTAINABLE/ACHIEVABLE

R = RELEVANT

T = TIMEBOUND

Evaluation question Anticipated Result

Measure(s) Data Collection Method/ Source

 Are we doing what we said we would be doing? (Activities)

 300 to 350 annual comprehensive diagnostic evaluations completed

75% of diagnostic reports completed within 14-day goal

 # of comprehensive diagnostic evaluations completed

# of diagnostic reports completed within 14 days/total # of diagnostic reports

 Tracking and tally form of diagnostic evaluations and date of completion

Tracking and tally form of diagnostic reports and date of completion

Are we reaching our target audience?

5 trainings for child development clinic staff; 5 to 10 staff attend each

# of trainings for child development clinic staff conducted

# of staff per training

Sign in sheet at trainings, to include position title of participants

Evaluation question

Anticipated Result

Measure(s) Data Collection Method/ Source

How did the target audience change as a result of our program?

 90% of trained case managers perceive that they have the tools to competently engage youth and families to develop transition plans

# of pediatric or family practices providing portable medical records increases each year of grant

% of case managers “agreeing” or “strongly agreeing” that their skill set has improved after attending training

# of pediatric or family practices providing portable medical records

Post training survey of participants perceptions of their improved knowledge and/or skills as a result of the training

Annual survey of practices targeted through the grant

Evaluation question

Anticipated Result

Measure(s) Data Collection Method/ Source

How did the system change as a result of our program?

Pediatric and family practices interviewed report that 75% of youth (> 14) have a documented transition plan

Change in % of practices reporting that 75% of YSHCN > 14 have a documented transition plan

Practices targeted by the grant have a higher percentage of YSHCN with documented transition plans compared to statewide average…

…compared to random selection of non-targeted practices

Baseline survey of pediatric and family practices compared to 3rd year survey of same practices

Post grant survey of targeted practices compared to secondary data source

Post grant survey of targeted practices compared to random sample of non-targeted practices

Rhode Island Example Pediatric Practice Enhancement

Project (PPEP) Placement of Parent Consultants in select practices Claims data analysis in collaboration with State’s

managed care organization for CYSHCN Pre/post PPEP participants Comparison of CYSHCN in PPEP practices

compared to CYSHCN in non PPEP practices

Rhode Island Findings

# of health care (HC) encounters 21% higher for PPEP compared to non-PPEP

Pre/post of PPEP group revealed fewer HC encounters in period following PPEP participation

Inpatient utilization was 34% lower for PPEP participants compared to non-PPEP

Annual HC costs were 27% lower for PPEP participants compared to non-PPEP

Exercise #2

Discuss audience for each goal/outcome Discuss how you know if you succeeded Reword in SMART language For each SMARTly-worded objective,

identify measures and data sources

Planning for Evaluation

Bring in partnersCosts of evaluation vs. scope of

evaluationOutside vs. inside evaluator

Resources

http://wwwlwkkf.org

W.K. Kellogg Foundation Evaluation Handbook Robert Wood Johnson. 2009. A Practical

Guide for Engaging Stakeholders in Developing Evaluation Questions.

Technical Assistance through National Centers and JSI

Thank you.