Program Evaluation, Research Design, and the Logic Model

Research Methods for Public AdministratorsDr. Gail Johnson

Adapted, with additional material, by

Dr. Mario Rivera for PADM 552,

Designing Applied Research

2

Multiple Methods—Mixed Methods Combination and Integration of Quantitative and Qualitative

Methods Neither is inherently better Realities of the situation rule Each work well in some situations, less well in others Arguably, all research is qualitative at its inception (choosing and

defining the research question) and at its end (interpreting, evaluating findings); most program evaluations and other applied research involve multi-method triangulation (triangulation of both method and data)

Quantitative and qualitative data collection often used together—mixed-methods approaches Available data with surveys Surveys with observations Pre-post data analysis (for instance, using t-tests) Surveys with focus groups

3

The Hypothesis DilemmaHypotheses are designed to express relationships between or among variables. They are testable propositions that formally state theoretical expectations about these relationships among. If this is the nature of your topic or question, a hypothesis can add to your research. However, if your topic is more descriptive or explorative and analytical, generating a hypothesis may not be appropriate. A hypothesis may not be appropriate if: You have not identified and articulated a particular theoretical construct You do not have a set of defined variables. Your question turns on phenomenological description Your question entails an analytical review of the problem posed Your question involves in whole or part the study of a cultural group You will both engage in and research the program in question

What to Evaluate? Projects: a single intervention in one location or a single

project implemented in several locations. Programs: an intervention comprised of various

activities or projects which are intended to contribute to a common goal.

Organizations: multiple intervention programs delivered by an organization. While seldom done, it is possible and sometimes desirable to evaluate an entire organization, or a complex of programs essentially defined by and defining an entire organization—e.g., the curriculum of a graduate program, its organizational core, without necessarily looking at all operational elements of the organization.

When to Evaluate? Before program starts:

To improve design During the implementation:

To improve implementation Identify barriers to be removed Lessons learned about implementation To assess a critical element of a program under

review, for instance a risk-management training component of an equal opportunity office in a public agency.

When to Evaluate Mid-term evaluation

Relevance, effectiveness, efficiency Lessons learned: management tool

Impact evaluation (versus outcome evaluation) Either at the end of the project or a few years after the program has

been operating: assessing a mature program Can also look at: effectiveness, efficiency, early signs of impact and

sustainability Lessons learned for future projects or programs Distinction between short- and medium-term outcomes or results

versus longer-term impacts. An impact is the net, long-term effect or complex of effects ensuing from a program. Gross impacts are affected by other causes, programs, secular changes (e.g., in attitudes toward smoking). If one can gauge gross impacts (the longer after the program the more the exogenous influences), can one isolate and evaluate net program impacts?

Why Is Evaluation Useful?

Feedback Accountability Learning Improvement Results Testing underlying assumptions or theory (change

model or theory of change, action model or action theory)

Funding decisions—publicizing and disseminating program results

Evaluation QuestionsCompliance/ Accountability Questions

Did the promised activities actually take place as they were planned? Implementation fidelity

“How” Questions What was the sequence or processes that led to successful (or unsuccessful) outcomes

Impact Questions Did the program achieve the desired results? What of positive but unplanned ones? Intended and unintended consequences

Types of Evaluations

Auditing: accounting for moneyIs the money being spent according to plan?Efficiency and effectiveness.

Monitoring: measuring implementation and resultsIs the intervention producing the intended results?

Process: measuring operations and service deliveryAre there problems in service delivery?

Types of Program Evaluations

Feasibility evaluations Before the program begins Intended to improve program design

Evaluability assessments Assesses potential usefulness of the evaluation Used to test out different strategies for conducting

an evaluation What is doable given the situation?

Evaluability Assessment

Helps to define the actual objectives, implementation and management of a program. The actual objectives may differ from those initially

planned. Determines the coherence of the program: are

goals, activities, program infrastructure linked?

Evaluability Assessment

Key steps in the process: Interview key program staff to actual program mission,

goals, objectives and activities. Site visits to observe and get a sense of what is going on.

May include interviews with key stakeholders. Observe program delivery.

Evaluability Assessment

Reach agreement as to: Whether to conduct the evaluation. Scope and objectives of the evaluation.

The decision could be to not conduct the evaluation.

Evaluability Assessment: Challenges

Key components of the program may not be well defined: Lack of agreement on program objectives. Lack clear, measurable indicators of performance

and/or impact. Target group may not be clearly defined. The delivery system is poorly articulated.

Types of Program Evaluations

Formative evaluations During implementation Feedback about operations and processes Used to make mid-course corrections

Definition: Performance Monitoring

Performance monitoring: the continuous process of collecting and analyzing data

to compare how well a project, program or policy is being implemented against expected results.

Performance measurement tells you the what of what has occurred, while program evaluation explores the why. Program evaluation requires performance data but brings it to interpretive effort eventuating in judgments of value—evaluative decisions, or evaluation.

Traditional vs. newer forms of assessment: Traditional focus on inputs, activities and outputs. Contemporary emphasis on if-then causal questions,

incorporating change and action modeling.

Types of Evaluation:Monitoring

Ongoing review: On-time On-budget On-target

Linked with ongoing management Measured against established baselines Indicators of progress toward targets

Types of Program Evaluations

Summative Evaluations At the end of the program or after the program has been running

long enough to achieve its goals (with “mature” programs) Identify lessons learned Other issues: unintended outcomes, program sustainability,

program efficiency, costs and benefits Sometimes called impact evaluations and ex-post evaluations

Program Evaluation

Summative Evaluation Question: Do Public Programs Work?

Implied cause-effect relationship Did the program cause a desired outcome?

Performance-based: Focus on outcomes, results, impacts, goal achievement.

Differences

Formative Evaluations Project Monitoring Best Suited to Early Years of Implementation Key Question:

Are we doing things right? Have we hired the right people with the right skills? Have we marketed the program effectively? Have we met our strategic objectives? Have we spent our money according to our plan?

Differences

Summative Evaluations Measuring Results or Impacts A longer time before results or impacts are visible Key Question:

Are we doing the right thing? This gets back to the theory or underlying assumptions of the

program: We can do an excellent job at training people but if the problem

derives from larger structural economic issues, a training program, no matter how well implemented, may show little result.

Participatory Evaluation

Responsibility for planning, implementing, evaluating and reporting is shared with all stakeholders.

A partnership based on dialogue and negotiation.

Participatory Traditional

Participant focus and ownership

Focus on learning Flexible design Rapid appraisal methods Outsiders are facilitators

Donor focus and ownership

Focus on accountability and judgment

Predetermined design Formal methods Outsiders are evaluators

Participatory Evaluation

Participants: Engage in group discussions Conduct interviews Conduct field workshops Analyze data and develop conclusions Write the report Disseminate the evaluation results

Participatory Evaluation

How is it done? No single right way Commitment to the principles of participation and inclusion

Core assumption: Those closest to the situation have valuable and necessary information

Develop strategies to develop trust and honest communication Information sharing and decision-making Create even tables

Participatory Evaluation

Benefits: Increased credibility of results Results are more likely to be used Increased buy-in, less resistance Increased sustainability A tool for empowering the affected populations More flexibility in approaches

Participatory Evaluation

Challenges: Time consuming Clarifying roles, responsibilities and process Skilled facilitation Just-in time training No predetermined evaluation plan May be seen as less objective

Participatory Evaluation

Is it the right approach for you? Is there a need for:

An independent outside judgment? Technical information?

Will stakeholders want to participate? Is there sufficient agreement among the stakeholders

so they can work together?

Rapid Assessments

Described as:

“fairly-quick and fairly-clean” as opposed to “quick and dirty” or “long and clean” studies.

Rapid Assessments Uses

Diagnostic evaluations: processes, problems and causes of problems.

Trouble-shooting: Why isn’t a program working as expected? What can be done to improve the program?

Rapid AssessmentsData Collection

Observe: Patterns, land use, behavior Converse: Talk with people, listen to

their concerns and views; interviews, meetings.

Record: Write everything down. Locate: Existing data: reports,

records, maps, prior studies.

Principles of Rapid Assessments

Process: Don’t rush when gathering information Probe and explore Listen rather than speak Be unimposing, open, non-judgmental Seek out the people most likely to be overlooked and

find out what concerns them

Rapid Assessments:No Fixed Definition

Intended to do evaluations quickly while obtaining reasonably accurate and useful information

Uses a systematic strategy to obtain just essential information

Focus is on practical issues

Sources: FAO.Org/docrep/

Principles of Rapid Assessments

Optimize trade-offs: quantity, relevance, accuracy and timeliness. Planned but ability to pursue serendipity.

Triangulation: use more than one technique and source of information.

Face-to-face, on site learning. Learn with a general blueprint that is adapted as more

information is obtained: use multiple methods, improvisation, cross-checking.

Source: FAO.Org/docrep/.

Rapid Assessments

Is it the right approach for you? Important but no major decisions based on the study. Not controversial so face validity is sufficient. Limited time and resources available. Information is already available or requires in-depth

understanding of process.

Social Assessment

Social assessment is a process that brings relevant social information into the decision-making process for program design, implementation, monitoring and evaluation.

It assists in forming key outcome measures to be used in evaluation.

Social Assessment

Tools and Approaches include: Stakeholder analysis Gender analysis Participatory Rapid Appraisal Observation, interviews, focus groups Mapping, Analysis of Tasks, Wealth Ranking Workshops Objective-oriented Project Planning

Working with Logic Models

Visualize a program in context Systems approach, within an environment

Identify the relationships between various components

Identify cause and effect Identify key assumptions

Models: Cause and Effect:

Did the program cause something to happen?

Education Employment

Reduced Poverty

Improved Quality of Life

Increased Income

Job

Training

Unemployed

Hierarchy of Objectives

Sequencing

Logic Models

The focus is on results or impacts rather than inputs and activities We are not training people just for the sake of training people We believe if we train the chronically unemployed, then there

quality of life will be improved and poverty will decrease. Our goal is to reduce poverty

Also called Program Outcome Model or Measuring for Results

Remember, models are not reality; avoid reification. What makes a good model?

Elements of the Logic Model

Inputs: what resources are used University inputs: budget, number of faculty,

number of staff, number of buildings, number of classrooms

Activities: what the program does University activities: teaching, research, and

service

Elements of the Logic Model

Outputs: the services or products produced University outputs: number of students that graduate,

number of articles and books published by faculty Outcomes: what happened: immediate results

Graduates are sought after, get good jobs, active alumni who donate big bucks

Faculty well-known, obtain big grants, enhance rating of university

Elements of the Logic Model

Impacts: the “so what.” Larger, long term results, usually tied to program goals. A more informed and engaged citizenry,

preserves democratic institutions, future leaders. Faculty research contributes to knowledge.

Logic Model

Logical Connections: Inputs to do activities Activities lead to outputs Outputs lead to one or more outcomes Outcomes lead to impacts

Logic Model: Training Program

Inputs Activities Outputs

Resources

•money

•staff

•Supplies

•mentors

•Training Programs

•Dress for success coaching

•Interview coaching

•Resume assistance

Products

•Number of graduates per training session

•% graduate rate

Outcomes

Benefits, changes

•Increased skills

•% Obtain jobs

•% Obtain high paying, quality jobs

•Increased self-esteem

Impacts

Goals

•Increased income

•Self-sufficiency

•Family stability

•Reduction in poverty

Mario Rivera 47

Complex effects chain in partnered programs—network logic models

Partners 1, 2, 3, etc.

Shared Common Outcomes

Attribution difficulties; transparency & accountability challenges

Mario Rivera 48

Participatory evaluation

Participatory evaluation is complex, requiring a correspondingly complex evaluative approach that can adequately deal with complex causality and interaction

The case study is one approach to evaluation that can capture such complexity, through “thick description.”

Mario Rivera 49

Logic Models The focus is on results or impacts rather than inputs and

activities, although all of these are specified, along with indicators and measures. For example, you are in effect saying that We are not training people just for the sake of training

people. We believe if we train the chronically unemployed, then they

might gain meaningful and sustainable employment. Their quality of life will be improved thereby, and with enough such results from this and other efforts poverty will decrease.

Our strategic goal is to help improve that quality of life and reduce poverty—these are the anticipated or hoped-for program impacts.

Also called the Program Outcome Model, Measuring for Results, etc.

Mario Rivera 50

Logic Models A logic model is a systematic and visual way to present

and share your understanding of the relationships among the resources you have to operate your program, the activities you plan to undertake, and the changes or results you hope to achieve

Provides stakeholders with road map describing the sequence of related events connecting the need for the planned program with the program’s desired results

A program design and planning tool A program implementation tool as the core of a focused

management plan Program evaluation and strategic reporting: presents

program information and progress toward goals, telling the story of the program’s commitments and successes.

Mario Rivera 51

Schematic Logic Model

Inputs

Activities

Outputs

Outcomes

Impacts

Mario Rivera 52

Logic Models and Planning Well established, easy-to-use tool for planning programs, eliciting

and using stakeholder input. Provide a common language among managers, program or project

principals, stakeholders including funders, and impacted communities.

Graphic way to articulate—make explicit—and communicate program theory to internal and external audiences.

Provides planners with a road map – asking them to determine where they want to end up (goals) and then chart their course of action. Logic modeling may help program managers in program planning as such, by bringing them to a fuller articulation of program goals and the connection among program aims, activities, and desired outcomes or impacts.

Logic Model example—Preventive Health Ed. for an

Immigrant Community

POPULATION

Characteristics; needs

INPUTS

Resources

ACTIVITIES

Strategies, services

OUTPUTS

program participation

OUTCOMES

desired changes in the population; impacts

–Low income, limited English-speaking community–Low use of health care coverage–Low use of preventive health services–Mostly employed in temporary and/or part-time positions–Community without an efficacious concept or custom of preventive health care in the way defined by the program (e.g., mammograms)

–Program and agency staffing, other resources–Funding–Existing curriculum and volunteer health educators–Prevention media–Verbal and written translation and interpreting resources

–Health care coverage review–Education about other available coverage–Prevention education sessions–Preventive health services in non-traditional locations–Focus groups–Regular tracking of health care coverage and preventive service use

–Number of new families signed up for coverage–Number of lapsed coverage renewed–Number attended education session about available resources–Number of contacts in non-traditional settings–Number of focus groups

–Immigrant families will better understand the importance of preventive health services–Participating families will schedule and complete an increased number of well-child visits, cancer screening, prenatal checkups, etc.–Immunization rates will increase among children in the target population–The number of workdays or school days missed due to illness will decrease

Another template: If-then sequencingInputs Assumption

or Needs or Underlying Condition

Activities Immediate Outcomes

Intermediate Outcomes

Long Term Outcomes,

Results,

or Impacts

Cascading Outcome-focused Logic Model

Workforce Development Outcome Model

Organizational-level Systemic Model

Systemic model outputs and outcomes cascade down to next program level, for example, as inputs and resources or program context.

Youth Development Outcome Model

Financial Literacy Training Outcome Model

•Human Capital•Technology

Logic Model for the New Mexico SPF SIGThe Strategic Prevention Framework State Incentive Grant (SPF SIG) is the

federal Substance Abuse and Mental Health Services Administration's (SAMHSA) major demonstration project of their new Strategic Prevention Framework (SPF) and a Center for Substance Abuse Prevention’s (CSAP) flagship initiative. The SIG is a five-year cooperative agreement from CSAP to states. States receive up to $2.35 million per year for five years of which 85% must go to communities and 15% for State administration activities including a state-wide needs assessment and evaluation. The required components to the SPF SIG are as follows):Create a State epidemiological workgroup and state advisory board Have data-driven planning set state/local priorities Have a funding mechanism for targeting communities Address underage drinking in needs assessment, focus on prevention In 2005, New Mexico was in the first cohort of states to receive a SPF SIG grant. In FY 2006, the state began in local communities. New Mexico had five years of funding and an additional 6th year of an unfunded extension. National & State SPF SIG Goals—the overarching national goals are to: prevent onset and reduce progression of substance abuse, including underage drinking; reduce substance-related problems in communities; build prevention capacities and infrastructure at State and community levels

SPF SIG New Mexico Community Logic ModelReducing alcohol-related youth traffic fatalities—change modeling

High rate of alcohol-

related crash mortality

Among 15 to 24 year olds

Low or discount PRICING of alcohol

Easy RETAIL ACCESS to Alcohol for youth

Easy SOCIAL ACCESS to Alcohol

Media Advocacy to Increase Community

Concern about Underage Drinking

Restrictions on alcohol advertising in

youth markets

SOCIAL NORMS accepting and/or encouraging

youth drinking

PROMOTION of alcohol use (advertising, movies,

music, etc)

Low ENFORCEMENT of alcohol laws

Underage

DRINKING AND DRIVING

Social Event Monitoring and

Enforcement

Bans on alcohol price promotions and

happy hours

Young Adult

BINGE DRINKING

Enforce underage retail sales laws

InterveningVariables

Strategies(Examples)

Substance-Related

Consequences

SubstanceUse

Low PERCEIVED RISK of alcohol use

Young Adult

DRINKING AND DRIVING

Underage

BINGE DRINKING

Multiplicity of Contextual and Causative Factors That Influence a Program’s

Community ImpactMulti-System, Cross-sector Collaboration

Economy

Community Impact

Grant Funding

Inter-agency Community Partnership

Project Outcomes

Policy ChangesCitizen Efforts

Other Funding

Time

Creative Commons

This powerpoint is meant to be used and shared with attribution

Please provide feedback If you make changes, please share freely and send

me a copy of changes: Johnsong62@gmail.com

Visit www.creativecommons.org for more information