program logic models - public health ontario · program logic models ... from original...

Program logic models

December 2012

1

Introductions

Presenter:

Areeta Bridgemohan, MPH – [email protected] Health Promotion Field Support Specialist, Evaluation

Assisted by:

Magdalena Wasilewska, MPH Health Promotion Coordinator

2

mailto:[email protected]

Webinar features

3

• Adobe Connect technology

• Chat function to ask questions or if you need help

• Webinar courtesy

• Polls

©iStockphoto.com/Adrian Assalve

Learning objectives

At the end of the session, participants will:

1. Understand and appreciate the purpose and use of program logic models.

2. Be able to identify and describe the components and characteristics of effective program logic models.

3. Know how to access our products and services to support your logic model development.

4

Agenda

• Overview of logic models: definition, purposes and examples

• Theory behind logic models

• Criteria for effective program logic models

• Questions

5

©iStockphoto.com/Creativeye99

Poll # 1: your experience with logic models

What is your experience with logic models?

1. I design and use them all the time

2. We use a logic model to guide and evaluate our work, but I’ve never had experience designing one

3. I know what they are but have never used them in my work

4. What is a logic model and why is it useful?

6

Overview of logic models ©iStockphoto.com/MG

8

A graphic depiction of the relationship between the key elements of a program or initiative

(i.e., goals, objectives, target groups, and activities)

What is a program logic model?

©iStockphoto.com/sorendls

What is a program logic model? (continued)

Shows the relationships or the “If …, then …” logic between:

• What: what does the

program/initiative do?

• Who: who are the recipients of

the program/initiative?

• Why: what outcomes are to be

achieved?

9

©iStockphoto.com/kalawin

10

© ScienceCartoonsPlus.com, 2012. Reproduced with permission.

Simple logic model

Goal: to increase capacity to understand logic models among webinar participants

Strategy: education

Activity: webinar on logic models

Population of interest: public health practitioners

(Short term) outcome objective : increased understanding of purpose and elements of logic

models

11

Parts of a logic model • Goal

• Strategies

• Activities

• Population of Interest

• Process Objectives/Outputs

• Resources/Inputs

• Outcome Objectives: • Short-term • Intermediate • Long-term

• Boxes and arrows

• What does the program do?

• With/for whom?

• Using what?

• What is the expected result for the population of interest?

• Show the link between items

12

Other parts of a logic model

• Risks

• Assumptions

• Indicators

• Summary of situational analysis

13

External factors

14

Sample program logic model

outline

Goal

Population(s) of

interest

Long-term

outcome objectives

Short-term

outcome objectives

Activities

Outputs/process

objectives

Resources

A logic model supports planning by:

• Involving stakeholders in program planning

• Building a common understanding of what a program is all about and how the parts fit together

• Identifying the most important desired outcomes

15

©iStockphoto.com/paci77

A logic model supports planning by:

• Depicting connections between key facets of a

program (how a program's components

contribute to the achievement of intended objectives)

• Noting gaps and inconsistencies within a

program

• Providing an effective communication tool

16

©iStockphoto.com/fabervisum

Logic models support evaluation by:

• Providing a starting point for engaging stakeholders

• Serving as a resource for evaluability assessment

• Identifying appropriate evaluation questions based on the program

• Developing indicators for the objectives

• Demonstrating accountability

17 ©iStockphoto.com/Photomorphic

Simple logic model

Goal: to increase capacity to understand logic models among webinar participants

Strategy: skill building

Activity: webinar on logic models

Population of interest: Public health practitioners

Outcome: increased understanding of purpose and elements of logic models

18

Poll #2: your use of logic models

As it relates to your current work, how could you use logic models?

1. As part of a program planning process

2. To clarify a program currently being implemented

3. As part of an evaluation

4. Other

19 ©iStockphoto.com/AtnoYdur

Critiques of logic models

• Time-consuming

• Linear

• Challenging when program is complex

• No time component

• Danger of misuse and miscommunication

• Inconsistent terminology

Ian David Moss ‘In defense of logic models’1

20

Examples of logic models 21

©iStockphoto.com/Vladimir Yudin

OPHS vaccine preventable diseases

22

School-based resilience program2

Goal(s) To empower all school community members to adapt to and/or bounce back from changes and challenges they face to attain and sustain optimal health and developmental potential. (To increase resiliency in our school communities.)

Audience(s) The target will be priority population school communities (to be identified through a pending assessment).Within the school communities students will be the primary audience. Secondary audience will include: teachers, parents and other school staff, school administration, and school boards.

.

Long-term outcome objectives

Medium-term outcome objectives

- To increase by 10% knowledge among school community members of resiliency related skills such as adaptive skills, social skills and healthy behaviours, by April 2011 (medium-term)

- To increase by 10% the percentage of students (in priority schools) who report confidence or high degrees of confidence in applying resiliency skills such as adaptive skills, social skills and healthy behaviours, by April 2011 (medium-term)

- To have 100% of participating priority schools develop a school health plan that is responsive to needs identified through the Roadshow whereby their plan takes a comprehensive school health approach and incorporates the four pillars of The Foundations for a Healthy School (Quality Instruction and Programs, Social Support and Policy, Physical Environment, and Partnerships), by June 2011 (medium-term)

Short-term (and other) outcome objectives

- To increase by 10% awareness among priority school community members of positive and negative stressors, as well as risk and protective factors for stress, by January 2011 (short-term)

.

Strategies Health Communication Self-help/mutual support Community mobilization Develop personal skills

.

Activities - Website (including podcasts, forum discussions, geographic mapping)

- Life planning tools

- School Community Needs Assessment - Customized Comprehensive School Health Programming Within Each School

- 5 Program Workshops (fuel, play, chill, chat, create) (part of roadshow)

.

24

Process objectives

- To produce a website for members of the school community, by fall 2010

- To produce a set of resiliency resources/tools (both hard copy and online), for students to use to set goals and create life plans, by October 2010

- To produce a set of questions and a documentation process, for school communities to use to gather input from school community members during the program workshops, by September 2010

- To produce a guided planning process and set of planning templates, for school communities to use to create a comprehensive school health plan, by September 2010

- To produce a list of schools who are clearly identified to be the priority populations, for Health Unit staff to focus their planning/programming time with, by August 2010

.

Resources allocated

The website will be a simpler version than originally planned/scoped. Resources for website development are limited plus access to internal technical support resources to aid in development.

$2500 for tool expansion/development + staff time/human resources.

Staff time. Approximately $5000/workshop for development (reduced budget from original vision/concept).

School-based resilience program (continued)

25

Comprehensive tobacco control logic model3

26

Prevention

27

Protection

28

Cessation

29

Smoke-free environments (US): nested logic model4

Tobacco cessation for pregnant women

30

RESOURCES ACTIVITIES OUTPUTS OUTCOMES

Short-term Intermediate Long-term

Objective: To initiate and

carry out community-based

activities and projects directed

towards helping pregnant and

post-partum women quit using

tobacco

Resources:

Health Promoter and/or Public

Health Nurse – public health

department

Pregnets Toolkit for Health

Care Professionals

Health department budget for

program implementation

Hospital staff/health care

providers to provide referrals

Develop home visit

counseling/NRT provision

program for pregnant and

post-partum women and their

partners as part of the Healthy

Babies Healthy Children

program

Conduct BabyTalk drop-in

clinics in outreach areas

Develop a relationship with

local hospitals and health care

providers to provide referrals

to program

Advertise program in local

media (radio/newspaper/tv

ads)

Provide counseling, NRT and

follow-up care to pregnant

and post-partum women and

their partners

Provide consultation and

support to external health care

providers

Conduct program evaluation

through a post-participation

survey

# of partnership with local

hospital/health care providers

# of referrals generated

# of radio ads

# of newspaper ads

# of TV ads

# of clients on program roster

# of clients provided with

counseling

# of clients provided with

NRT

# of clients who attending

Babytalk drop-in clinics

# of surveys completed

- Pregnant and post-partum

women are motivated and

aware of services available to

support them in their efforts to

quit using tobacco

- Increase # of pregnant and

post-partum women who call

for information to seek

assistance

-Increase # of pregnant and

post-partum women who

register for program

-Increase # of referrals

received

- Increase # of pregnant/post-

partum women and their

partners who remain clients of

the program three-months

post-registration

- Pregnant /post-partum

women and their partners

remain tobacco-free

These examples show that logic models may vary in terms of:

• How much they cover (scope) and level of abstraction • Nested logic models for multiple programs/initiatives

• The number of levels included • Short, medium and/or long-term outcomes?

• The naming of component parts

• The direction of information flow

• The visual layout

31

Theory behind logic models ©iStockphoto.com/Thomas Perkins

What is the theory of change approach?5

• The theory (or assumptions) that explain:

• the steps that lead to the long-term goal

• the connections between program activities and outcomes that occur at each step of the way

• Lays out the sequence of outcomes that are expected to occur as the result of an intervention

33

Benefits of the theory of change approach

• Helps to avoid implementing a mistake

• Strengthen ability to claim credit for outcomes predicted by the theory

• Can plan an evaluation strategy to track whether expected outcomes are produced

34

Four types of theory of change assumptions

1. Assertions about the connections among long-term, intermediate, and early outcomes on the map

2. Justifications supporting the links between program activities and the outcomes they are expected to produce

3. Substantiation for the claim that all of the important preconditions for success have been identified

4. Identification of contextual or environmental factors that will support or hinder progress toward the realization of outcomes

35

Example of theory of change approach13

Issue: bone fractures in the elderly

Organizational mandate: physical activity

• Why do bone fractures occur? • Because of thinning bones

• Why do thinning bones occur? • Lack of weight bearing exercise

• Sedentary lifestyle

• Why do some people not engage in exercise? • Lack of knowledge

37

38


Other possible factors: • (Stages of change) self-efficacy • (Theory of reasoned action) Social norms • (Health belief model) Perception of susceptibility • (PRECEDE-PROCEED) predisposing, behavioural,

environmental factors

39


40


Testing the logic in your logic models

• Is the logic sound?

• Are the assumptions flawed or accurate?

• Are explanations of how change happens and expected outcomes consistent with what is known?

• Are contextual conditions (opportunities and barriers) accounted for?

• Plausibility

• Do-ability

• Testability

41

6

Poll #3: your perception of theory of change

What is your perception of the utility of theory of change in your program planning?

1. Very useful

2. Useful

3. Somewhat useful

4. Not useful

5. Unsure

42 ©iStockphoto.com/AtnoYdur

What makes a good logic model?

• Logic!

• Completeness

• Presentation

• Useful

• Leads to evaluation possibilities: evaluation can be considered a test of the logic behind a program.

44

Completeness

• Include all appropriate levels and populations of interest?

• Identify clear and measurable short- and long-term objectives?

• Process objectives listed under an appropriate strategy?

• Identify a wide range of resources, including financial, staff, and in-kind?

45

©iStockphoto.com/tumpikuja

46

Logic

• Is this the most logical set of strategies?

• Are the chosen process objectives likely to result in meeting the short-term objectives?

• Will the short-term objectives lead to the long-term objectives?

• Are the resources sufficient to drive strategies and process objectives?

©iStockphoto.com/fotosipsak

• Passing the ‘reasonable person’ test:

• Would someone understand your program or initiative based on the logic model?

• Would they believe that you would likely achieve the desired outcome?

47

Logic

©iStockphoto.com/drbimages

48

Presentation

• Are there too many boxes on the page?

• Is it easy to follow the arrows and flow of logic?

• Are the levels in an order that is useful for you and your stakeholders?

• Is the model user-friendly and easy to follow?

©iStockphoto.com/matspersson0

Component of Logic Model

Worksheet to Complete

Goals, outcome objectives

Worksheet 3.1: Set Goals, Audiences, Outcome Objectives

Audiences Outcome Objective (Long, Medium, Short-term)

Strategies Worksheet 4.1: Choose Strategies and Brainstorm Activities

Activities Worksheet 4.2: Assess and Choose Activities (Must do at least the final question in this series #6 – Choose activities)

Process objectives Worksheet 4.4: Develop Process Objectives

Resources Worksheet 4.3: Assign Resources and Outcome Objectives (to activities)

50

OHPP logic model worksheets

Logic model resources

1. In defense of logic models1

2. Theory of change website5

3. The community builder’s approach to theory of change: A practical guide to theory development6

4. Logic models workbook7

5. University of Wisconsin-Extension-cooperative extension Local evaluation project8

6. Online Health Program Planner (OHPP)9

7. Kellogg Foundation logic model development guide10

8. Enhancing program performance with logic models: interactive online course11

51

52

Logic model resources

9. CDC Heart and Stroke logic model guide12

10. A three-step approach to teaching logic models13

Health Promotion Capacity Building Services

• Free to those working on Ontario-focused projects.

• Scope varies, depending on need: • training sessions;

• brief, one-time advice;

• consultations;

• review your work or product;

• links to other sources of information and resources.

• Consultation request form

http://www.thcu.ca/consultation/request_form.cfm

53



• Areeta Bridgemohan, Health Promotion Field Support Specialist - Evaluation

• Nimira Lalani, Health Promotion Field Support Specialist – Planning and Policy Development

• Ben Rempel, Health Promotion Field Support Specialist – Alcohol Policy

• Sophie Rosa, Health Promotion Field Support Specialist – Health Communication

Health promotion capacity building services

Public Health Ontario wishes to acknowledge and thank THCU staff and many partners who contributed to an earlier version of this document. THCU (originally known as The Health Communication Unit, started in 1993 at the University of Toronto) moved to Public Health Ontario’s Health Promotion, Chronic Disease and Injury Prevention Department in 2011.

References

1. Moss, David Ian. In defense of logic models [document on the internet]. c2012 [cited 2012 December 18]. TheHuffingtonPost.com, Inc. Available from: http://www.huffingtonpost.com/ian-david-moss/in-defense-of-logic-model_b_1633676.html

2. Sudbury District Health Unit. Can you feel it resiliency program [document on the internet]. [cited 2012 December 18]. Available from: http://www.thcu.ca/ohpp/plan_library.cfm?lib_planID=9&action=show_desc#116630

3. Public Health Ontario, Smoke-Free Ontario – scientific advisory committee. Evidence to guide action: comprehensive tobacco control in Ontario. Toronto, ON: Queen’s Printer for Ontario; 2010.

4. Butterfoss FD. Coalitions and partnerships in community health [homepage on the internet]. c2007 [cited 2012 December 18]. San Francisco, CA: Jossey-Bass; John Wiley & Sons. Available from: www.coalitionswork.com/

57

http://www.huffingtonpost.com/ian-david-moss/in-defense-of-logic-model_b_1633676.html














http://www.thcu.ca/ohpp/plan_library.cfm?lib_planID=9&action=show_desc



References (continued)

5. ActKnowledge Inc. Theory of change [homepage on the internet]. c2012 [cited 2012 August 24]. www.theoryofchange.org

6. Anderson Andrea. The community builder’s approach to theory of change: a practical guide to theory development. The Aspen Institute Roundtable on Community Change.

7. The Health Communication Unit, Centre for Health Promotion, University of Toronto. Logic models workbook. Toronto, ON: University of Toronto; 2001

8. University of Wisconsin-Extension-cooperative extension. Local evaluation project [document on the internet]. c2003 [cited 2007 January 18]. Available from: http://www.uwex.edu/ces/pdande/evaluation/pdf

58


9. Public Health Ontario and the National Collaborating Centre for Methods and Tools [internet]. Online Health Program Planner. [updated 2011; cited 2012 December 18]. Available from: www.thcu.ca/ohpp

10. W.K. Kellogg Foundation. Kellogg Foundation logic model development guide [document on the internet]. [cited 2012 November 2]. Available from: http://www.wkkf.org/knowledge-center/resources/2006/02/WK-Kellogg-Foundation-Logic-Model-Development-Guide.aspx

11. University of Wisconsin. Enhancing program performance with logic models [interactive online course]. [cited 2012 November 2]. Available from: http://www.uwex.edu/ces/lmcourse/

12. Centers for Disease Control and Prevention. State program evaluation guides: developing and using a logic model [webpage on the Internet]. [cited 2012 December 11]. Available from : http://www.cdc.gov/dhdsp/programs/nhdsp_program/evaluation_guides/logic_model.htm

59

http://www.uwex.edu/ces/lmcourse/

http://www.uwex.edu/ces/lmcourse/

http://www.cdc.gov/dhdsp/programs/nhdsp_program/evaluation_guides/logic_model.htm



13. Renger R, Titcomb A. A three-step approach to teaching logic models. American Journal of Evaluation. 2002 [cited 2012 Dec 18]; 23(4):493-503. Available from: http://host-124-203.host.lemoyne.edu/courseinformation/Ridzi/402%20Prog%20Eval%20Policy%20Analysis/Samples%20for%20Portfolio/02Logic%20Models/Teaching%20Logic%20Models%202002.pdf

60


http://host-124-203.host.lemoyne.edu/courseinformation/Ridzi/402 Prog Eval Policy Analysis/Samples for Portfolio/02Logic Models/Teaching Logic Models 2002.pdf







program logic models - public health ontario · program logic models ... from original...

Documents