leveraging evaluation resources in lean financial times: featuring maine’s program with previews...

Leveraging Evaluation Resources in Lean Financial Times:

Featuring Maine’s Program with Previews

From Arkansas

Leveraging Evaluation Resources in Lean Financial Times:

Featuring Maine’s Program with Previews

From ArkansasMichael D. Johnson, Ph.D.Michael D. Johnson, Ph.D.Howard A. Fishbein, Dr.PHHoward A. Fishbein, Dr.PHThe Gallup OrganizationThe Gallup Organization

Gallup’s Commitment to Tobacco Control

• Gallup Est. 1935 – No Tobacco Industry Work Ever !

• Dr. Howard Fishbein, VP Healthcare Research Begins Tobacco Control at Gallup in 1995

• Dr. Michael Johnson, Senior Consultant, Joins Gallup Team coming from California DHHS in 1999

• Dr. Manas Chattopadahay, Senior Statistician also helps lead staff of Data Researchers & Analysts

• Key Staff Brought Onto Projects as Needed

State Acknowledgements

• Maine Bureau of Health- Healthy Maine Partnerships (HMP) includes the Partnership for a Tobacco-Free Maine (PTM) and the Maine Cardiovascular Health Program (MCVHP)- Lead is Ms. Patti Robinson

• Arkansas Department of Health- Tobacco Control Program – Lead is Ms. Lynda Lehing and Dr. David Bourne

• Arkansas Cancer Coalition & Arkansas Cancer Control Program – Lead is Ms. Tina Gill

Objectives for Today

• To describe how evaluation design and implementation can be shared between tobacco control, cardiovascular disease prevention, and cancer control programs.

• To illustrate Gallup’s current efforts in doing this with state examples.

• To send you home thinking about how your program and the populations you serve can benefit by adopting some elements of this approach.

A Collective 2003 Problem…

• State tobacco control programs (TCP) are struggling to survive with limited financial resources.

• Despite every program’s limited monies, the question of “What Works?” is still asked of TCP’s and other public health programs such as cardiovascular disease prevention programs and cancer control programs.

• How do programs that need to keep their administrative reporting, and programs separate from one another, move to a higher level of organization and collaboration?

“What will work best for all and still work best for each program ?”

The Challenge/Solution/Pledge

• To pool state public health resources where appropriate (e.g. administrative), to address and evaluate multiple health problems where appropriate (e.g.scientific), without diluting the emphasis on any particular program (e.g., tobacco control); and to do this in a seamless fashion.

• Size or dollar amount of the programs should not limit these efforts.

For years we have known the scientific basis and recognized the administrative benefits of coordinating disease prevention and control efforts when combating chronic diseases where risk factor behaviors overlap… Multiphasic Screening Programs, 1960’s

Mr. Fit, Stanford 5-Cities, Pawtucket Heart Health, North Karelia,

1970’s

xemplary State Programs – Massachusetts’ “Great Little Decisions” Program, 1986

Primary Prevention Program, Pasteur Institute, 1992

Past Experiences and Successes Were Mixed…

Why?• Impatience with waiting to see meaningful changes in disease

morbidity and mortality.

• Not sure which risk factor to focus on and in what order. Difficult to

keep traditionally categorical programs working in unison and loyal

to an overall centralized focus such as Healthy People 2000 or 2010.

• Program leaders and contractors would often change causing loss of

continuity, historical memory, and causing unnecessary re-starts or

no champion of the program left……AND

A comprehensive integrated evaluation plan - across programs and looking at a range of intermediate and ultimate

outcomes was never available…but now we have Maine and Arkansas doing this.

We Can Preview that in Arkansas in 2003:

• ARKANSAS CANCER CONTROL and TOBACCO CONTROL and CARDIOVASCULAR HEALTH …Programs, Interventions, and Outcomes overlap. Example:

Goal: Reduce the proportion of Arkansans who use tobacco products.

• Objective: Reduce the proportion of young people who have ever used tobacco products to 32% by 2005. From 1999 to 2001: 39.6% to 33% reported having ever used tobacco products.

• Goal: Encourage children and adults to adopt risk reduction habits.

• One Objective is to get to 5+ servings to be reported among children to reach 25% of the total population by 2005. From 1999 to 2001: 18.8% to 20% of school age children eat 5+ servings of fruits/vegetables per day.We are building an evaluation plan that will apply to several programs.

What is different in Maine in 2003 to foster a successful

evaluation? • Longevity:The Gallup evaluation team has been working

with BOH since 1998. • An integrated Logic Model of the PTM, MCVHP, and the

HMP programs has been developed.• A detailed Evaluation Plan looking at specific indicators

of success for each of the three initiatives and the overall Healthy Maine Partnerships has been developed.

• Gallup has senior staff with experience in this area that are committed to this type of public health approach.

• Gallup has a local Senior Consultant in Maine, Ms. Patricia Hart, very active in helping lead the evaluation efforts in Maine.

Partnership for a Tobacco-Free Maine (PTM)

• 31 Local HMP’s• Statewide Tobacco Treatment Services

(Helpline, Medication Vouchers, Provider Training)

• Statewide Media Campaign• Enforcement Activities• Youth Advocacy Network• Tobacco-Free Athletes Program• Programs Targeting Disparate Populations• Evaluation and Surveillance Activities

Maine Cardiovascular Health Program (MCVHP)

• Public Policy Initiatives• 31 Local HMP’s• Nutrition Initiative (Action Kits)• Physical Activity Initiative (Maine Walks)• Statewide Media Campaign• Secondary Prevention (Early Detection)• Worksite Programs• Youth and School-based Programs (YAP and CATCH)• Priority Populations (Low SES, Diabetes, Women,

and Native Americans)• Evaluation and Surveillance

Overview of Comprehensive Evaluation

• Provides first comprehensive evaluation, previous work was evaluative research but was not funded to look at all program components

• Evaluates the progress, outcomes and interrelationships of the Healthy Maine Partnerships, including the a) Maine Cardiovascular Health Program; b) the Partnership For A Tobacco-Free Maine; 3) the 31 local HMP intervention sites

• Leverages evaluation resources across three programs, recognizing program and outcome interdependencies

PTM, MCVHP, and HMP Logic Model

Inputs

Activities Outputs Initial Intermediate Long-Term Goal

OUTCOMES

·   Statewide Health Department Infrastructure·   Local HMP Programs·   Maine Coalition on Smoking and Health·   Maine Nutrition Network·   Maine Cardiovascular Health Council·   Maine Department of Education·   Maine Department of Transportation·   American Heart Association (NE)·   Statewide Health Voluntary Agencies·   Advisory Council·   Maine Hospital Association·   Maine Center for Public Health·   Maine Association of Health, Physical Education, Recreational, and Dance·   Governor’s Council on Physical Fitness and Sports·   ME Cares· 

Statewide Media Campaign/Counter-

marketing/Alternative Sponsorship

Local HMP Programs, Nutrition Initiative, Physical Activity

Initiative, Secondary Prevention Programs, Priority Population

Programs (Low SES/Rural, Women Diabetes, and Native

Americans), Treatment Initiatives (Helpline, Provider Training,

Medication), Tobacco Free Athletes, School-based Prevention and

Treatment

Exposure to nutrition, physical activity, and anti-

tobacco health messages

Increase access to and use of

services and programs

Creation of nutrition, physical

activity and no smoking policies, regulations, and

bans

Changes in awareness,

knowledge, attitudes and behaviors

Increase physical activity, healthy nutritional habits among youth and

adults

Compliance with, support for, and

enforcement of bans, regulations and

policies

Reduce tobacco initiation

among youth

Increase tobacco cessation among youth and adults

Public Policy Initiative, Worksite Programs,

Youth and School-based Programs (CATCH), Enforcement, Youth

Advocacy (MY TURN), YAP

Increased regular

physical activity and

healthy nutritional

habits

Decreased tobacco use prevalence

and consumption

Increased access to physical activity and

healthy nutrition

Reduced exposure to ETS

Increased environments that support physical activity, healthy

nutritional practices, and no

smoking

Reduced CHD-related morbidity

and mortality

Reduced CHD-related disparity

The Evaluation Plan is Centered on a Goal-Based Approach

• Mapping HMP outcomes as they relate to state health goals described in Healthy Maine 2010

• Following a detailed Evaluation Plan to identify key goals, objectives, strategies and measures of success

• Using a range of tools to inform results including program data, focus groups, key-informant interviews, surveys and web-based monitoring system

• Including input from a wide-range of stakeholders in the plan development

Evaluation Plan

• Lists Goals, Objectives, Indicators (short and long-term), Data Sources, and Frequency and Utility of Data

• Statewide HMP Initiative• Local HMP Initiative• PTM• MCVHP

PTM Example

• Goal 1: Prevent Initiation of tobacco use by youth

• Objective 2: Reduce the rate of illegal sales of tobacco products to minors…

• Indicator 4: The rate of tobacco sales to minors

• Data Source (s): Maine Dept. of the Attorney General

PTM Example

• Goal 2: To motivate and assist smokers to quit

• Objective 1: Reduce adult smoking (age 18+) prevalence to 19% by 2010

• Indicator 1: The prevalence of cigarette smoking in adults age 18 +

• Data Source (s): Maine BRFSS; Maine ATS

MCVHP Example

• Goal 1: To build an infrastructure at the state and local level to improve CVD health…

• Objective 1: Reduce coronary heart disease deaths to no more than 166 per 100,000 by 2010

• Indicator 1: The number of coronary heart disease deaths in Maine

• Data Source: Maine BOH, Office of Vital Statistics

Advantages of Integration in Evaluation

• Streamline data collection by combining surveys and other data collection instruments

• Demonstrate how the short-term outputs of several initiatives can lead to long-term positive health outcomes (e.g. tobacco use reduction is a key outcome for improved cardiovascular health)

• Address the program relationships in the evaluation and analysis (local intervention site feedback to state-designed initiatives)

Early Findings Show Positive Results of Integration

• In the first year, the local HMPs had more than 230 local policy/environmental changes with an average of seven per site

• The 31 local HMPs use the programming and strategies provided by the state-level programs like the MCVHP to enlist coalition members and create policy and environmental changes

• Benefits of integration include shortened program development time, sharing lessons learned and “doing what others are doing”

Environmental Indicators: A Baseline 2003

Environmental Indicators: A Baseline 2003

• Nine percent of the sampled towns in Maine have written policies requiring tobacco smoke free entrances in municipal buildings.

• Eleven percent of the sampled towns in Maine have a committee delegated to address walking and biking issues

Environmental Indicators: A Baseline 2003

• Fifty-three percent of the sampled towns in Maine had an indoor physical activity facility available at a school or college to community members for low or no cost

• Only 10% of the sampled towns in Maine have a community garden program where residents can grow fruits and vegetables. But 70% of towns in Maine have a nearby direct-buy farm such as a farmer’s market

Outcome Indicators

1. What’s your number(s) ?

Prevalence of Smoking

Smoking Prevalence Data

0

10

20

30

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

Nationwide Maine

Copyright © 2003 The Gallup Organization, Princeton, NJ. All rights reserved.THE GALLUP ORGANIZATION

Consumption of Tobacco

Consumption of Tobacco

The faces of The Tobacco Industry

light up every time you do.

T H E T O B A C C O I N D U S T R Y. T H E Y P R O F I T. Y O U L O S E.

60.0

80.0

100.0

120.0

140.0

160.0

180.0

1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001

ME MaineTobacco Tax Increase

Per Capita Consumption Trends In Maine

2. Who Cares ?

Create smoke-free environments

Policies & Protection

90.0%

80%

100%

Maine

Secondhand Smoke Exposure 2001

Adults agreeing or strongly agreeing that

people should be protected from

Secondhand Smoke

63.0

0

20

40

60

80

2002

Protection from Secondhand Smoke in the Home - MaineProtection from Secondhand Smoke in the Home - MaineP

erc

en

tag

e o

f p

eop

le p

rote

cte

d

Reduce or Eliminate Youth Access to Tobacco

Reduce or Eliminate Youth Access to Tobacco

Fewer Stores are Selling Tobacco to Kids - MaineFewer Stores are Selling Tobacco to Kids - Maine

17.0

7.0

0

2

4

6

8

10

12

14

16

18

1997 2001

Pe

rce

nt

Pre

va

len

ce

Quitting Smoking

59.0%

0%

20%

40%

60%

80%

100%

Maine

Attempted to Quit in the last year 2001

Adult smokers who have tried to quit in the

last 12 months

3. What Works ?

PTM Tobacco Treatment Initiative

Of approximately 4,000 tobacco users assisted by the HelpLine, over 1,500 quit temporarily and 730 quit long-term.

The 730 individuals who quit long-term will produce a savings of of approximately $2.47 million in productivity and medical expenditures.§

Six-month Quit Rates: Counseled by HelpLine vs. Those Who Quit On Their Own

0%

10%

20%

30%

HelpLine CallersCounseled

No Assistance

Percent quit

95%

Confidence Inte

rval

How Effective is the Maine Tobacco HelpLine?

22% were not smoking 6 months after HelpLine counseling.

Findings Point out the Challenges and Benefits for

Integrating Program and Evaluation Needs

• Advance planning and communicating plans are important-- for buy-in, contributions, and smooth hand-offs among program staff.

• As work units and skill sets become specialized around a public health issue, communication can become internally focused and can benefit by building bridges with other staff and programs.

• Field staff and central office staff will have clear expectations, clear roles and clear lines of authority, and communicate often.

Challenges and Benefits for Integrating Program and

Evaluation Needs• Coordination among related programs brings a

united front in times of glory and downsizing.

• A proper Evaluation design will allow for individual program growth while contributing to an even larger overall public health perspective

• You will bring real meaning to the term “synergistic.” Program organization through evaluation can be a lynchpin to fostering great results….so it is worth all the effort !

Thanks For The Opportunity To Share Our Experiences of Being in

the Field With Programs in Arkansas and in Maine

With You.

Have a Great Rest of the Conference !

To Contact Us at Gallup:

• Dr. Michael Johnson in California at (530) 268-8218 Direct Line

• Michael_Johnson@gallup.com

• Dr. Howard Fishbein in Washington,DC at (202) 715-3178 Direct Line

• Howard_Fishbein@gallup.com