leveraging evaluation resources in lean financial times: featuring maine’s program with previews...
TRANSCRIPT
Leveraging Evaluation Resources in Lean Financial Times:
Featuring Maine’s Program with Previews
From Arkansas
Leveraging Evaluation Resources in Lean Financial Times:
Featuring Maine’s Program with Previews
From ArkansasMichael D. Johnson, Ph.D.Michael D. Johnson, Ph.D.Howard A. Fishbein, Dr.PHHoward A. Fishbein, Dr.PHThe Gallup OrganizationThe Gallup Organization
Gallup’s Commitment to Tobacco Control
• Gallup Est. 1935 – No Tobacco Industry Work Ever !
• Dr. Howard Fishbein, VP Healthcare Research Begins Tobacco Control at Gallup in 1995
• Dr. Michael Johnson, Senior Consultant, Joins Gallup Team coming from California DHHS in 1999
• Dr. Manas Chattopadahay, Senior Statistician also helps lead staff of Data Researchers & Analysts
• Key Staff Brought Onto Projects as Needed
State Acknowledgements
• Maine Bureau of Health- Healthy Maine Partnerships (HMP) includes the Partnership for a Tobacco-Free Maine (PTM) and the Maine Cardiovascular Health Program (MCVHP)- Lead is Ms. Patti Robinson
• Arkansas Department of Health- Tobacco Control Program – Lead is Ms. Lynda Lehing and Dr. David Bourne
• Arkansas Cancer Coalition & Arkansas Cancer Control Program – Lead is Ms. Tina Gill
Objectives for Today
• To describe how evaluation design and implementation can be shared between tobacco control, cardiovascular disease prevention, and cancer control programs.
• To illustrate Gallup’s current efforts in doing this with state examples.
• To send you home thinking about how your program and the populations you serve can benefit by adopting some elements of this approach.
A Collective 2003 Problem…
• State tobacco control programs (TCP) are struggling to survive with limited financial resources.
• Despite every program’s limited monies, the question of “What Works?” is still asked of TCP’s and other public health programs such as cardiovascular disease prevention programs and cancer control programs.
• How do programs that need to keep their administrative reporting, and programs separate from one another, move to a higher level of organization and collaboration?
“What will work best for all and still work best for each program ?”
The Challenge/Solution/Pledge
• To pool state public health resources where appropriate (e.g. administrative), to address and evaluate multiple health problems where appropriate (e.g.scientific), without diluting the emphasis on any particular program (e.g., tobacco control); and to do this in a seamless fashion.
• Size or dollar amount of the programs should not limit these efforts.
For years we have known the scientific basis and recognized the administrative benefits of coordinating disease prevention and control efforts when combating chronic diseases where risk factor behaviors overlap… Multiphasic Screening Programs, 1960’s
Mr. Fit, Stanford 5-Cities, Pawtucket Heart Health, North Karelia,
1970’s
xemplary State Programs – Massachusetts’ “Great Little Decisions” Program, 1986
Primary Prevention Program, Pasteur Institute, 1992
Past Experiences and Successes Were Mixed…
Why?• Impatience with waiting to see meaningful changes in disease
morbidity and mortality.
• Not sure which risk factor to focus on and in what order. Difficult to
keep traditionally categorical programs working in unison and loyal
to an overall centralized focus such as Healthy People 2000 or 2010.
• Program leaders and contractors would often change causing loss of
continuity, historical memory, and causing unnecessary re-starts or
no champion of the program left……AND
A comprehensive integrated evaluation plan - across programs and looking at a range of intermediate and ultimate
outcomes was never available…but now we have Maine and Arkansas doing this.
We Can Preview that in Arkansas in 2003:
• ARKANSAS CANCER CONTROL and TOBACCO CONTROL and CARDIOVASCULAR HEALTH …Programs, Interventions, and Outcomes overlap. Example:
Goal: Reduce the proportion of Arkansans who use tobacco products.
• Objective: Reduce the proportion of young people who have ever used tobacco products to 32% by 2005. From 1999 to 2001: 39.6% to 33% reported having ever used tobacco products.
• Goal: Encourage children and adults to adopt risk reduction habits.
• One Objective is to get to 5+ servings to be reported among children to reach 25% of the total population by 2005. From 1999 to 2001: 18.8% to 20% of school age children eat 5+ servings of fruits/vegetables per day.We are building an evaluation plan that will apply to several programs.
What is different in Maine in 2003 to foster a successful
evaluation? • Longevity:The Gallup evaluation team has been working
with BOH since 1998. • An integrated Logic Model of the PTM, MCVHP, and the
HMP programs has been developed.• A detailed Evaluation Plan looking at specific indicators
of success for each of the three initiatives and the overall Healthy Maine Partnerships has been developed.
• Gallup has senior staff with experience in this area that are committed to this type of public health approach.
• Gallup has a local Senior Consultant in Maine, Ms. Patricia Hart, very active in helping lead the evaluation efforts in Maine.
Partnership for a Tobacco-Free Maine (PTM)
• 31 Local HMP’s• Statewide Tobacco Treatment Services
(Helpline, Medication Vouchers, Provider Training)
• Statewide Media Campaign• Enforcement Activities• Youth Advocacy Network• Tobacco-Free Athletes Program• Programs Targeting Disparate Populations• Evaluation and Surveillance Activities
Maine Cardiovascular Health Program (MCVHP)
• Public Policy Initiatives• 31 Local HMP’s• Nutrition Initiative (Action Kits)• Physical Activity Initiative (Maine Walks)• Statewide Media Campaign• Secondary Prevention (Early Detection)• Worksite Programs• Youth and School-based Programs (YAP and CATCH)• Priority Populations (Low SES, Diabetes, Women,
and Native Americans)• Evaluation and Surveillance
Overview of Comprehensive Evaluation
• Provides first comprehensive evaluation, previous work was evaluative research but was not funded to look at all program components
• Evaluates the progress, outcomes and interrelationships of the Healthy Maine Partnerships, including the a) Maine Cardiovascular Health Program; b) the Partnership For A Tobacco-Free Maine; 3) the 31 local HMP intervention sites
• Leverages evaluation resources across three programs, recognizing program and outcome interdependencies
PTM, MCVHP, and HMP Logic Model
Inputs
Activities Outputs Initial Intermediate Long-Term Goal
OUTCOMES
· Statewide Health Department Infrastructure· Local HMP Programs· Maine Coalition on Smoking and Health· Maine Nutrition Network· Maine Cardiovascular Health Council· Maine Department of Education· Maine Department of Transportation· American Heart Association (NE)· Statewide Health Voluntary Agencies· Advisory Council· Maine Hospital Association· Maine Center for Public Health· Maine Association of Health, Physical Education, Recreational, and Dance· Governor’s Council on Physical Fitness and Sports· ME Cares·
Statewide Media Campaign/Counter-
marketing/Alternative Sponsorship
Local HMP Programs, Nutrition Initiative, Physical Activity
Initiative, Secondary Prevention Programs, Priority Population
Programs (Low SES/Rural, Women Diabetes, and Native
Americans), Treatment Initiatives (Helpline, Provider Training,
Medication), Tobacco Free Athletes, School-based Prevention and
Treatment
Exposure to nutrition, physical activity, and anti-
tobacco health messages
Increase access to and use of
services and programs
Creation of nutrition, physical
activity and no smoking policies, regulations, and
bans
Changes in awareness,
knowledge, attitudes and behaviors
Increase physical activity, healthy nutritional habits among youth and
adults
Compliance with, support for, and
enforcement of bans, regulations and
policies
Reduce tobacco initiation
among youth
Increase tobacco cessation among youth and adults
Public Policy Initiative, Worksite Programs,
Youth and School-based Programs (CATCH), Enforcement, Youth
Advocacy (MY TURN), YAP
Increased regular
physical activity and
healthy nutritional
habits
Decreased tobacco use prevalence
and consumption
Increased access to physical activity and
healthy nutrition
Reduced exposure to ETS
Increased environments that support physical activity, healthy
nutritional practices, and no
smoking
Reduced CHD-related morbidity
and mortality
Reduced CHD-related disparity
The Evaluation Plan is Centered on a Goal-Based Approach
• Mapping HMP outcomes as they relate to state health goals described in Healthy Maine 2010
• Following a detailed Evaluation Plan to identify key goals, objectives, strategies and measures of success
• Using a range of tools to inform results including program data, focus groups, key-informant interviews, surveys and web-based monitoring system
• Including input from a wide-range of stakeholders in the plan development
Evaluation Plan
• Lists Goals, Objectives, Indicators (short and long-term), Data Sources, and Frequency and Utility of Data
• Statewide HMP Initiative• Local HMP Initiative• PTM• MCVHP
PTM Example
• Goal 1: Prevent Initiation of tobacco use by youth
• Objective 2: Reduce the rate of illegal sales of tobacco products to minors…
• Indicator 4: The rate of tobacco sales to minors
• Data Source (s): Maine Dept. of the Attorney General
PTM Example
• Goal 2: To motivate and assist smokers to quit
• Objective 1: Reduce adult smoking (age 18+) prevalence to 19% by 2010
• Indicator 1: The prevalence of cigarette smoking in adults age 18 +
• Data Source (s): Maine BRFSS; Maine ATS
MCVHP Example
• Goal 1: To build an infrastructure at the state and local level to improve CVD health…
• Objective 1: Reduce coronary heart disease deaths to no more than 166 per 100,000 by 2010
• Indicator 1: The number of coronary heart disease deaths in Maine
• Data Source: Maine BOH, Office of Vital Statistics
Advantages of Integration in Evaluation
• Streamline data collection by combining surveys and other data collection instruments
• Demonstrate how the short-term outputs of several initiatives can lead to long-term positive health outcomes (e.g. tobacco use reduction is a key outcome for improved cardiovascular health)
• Address the program relationships in the evaluation and analysis (local intervention site feedback to state-designed initiatives)
Early Findings Show Positive Results of Integration
• In the first year, the local HMPs had more than 230 local policy/environmental changes with an average of seven per site
• The 31 local HMPs use the programming and strategies provided by the state-level programs like the MCVHP to enlist coalition members and create policy and environmental changes
• Benefits of integration include shortened program development time, sharing lessons learned and “doing what others are doing”
Environmental Indicators: A Baseline 2003
Environmental Indicators: A Baseline 2003
• Nine percent of the sampled towns in Maine have written policies requiring tobacco smoke free entrances in municipal buildings.
• Eleven percent of the sampled towns in Maine have a committee delegated to address walking and biking issues
Environmental Indicators: A Baseline 2003
• Fifty-three percent of the sampled towns in Maine had an indoor physical activity facility available at a school or college to community members for low or no cost
• Only 10% of the sampled towns in Maine have a community garden program where residents can grow fruits and vegetables. But 70% of towns in Maine have a nearby direct-buy farm such as a farmer’s market
Outcome Indicators
1. What’s your number(s) ?
Smoking Prevalence Data
0
10
20
30
1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002
Nationwide Maine
Copyright © 2003 The Gallup Organization, Princeton, NJ. All rights reserved.THE GALLUP ORGANIZATION
Consumption of Tobacco
Consumption of Tobacco
The faces of The Tobacco Industry
light up every time you do.
T H E T O B A C C O I N D U S T R Y. T H E Y P R O F I T. Y O U L O S E.
60.0
80.0
100.0
120.0
140.0
160.0
180.0
1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001
ME MaineTobacco Tax Increase
Per Capita Consumption Trends In Maine
2. Who Cares ?
Create smoke-free environments
Policies & Protection
90.0%
80%
100%
Maine
Secondhand Smoke Exposure 2001
Adults agreeing or strongly agreeing that
people should be protected from
Secondhand Smoke
63.0
0
20
40
60
80
2002
Protection from Secondhand Smoke in the Home - MaineProtection from Secondhand Smoke in the Home - MaineP
erc
en
tag
e o
f p
eop
le p
rote
cte
d
Reduce or Eliminate Youth Access to Tobacco
Reduce or Eliminate Youth Access to Tobacco
Fewer Stores are Selling Tobacco to Kids - MaineFewer Stores are Selling Tobacco to Kids - Maine
17.0
7.0
0
2
4
6
8
10
12
14
16
18
1997 2001
Pe
rce
nt
Pre
va
len
ce
Quitting Smoking
59.0%
0%
20%
40%
60%
80%
100%
Maine
Attempted to Quit in the last year 2001
Adult smokers who have tried to quit in the
last 12 months
3. What Works ?
PTM Tobacco Treatment Initiative
Of approximately 4,000 tobacco users assisted by the HelpLine, over 1,500 quit temporarily and 730 quit long-term.
The 730 individuals who quit long-term will produce a savings of of approximately $2.47 million in productivity and medical expenditures.§
Six-month Quit Rates: Counseled by HelpLine vs. Those Who Quit On Their Own
0%
10%
20%
30%
HelpLine CallersCounseled
No Assistance
Percent quit
95%
Confidence Inte
rval
How Effective is the Maine Tobacco HelpLine?
22% were not smoking 6 months after HelpLine counseling.
Findings Point out the Challenges and Benefits for
Integrating Program and Evaluation Needs
• Advance planning and communicating plans are important-- for buy-in, contributions, and smooth hand-offs among program staff.
• As work units and skill sets become specialized around a public health issue, communication can become internally focused and can benefit by building bridges with other staff and programs.
• Field staff and central office staff will have clear expectations, clear roles and clear lines of authority, and communicate often.
Challenges and Benefits for Integrating Program and
Evaluation Needs• Coordination among related programs brings a
united front in times of glory and downsizing.
• A proper Evaluation design will allow for individual program growth while contributing to an even larger overall public health perspective
• You will bring real meaning to the term “synergistic.” Program organization through evaluation can be a lynchpin to fostering great results….so it is worth all the effort !
Thanks For The Opportunity To Share Our Experiences of Being in
the Field With Programs in Arkansas and in Maine
With You.
Have a Great Rest of the Conference !
To Contact Us at Gallup:
• Dr. Michael Johnson in California at (530) 268-8218 Direct Line
• Dr. Howard Fishbein in Washington,DC at (202) 715-3178 Direct Line