wisconsin evaluation plan tobacco prevention and control … · 2018-04-09 · national tobacco...

Last Revision: Aug 2016

WISCONSIN EVALUATION PLAN

Tobacco Prevention and Control Movement

2016 – 2020

The Wisconsin Tobacco Prevention and Control Program (TPCP) coordinated local and state partners to form an Evaluation Work Group to create an Evaluation Plan for the statewide movement. This plan covers 2016 through 2020 and focuses on the “zoomed out”, system-level tobacco prevention and control movement, instead of specific programs that work towards prevention and cessation of tobacco. This plan was created by the Evaluation Work Group which was made up of local and state partners. Name Organization Partner Level

Vicki Huntington TPCP (DHS) State Randy Glysch TPCP (DHS) State Disa Patel TPCP (DHS) State Michelle Mercure American Lung Association State Andrea Gromoske Family Health Section (DHS) State Lisette Khalil Wisconsin Women’s Health Foundation State Elaine Valliere Great Lakes Inter-Tribal Council Tribal Lorraine Lathen Jump at the Sun, LLC Local Bruce Christiansen UW-Center for Tobacco Research Intervention Academic Karen Palmersheim UW-Center for Urban Initiatives and Research Academic Robin Moskowitz UW-Population Health Sciences Academic John Stevenson UW-Survey Center Academic Elizabeth Hagen Western Wisconsin Working for Tobacco-Free Living Local



2

Evaluation Question Overview

Based off the Tobacco Prevention and Control Movement Logic Model (Appendix A), the Evaluation Work Group determined overarching evaluation questions that are answered through the evaluation indicators found in Appendix B.

Cessation: Are Wisconsin cessation programs (i.e. Quit Line, First Breath, Freedom from Smoking, Community Programs, WiNTiP, UW-CTRI Outreach) effective in reaching vulnerable populations?

Rationale: Increased awareness of services Increased use of services Decreased smoking rate Indicators: 1-11

Disparities: Is the movement effective in reaching populations disparately impacted by tobacco?

Rationale: Increased resources created for and on disparate and vulnerable populations Increased activities address disparate populations Decreased disparity between populations using tobacco Indicators: 1-2, 12-14

Initiation: Is the prevalence of tobacco products decreasing among youth and young adults?

Rationale: Decreased access to products Decreased use of new and old products (including e-cigarettes) Decreased prevalence of tobacco initiation and use among youth and young adults Indicators: 1-2, 23-30

Secondhand Smoke Exposure: Is the prevalence of secondhand smoke exposure decreasing among youth and adults in the home?

Rationale: Increased smoke-free housing Increased cessation efforts Decreased exposure to youth and adults Indicators: 31-35

Intended Use and Users

This plan’s results will be used to monitor progress of the movement. Results are to be shared with stakeholders (e.g. topic specific work groups, funders, DHS leadership, partners, media, public) through various channels (e.g. press release, fact sheets, reports) as detailed in the Dissemination Plan. Primarily, results will be used by TPCP, funded partners, and the statewide coalition. The statewide coalition is going through a reformation process in 2016, and will receive evaluation results from the Evaluation Work Group after its creation. The statewide coalition will identify needs based on recommended focus areas from the Evaluation Work Group. TPCP will determine any adjustments to the implementation of activities and strategies (process measures) along with the impact of the program (outcome measures). Additionally, results will be used to ensure effective reporting and shared with partners and the public to



3

educate on program activities and outcomes. The statewide coalition will use the evaluation results to determine their policy focus for the upcoming years. Movement Description

The statewide Tobacco Prevention and Control Movement (referred to as “movement”) is supported through the efforts of the Department of Health Services’ Tobacco Prevention and Control Program (TPCP) and is illustrated in the movement’s logic model (Appendix A). TPCP coordinates efforts from local coalitions, statewide non-profits, academic settings including outreach and researchers, other Department of Health Services programs, and national partners to ensure synergy among activities and outputs. TPCP brings guiding plans (e.g. Healthy Wisconsin 2020, Wisconsin Tobacco Prevention and Control State Plan) to the forefront for works groups and the statewide coalition to plan and strategize activities. Additionally, the movement works within environmental factors that can have both positive and negative consequences. These factors include past and current tobacco science and research; local, state, and national policies; and the tobacco industry’s marketing and products. A majority of activities are led by local coalitions and state partners. Local coalitions educate stakeholders on tobacco prevention and cessation best practices and serve as content experts to aide in implementing local policies as needed. State partners provide services such as the Quit Line and cessation outreach specific to pregnant women and behavioral health care workforce and population (i.e. First Breath, WiNTiP, UW-CTRI Outreach). Additionally, state partners coordinate activities of statewide youth and young adults’ efforts (i.e. FACT, N-O-T, Spark, Wisconsin Wins), and of smoke-free housing efforts (i.e. Clear Gains). These activities lead to several outputs which are captured in the movement’s logic model (Appendix A) and key process outcomes are highlighted in the Evaluation Question Matrix (Appendix B). The long-term outcomes of the movement are to eliminate tobacco-related disparities, eliminate secondhand smoke exposure, increase cessation, prevent youth initiations, and have an effective, sustainable infrastructure. To reach these, the Evaluation Work Group determined short-term and intermediate outcomes that build to the long-term outcomes. These short-term and intermediate outcomes directly relate to the CDC required outcomes that were laid out in the CDC FOA DP15-1509. Appendix A has a logic model that overviews the movement’s inputs, activities, outputs, and outcomes. The statewide movement is multifaceted and is experiencing a couple stages of development. Overall, the movement is in maintenance as various components have evolved responding to environmental contexts such as the implementation of a statewide smoke-free air law and increased taxes on tobacco products. This has shifted the focus of the movement to educating on new products (i.e. e-cigarettes) and to refocus efforts on addressing tobacco-related disparities, which is one facet of the movement that is in the planning stage. Other facets in the planning stage are the statewide coalition as it is being reformed, and planning work with Counter Tools to gather data systematically on the local level that will tell a statewide story. These efforts are currently being strategized.



4

Evaluation Focus

The Evaluation Work Group prioritized questions within each long-term outcome. These questions were narrowed using specific criteria and assumptions. Similar questions were then merged and adjusted to be indicators. Additionally, indicators include a statement that demographic make-up will be determined for a majority of the evaluation questions. Although it is assumed looking at demographics will occur as part of assessing indicators, this is specifically stated to ensure assessing disparate populations occurs. These demographics will include: race/ethnicity, sexual orientation, gender identity, income, pregnant women, or education level. Additionally, questions that were not ranked by the group, questions without a viable data source, and questions that were program specific (e.g. FACT participation rate) were removed from the table. An assumption being made is that each review will ask, “What are the changes over time?” which will help in identifying trends. Overall, thirty evaluation indicators (twelve required by CDC as performance measures) were prioritized. TPCP proposed overarching evaluation questions (see Questions Overview) that encompasses and uses the thirty indicators as data to analyze. The thirty indicators are feasible to answer as a majority of the questions on the matrix are already gathered through surveillance. For those indicators that are not gathered through annual/bi-annual surveillance surveys, case studies can be conducted using TPCP staff to analyze current reporting on contractors’ activities (e.g. environmental scans). TPCP is confident using the Evaluation Indicators Matrix is an efficient use of time and resources. Furthermore, the creation of an internal tracking system will ensure questions are answered and discussed with the Evaluation Work Group for next steps. Stakeholder Engagement

TPCP coordinated the Evaluation Work Group by using the CDC’s Developing an Effective Evaluation Plan workbook. The Evaluation Work Group comprised of local and state partners who represent diverse organizations and who have experience in evaluation, research, program history, or surveillance. The Work Group convened for two in-person meetings where the group revised a movement-wide logic model and brainstormed and prioritized evaluation questions categorized by the National Tobacco Control Program (NTCP) goal areas. TPCP proposed a movement-wide logic model that the group revised by adding partners and connections to ensure a logical flow by adding activities, outputs, and outcomes for addressing tobacco-related health disparities, ensuring an effective infrastructure, and other long-term outcomes. Additionally, the group brainstormed questions with no restraints to feasibility as members of the group may have been unaware of available data resources and this facilitated creativity among members. Evaluation questions were to be based on a review of available data and program activities shown through the logic model (Appendix A). A subgroup of the Evaluation Work Group began prioritizing questions and added questions that were missing based on the



5

logic model and program description. Afterwards, the entire group prioritized questions within each long-term outcome. TPCP then narrowed these questions using specific criteria and assumptions (see Evaluation Focus). The Evaluation Work Group decided to meet every six months with quarterly teleconferences to monitor the program’s Evaluation Plan. Monitoring the Evaluation Plan allows TPCP to measure program implementation and impact, and identify areas of focus to improve overall program quality. Partners aid in choosing feasible evaluation and performance measures that best support TPCP’s four priority goals of the State Plan. Furthermore, the Evaluation Work Group will collaborate to determine which results should be shared with stakeholders. Indicators and performance measures Key evaluation questions and methods for answering questions are in Appendix B, Evaluation Question Matrix. These questions include required CDC Performance Measures. The Evaluation Work Group and TPCP focused the evaluation questions. After these were focused, TPCP shared overarching questions with the Evaluation Work Group that could be answered with the Evaluation Questions Matrix. Data sources TPCP has developed and will use data collection systems to collect process outcomes and measure effectiveness of program activities. Program reporting will encompass local and state partners’ activities, and TPCP will monitor implementation of activities such as media, resource development, and funding level. In addition to contractors’ and TPCP’s program reporting, the Research Scientist will continue to coordinate state surveillance systems with the Behavioral Risk Factor Surveillance System (BRFSS) and the Youth Tobacco Survey (YTS). Furthermore, the Research Scientist will work with the Maternal and Child Health Program (MCH) and the Division of Health Care Access and Accountability (DHCAA) to analyze birth certificates, Pregnancy Risk Assessment Monitoring System (PRAMS), Women, Infants, and Children (WIC), and Medicaid data. Methods Various methods will be used to assess both process and outcome measures. Majority of the information comes from either annual or bi-annual surveys (e.g. BRFSS, YTS) or from TPCP or contractor reporting. This aids in feasibility as a majority of the information has been gathered for several years. Also, the Department of Health Service continues to encourage program integration to aid collaborative efforts. TPCP will work with MCH along with DHCAA to answer evaluation questions related to infant exposure to secondhand smoke and Medicaid use of cessation treatment. TPCP is currently in the process of creating an internal tracking system to ensure questions are answered and tracked when reported. Part of the criteria for including an evaluation question was the ability to adequately measure the question and the most appropriate method and data exist to answer the question, and if the appropriate resources are available to accomplish this. Quality assurance procedures (e.g.



6

ensure adequate sample size, reliability, viability of data sources) will be put into place to ensure that the data used to answer the evaluation questions are reliable and checked for accuracy. Both quantitative and qualitative data collection methods will be used to collect the needed data to answer the evaluation questions, and only data with acceptable confidence intervals and representative samples will be used. Data sources being used are credible, national sources. Roles and responsibilities The Research Scientist will track information when survey data becomes available, and program data will be collected at mid and year-end reports. Questions and data will then be shared with the Evaluation Work Group. The Evaluation Work Group will then determine if any adjustments are needed with program activities to reach project period objectives. Credibility of evaluation information The majority of data sources are state or national surveillance surveys, lending to highly credible information. Other sources are program reporting which is reliable as it is reported on at least a quarterly basis. Creating an internal tracking system for TPCP will aid in reliable data as information will be collected regularly. Even with the use of these data sources, there is a need to focus data gathering for disparate populations. TPCP will capture more reliable statewide LGBT prevalence data in the 2015 BRFSS, has added an LGBT question to the Quit Line, and will add questions to BRFSS and conduct oversampling as needed to ensure adequate sample size for various population groups. Analysis and Interpretation Plan

The Research Scientist will coordinate and collaborate with members of the Evaluation Work Group to analyze data from the Evaluation Questions Matrix. These interim results will be shared with TPCP and the Evaluation Work Group through the Evaluation Questions Matrix. The Evaluation Work Group will hold quarterly calls to assess if enough information has been collected to answer any of the overarching questions. If data is gathered during a time the Evaluation Work Group would not be meeting, either an additional call will be scheduled or TPCP will propose conclusions to the Evaluation Work Group. The Evaluation Work Group will critique the data gathered and ask additional questions associated with the movement logic model. Additionally, the group will formulate conclusions from the evaluation to be shared with appropriate stakeholders (e.g. topic-specific work groups, funders, DHS). These stakeholders will have the opportunity to provide feedback on conclusions, both positive and negative. The Evaluation Work Group will then use this information to determine if the conclusions have been thoroughly justified, and determine what reports should be shared with external partners.



7

Dissemination Plan

The Evaluation Work Group will meet in-person twice a year to review data that has been collected. TPCP will share information from the Evaluation Question Matrix and the group will focus on the overarching questions that will aid in synthesizing the evaluation findings. The findings will be summarized for a yearly report. The yearly report (20-25 pages) will be created to detail progress made throughout the year and action steps for the coming year. The report will also be summarized (1-3 pages) for distribution to partners. A timeline for the Evaluation Work Group meetings and dissemination is found in Appendix C. Findings will be shared by NTCP goal area as the overarching evaluation questions are similarly categorized. TPCP has an infrastructure that includes topic-specific work groups: Youth and Young Adults, Multi-Unit Housing, Coalition Building, Communications, Other Tobacco Products, Sustaining States, and Disparities. These work groups will receive factsheets and/or issue briefs on the evaluation findings and proposed action steps as information is collected. These work groups will be able to use the information to determine appropriate next steps. TPCP will continue to coordinate the Evaluation Work Group to ensure effective communication and to facilitate ongoing program improvement. The group will determine results to be shared with stakeholders and update the Evaluation Plan’s Dissemination Plan at in-person meetings. The group will inform the design of materials tailored for specific audiences. Target Audience Goals Tools Timeline

TPCP Evaluation Work Group CDC

Detail findings of the evaluation report and identify next steps. Share a success story identified by the evaluation results

20-25 page Report

Annually

Local and state partners DHS

Share highlights of year and actions for the coming year

1-3 page summary

Annually

Topic-specific work group

Share findings from evaluation report for group to determine action steps

Factsheet or Issue Brief

6-month review

Local and state partners

Share plan summary (and logic model) with stakeholders to be transparent and ensure synergy for next five years

Evaluation Plan 1-3 page summary

April 2016

Local and state partners, media, general public

Share tobacco-related data specific to Wisconsin

Factsheet Annually



8

Appendix A Movement Logic Model



9

Appendix B Evaluation Indicators Matrix Bold = CDC Required Performance Measure

# NTCP Goal Area

Process/ Outcome Method Indicator Data Source Frequency

Responsibility Assumed TPCP unless otherwise stated

1 All Outcome Report Proportion of cigarette smoking among youth and adults What is the demographic make-up?

BRFSS Birth Certificates PRAMS YTS YRBSS

BRFSS, Birth Certificates, PRAMS – Annually YTS – Even Years YRBSS – Odd Years

TPCP YRBSS – DPI

2 All Outcome Report Proportion of tobacco use among youth and adults What is the demographic make-up?

BRFSS YTS YRBSS

BRFSS – Annually YTS – Even Years YRBSS – Odd Years

TPCP YRBSS – DPI

# NTCP Goal Area

Process/ Outcome Method Indicator Data Source Frequency Responsibility

3 Cessation Outcome Report Total quitline call volume by quarter What is the demographic make-up of callers?

Quit Line NQDW

Quarterly UW-CTRI

4 Cessation Outcome Report Total number of quitline tobacco users who receive a service What is the demographic of users who receive a service?

Quit Line NQDW

Quarterly UW-CTRI

5 Cessation Outcome Report Quitline reach to persons with low education levels Quit Line NQDW

6 months UW-CTRI

6 Cessation Outcome Survey Proportion of adult smokers who have made a quit attempt What is the demographic make-up of those who have attempted to quit?

BRFSS Annually TPCP



10

# NTCP Goal Area


7 Cessation Outcome Survey Proportion of young smokers who have made a quit attempt What is the demographic make-up of those who have attempted to quit?

YTS YRBSS

YTS – Even Years YRBSS – Odd Years

TPCP YRBSS – DPI

8 Cessation Outcome Survey Proportion of adults aware of the Quit Line, First Breath, Community Programs, and Freedom from Smoking What is the demographic make-up?

BRFSS Annually TPCP

9 Cessation Process Survey Proportion of patients asked about their tobacco use by their dentist or doctor What is the demographic make-up?

BRFSS Annually TPCP

10 Cessation Process Report Proportion of Wisconsin Medicaid recipients receiving cessation services What is the demographic make-up?

DHCAA Medicaid

Annually DHCAA

11 Cessation Outcome Report Number of electronic referrals occur with the implementation of Electronic Medical Records What is the demographic make-up?

UW-CTRI 6 months UW-CTRI

# NTCP Goal Area Process/ Outcome Method Indicator Data Source Frequency Responsibility

12 Disparities Process Program Tracking

Proportion of interventions and strategies implemented address disparate populations What is the breakdown of interventions by demographics?

Reporting 6 months TPCP

13 Disparities Process Program Tracking

Number resources created for priority populations (LGBTQ, Mental Health, Low SES)

Reporting 6 months TPCP

# NTCP Goal Area


14 Infrastructure Process Program Tracking

Percentage of funding (state, CDC, and other) used to meet CDC-recommended funding levels outlined in Best Practices – 2014

Reporting Annually TPCP



11


Number and type of staff positions maintained throughout the entire funding year to support the tobacco control program (e.g., program director, policy coordinator, communications specialist, cessation coordinator, surveillance and evaluation staff, fiscal management systems staff, and administrative staff)



Levels of infrastructure increased or maintained as defined by the Component Model of Infrastructure included in Best Practices – 2014)


17 Infrastructure S&E

Process Program Tracking

Number and type of tobacco-related surveys implemented during the funding year (e.g. BRFSS, YTS) and type of tobacco-related modules implemented (e.g., BRFSS, YRBSS)


18 Infrastructure S&E


Number and type of tobacco-related indicators developed and implemented in state surveillance systems during the funding year (e.g. ATS, YTS< BRFSS< YRBSS)


19 Infrastructure Communications


Number of monthly speaking opportunities by trained tobacco control spokespersons to educate decision makers, stakeholders, and public


20 Infrastructure Communications Disparities


Number of paid and earned media efforts that target populations or areas with high concentrations of smoking prevalence, secondhand smoke exposure, and chronic disease What is the demographic make-up?


21 Infrastructure Communications


Types of social media activities used to complement traditional paid and earned media efforts (e.g., social media campaign, posting content) and what is the reach of social media activities by social media site used (e.g., Facebook, Twitter, YouTube)

TPCP Tracker Annually TPCP

# NTCP Goal Area


22 Initiation Outcome Survey Average age at which young people first smoked a whole cigarette Age first tried tobacco products among youth and adults What is the demographic make-up?

YRBSS YTS BRFSS

YRBSS – Odd Years YTS – Even Years BRFSS – Annually

YRBSS – DPI TPCP



12

23 Initiation Outcome Survey Proportion of young people who report never having tried a cigarette What is the demographic make-up?

YRBSS Odd Years TPCP

24 Initiation Outcome Survey Proportion of youth able to buy tobacco What is the demographic make-up?

WI Wins, SYNAR Annually TPCP

25 Initiation Outcome Survey Prevalence of e-cigarette use by youth, young adults, & adults What is the demographic make-up?

YTS, BRFSS BRFSS – Annually YTS – Even years

TPCP

26 Initiation Outcome Survey Locations youth are getting their tobacco products YTS Even Years TPCP

27 Initiation Disparities

Outcome Program Reporting

Number colleges and universities are smoke/tobacco-free What is the demographic make-up? (i.e. tech schools, 2- or 4-year schools)

Spark Annually Spark

28 Initiation Outcome Reporting Number of cigarettes being sold per capita (adults) What is the demographic make-up? (i.e. county variance)

Monthly Revenue Data

6 months TPCP

# NTCP Goal Area


29 SHS Outcome Program Reporting

Proportion of public housing authorities with 100% smoke-free indoor air policies

Clear Gains Tracker 6 months Clear Gains

30 SHS Outcome Reporting Proportion of the population reporting exposure to secondhand smoke at workplace

TUS-CPS Annually TPCP

31 SHS Outcome Survey Prevalence of SHS exposure in vehicles and homes for youth and adults What is the demographic make-up?

BRFSS YTS

BRFSS – Annually YTS – Even years

TPCP

32 SHS Outcome Survey Proportion of households with infants have adult smokers What is the demographic make-up?

Birth Certificates PRAMS WIC

Annually TPCP



13

Appendix C Evaluation Work Group Timeline

Date Activity Topics Channel

2016 March Disseminate Evaluation Plan to partners State and local

partners

October Evaluation Work Group in-person meeting to review findings

December Disseminate findings

2017 April Evaluation Work Group in-person meeting to

review findings

June Disseminate findings




review findings





review findings





review findings




wisconsin evaluation plan tobacco prevention and control … · 2018-04-09 · national tobacco...

Documents