orientation: research and evaluation section (res) – key activities & services october 2012
TRANSCRIPT
Orientation:Research and Evaluation Section (RES) –
Key Activities & Services
October 2012
Overview of Webinar
• Research and Evaluation Section overview (slides 3-19)
• Impact/Outcome Evaluation (slides 20-65)
• Activity Tracking Form (slides 66-80)
• Policy, Systems, Environmental Change – SOW Objective 9 (slides 81-88)
2
3
Research and Evaluation Functions• Maintaining scientific integrity in all the work the Network
conducts
• Surveillance – statewide population-based surveys
• Process tracking evaluation – ATF for EARS & SAAR
• Program planning framework Communities of Excellence (CX 3)
• Outcome/Impact local evaluation and collaborative case studies
• Formative, process, and outcome/impact evaluation for other Network unit projects and emerging issues
• Developing and providing resources and instruments for program work and evaluation and disseminating information
• Technical assistance for awardees and collaboratives
• Collaboration on external research projects
4
Evaluation of Network Priority Outcomes:• Increased access to healthier foods and
beverages
• Increased opportunities for physical activity
• Healthier dietary habits
• Increased physical activity
• Increased Food Security
• Norm change around nutrition and physical activity
5
Network Surveys
Surveillance surveys are used to monitor the current dietary and physical activity practices, and related habits, attitudes, and beliefs of Californians
• California Dietary Practices Survey (CDPS)
Adults > 18 years
• California Teen Eating, Exercise, and Nutrition Survey (CalTEENS)
Teens 12 – 17 years
• California Children’s Healthy Eating and Exercise Practices Survey (CalCHEEPS)
Children 9 – 11 years
6
Surveillance data are also used to…
• represent the diverse racial/ethnicity characteristics of Californians and identify health disparities
• identify problem areas and track trends over time
• produce reports which are references for public health and other professionals
• generate news coverage
• provide information for legislators and other policymakers that can lead to positive change
7
Research Partnerships to Enhance Surveillance
• RES collaborates to get questions about Network issues included on statewide surveillance surveys conducted by other CDPH programs
• To develop local level data
– California Health Interview Survey (CHIS)
• To secure access to additional data for low-income women
– California Women’s Health Survey (CWHS)
• To ensure that key questions related to Network objectives are included each year
– CDC Behavioral Risk Factor Survey (BRFS)
Network survey data has been used to develop fact sheet reports about specific populations:
Latino Adults
Adults
in the Workplace
African American Adults
8
RES has compiled and posted external data at the Network Regional level
RES staff are available to answer questions about Network and non-Network surveys and other data sources for all grantees 9
http://www.cdph.ca.gov/programs/cpns/Pages/DataSources.aspx
10
Network Process Evaluation
• Is used to develop and maintain an effective evaluation system for tracking of all direct service projects overseen, managed, or funded by the Network
• Allows for the merging of similar information from a large variety of programs
• Helps programs and regions to measure their own progress
• Gathers data from funded grantees using an activity tracking form (ATF)
11
CX3 – Communities of Excellence
• A planning model to assess communities in relation to a variety of obesity prevention benchmarks known as community indicators and assets.
• 3 = nutrition, physical activity and obesity prevention
• Standardized indicators & assets – big plus!
• Set standards of excellence.
12
Outcome/Impact Evaluation (IOE) Studies of Grantee Interventions
• RES provides technical assistance, analysis, and summary reports on outcome/impact evaluation projects from grantees receiving $350,000+
• IOE measures behavior and behavioral determinants
• Knowledge is not enough to demonstrate outcome
• Many grantees have been demonstrating positive outcomes
13
Formative Evaluation
• Development of social marketing campaigns, materials, and interventions
• Exploration of new target population or health problem
• Data sources: focus groups, key informant interviews, pilot tests, satisfaction surveys
• FFY 12 - Rethink Your Drink
14
Process, Outcome and Impact Evaluation Studies of New Network Interventions and Resources• Formal evaluations are conducted of specific interventions and
curricula, particularly when there are plans to disseminate widely
• Formal evaluations are conducted to see if a program works in specific channels, especially those new to public health
• Network FFY 2012:• Youth Engagement Initiative Process Evaluation
• African American Beauty Shop Pilot Study
• Retail Point of Purchase Marketing Study
• Children’s Power Play! Impact Evaluation
• Latino Campaign Impact Evaluation
15
Developing Consumer-Tested Media Messaging and Nutrition Education Materials
• Focus groups guides, structured interviews, self-administered questionnaires:
• Rethink Your Drink
• CalFresh
• Champions for Change – Latino
• Champions for Change – African American
• Champions for Change – Multi-cultural
• An array of Fruit, Vegetable, and Physical Activity Campaign and Program consumer materials
16
Evaluating Communications Strategy
• Benchmark Media Tracking Survey
– Annual
– Random Digit Dial for general population, random dial from CalFresh participant list, and mall survey of other low-income populations
– Exposure to Network media messaging – TV, radio, and outdoor
– Exposure to other nutrition and physical activity programming
– FV, PA, and SSB behavior, attitudes, norms, opinions
17
Resources and Disseminating Information
• GIS - http://www.cnngis.org/
• Interactive, internet-based Geographic Information System (GIS) that allows users to view and query mapped Network-relevant data
• Used to identify populations and features of interest
• Can be mapped by county, city, census tract, Network Region, or Zip Code
• Extensive data layers include, among others:– Up to date American Community Survey data for 185% and 125%
FPL for All Races and Race/Ethnic Groups
– Grocery stores, restaurants, and fast food places
– CalFresh participant data
– CalFresh and WIC vendor data
18
In-House Consultation and Technical Assistance for Grantees and
Collaboratives Facts and Figures
• Grantee and collaborative questions
• Training and meeting evaluations
• Grant applications
• Fact sheets
• Press materials
• Talking points
• Policy briefs and bill analysis data
• Requests from senior management
• Information for the department and Office of Press Affairs
To learn more about Network Research and Evaluation• http://www.cdph.ca.gov/programs/cpns/
Pages/ResearchEvaluation.aspx
• Contact information
Sharon Sugerman, MS, RD
916-449-5406
19
Presented by:
Carolyn Rider & Amanda Linares Research Associates, Research & Evaluation Section
An Orientation for Local Health Departments
Impact/Outcome
Evaluation
How it works…
• Objective– Identify successful interventions and potential best
practices and provide direction for program improvement, refinement, and redirection of effort
• Participants– Local Health Departments (LHDs) receiving >$350,000
or choosing Objective 11 as an optional objective
• Evaluation Support– Research & Evaluation Section (RES) provides training;
quality surveys, evaluation tools and resources; and year-round one-on-one evaluation guidance
How it works…
• State level analysis of compiled LHD findings enables the Network to assess overall project accomplishment
• Individual and compiled LHD findings are shared with USDA annually
How it works…
• Some aspects of evaluations are standardized – Minimum sample sizes– Standardized survey tools– Pre-test/post-test design
• Other evaluation components are designed locally, for example:– Optional survey modules can be added– Control groups
Why Evaluate?
• Your organization– How can we improve our intervention?– What works?
• USDA and other funders– What are we getting from the money we are
spending on these programs?
• Public– There is a high level of interest in improving
diets and physical activity and reducing the prevalence of overweight and obesity
POLL QUESTION #1
What is your level of experience with evaluation? (select only one)
•I am not familiar with evaluation at all.
•I am familiar with evaluation, but have not participated in one
•I have worked on an evaluation project outside of the Network
•I have worked on a Network evaluation project
What does “Impact/Outcome Evaluation” Mean?• Impact Evaluation and Outcome Evaluation
are similar types of evaluation – either can be used to meet Objective 11
• Both use a pre-test/post-test survey design
• Impact Evaluation is more rigorous and allows evaluators, program staff, and stakeholders to draw stronger conclusions about evaluation results and the impact of an intervention on program goals
Impact Evaluation
• Includes a control or comparison group
• Strongest designs include randomization
• Provides evidence that outcomes are a result of the intervention
• Excludes alternative interpretations
• Only a few of our IOE projects conduct Impact Evaluation
Outcome Evaluation
• No control or comparison group
• Determines if changes occur in conjunction with an intervention
• Cannot exclude alternative explanations, i.e. does not prove that the observed outcomes are attributable to the intervention
• Most of our IOE projects are outcome evaluation
Impact vs. Outcome Evaluation
Impact Outcome
Pre/Post-test Survey Design X X
Control Group X
Evidence that intervention causes outcomes
X
Evidence that outcomes occur in conjunction with intervention
X
Local Health Department Scope of Work (SOW)
• GOAL 1: – The target population (Supplemental Nutrition Assistance Program-
Education (SNAP-Ed)/Nutrition Education and Obesity Prevention (NEOP) participants and those eligible up to 185% Federal Poverty Level (FPL)) is empowered and enabled to select healthy foods and beverages and increase physical activity through nutrition education, social marketing and environmental supports.
• Objective 11: – (Impact/Outcome) Annually, conduct and report on Impact Outcome
Evaluation (IOE) to assess change in healthy food and beverage consumption and related factors such as perceived benefits of eating healthier foods and beverages, perceived control, self-efficacy, readiness to consume healthier foods and beverages, and perceived diet quality among a group of at least 100 participants.
The IOE Cycle
The IOE Cycle
Intervention
LHDs are expected to develop an intervention that has face-to-face contact with the same individuals for at least 30 minutes, on five or more different occasions
– This may include contacts made by subcontractors or collaborating agencies if it can be demonstrated that the individual has contact at the other site
– We require a minimum of 100 “matched” pre and post tests, i.e. the same participant tracked throughout the evaluation period
• To get 100 matched pairs, your intervention will need to include many more than 100 participants, especially if working with an adult population
– LHDs work with their PMs to develop their intervention
POLL QUESTION #2
With what population(s) do you plan to conduct an intervention series? (select all that apply)
•Children (grades 3-8)
•High school
•Adults (including parent education)
•Not yet decided
Selecting an Intervention
What are some common types of interventions?•Youth
– Power Play!
– Harvest of the Month
– Rethink Your Drink
– Or some combination
•Adults– Custom
•Interventions or components of them should be taken from the Network’s approved materials list: http://www.cdph.ca.gov/programs/cpns/Documents/Network-LHD-NutEdList.pdf•Must be Program Manager-approved if not on approved materials list
The IOE Cycle
Selecting the Appropriate Survey Instrument• Mandatory Survey Instruments
– Youth: Network Youth Survey
– High School: Network High School Survey or Beverage and Snack Questionnaire 2 (BSQ2)
– Adult: Food Behavior Checklist, Fruit and Vegetable Checklist, or Adult Rethink Your Drink Survey
• Optional Survey Instruments– Network Compendium of Surveys is a comprehensive collection
of Network RES-approved survey instruments
Compendium of Surveys
• Updated regularly with the best program evaluation measures we have to offer
• Instrument topics include knowledge, self-efficacy, social norms, social support, food security, access to/availability of healthy food and physical activity opportunities, and many more.
• A new edition is currently in preparation for release by Nov. 2012
The IOE Cycle
Evaluation Plan
• FFY 13 evaluation plans are due by November 30th
• This templated form will ask about:
– Detailed intervention information
– Target population
– Evaluation design
– Survey selection
– Site of intervention
– Evaluation cost
What if my LHD is not ready to submit an IOE Plan yet?• You may not be ready to submit an IOE plan to us
on November 30th for various reasons, including the need to hire subcontractors, not yet identifying an appropriate intervention, and evaluation staff vacancies
• If this is the case, you can fill out and return to us your FFY 13 IOE Pre-Plan
• This document will keep us informed about your readiness to conduct IOE in FFY13, and we will follow up on your progress regularly
POLL QUESTION #3
Will your LHD conduct your intervention directly, or will your department hire one or more sub-contractors to conduct the intervention? (select one response)
•The LHD will do the intervention work w/o sub-contracting.
•The LHD will hire one sub-contractor to conduct an intervention.
•The LHD will hire multiple subs to conduct one or more interventions.
POLL QUESTION #4
If your department is sub-contracting the intervention work:
What is the status of sub-contracts for your intervention? (select all that apply)
•Sub-contracts are in place.
•Subs are identified, contracts are not yet executed.
•Sub-contractor selection is in process.
•Sub-contractor selection process has not begun.
Evaluation Design Considerations• How will you obtain your sample?
– Convenience or random sample
• Will you follow the standard pre/post-test design or add a follow-up test to measure sustained change?
• Will you use a control group?
– A group of participants not exposed to the intervention and compared to the intervention group
– Helps you to conclude whether or not your evaluation findings are a result of your intervention
– It’s easier to acquire a control group with some populations than others
The IOE Cycle
Teleconferences
• Who is there?– All necessary staff from your LHD– Any subcontractors essential to your project– Network Program Manager– Network Research & Evaluation representative
• What will happen?– In FFY 13, we will discuss your plan and ask you to
make changes as necessary; troubleshoot any potential issues with intervention and evaluation
– Beyond FFY 13, we will use this hour to discuss your Final Report from the previous year and your plan for next year
The IOE Cycle
Administration of Pre-Tests
The administration of pre-tests should only take place AFTER your IOE Plan is approved, but BEFORE any intervention begins.
Administration of Pre-Tests
We provide instruction guides/protocols to assist in the administration of these surveys:
– Network Youth Survey
– Network High School Survey
– Fruit and Vegetable Checklist
– Food Behavior Checklist
Assigning IDs for Pre-Tests
• Each participant needs to have a unique ID assigned and labeled on their pre-test
• This ID will be used to:– maintain confidentiality of participants’
survey responses– match pre-test and post-test surveys – enter survey data
The IOE Cycle
FFY 2012 LHD Interventions
In FFY 2012, seven Network LHDs completed IOE. Most evaluated interventions with one age group only, while one worked with both children and adults.
Population Children High School Adult
# of Interventions 5 0 3
# of Successful Interventions*
4 n/a 2
*Successful interventions are those reporting statistically significant increases in fruit and/or vegetable consumption and/or decreases in sugar-sweetened beverage consumption.
Insights from the Field
Speaking: María Ortiz-Padilla, MPA
Community Wellness & Prevention Program
Contra Costa Health Services
The IOE Cycle
Administration of Post-Tests
• Remember that the goal is to match the participants completing post-test surveys with completed pre-test surveys
• Be sure to follow all of the same procedures used during pre-tests for the best evaluation results – use your instruction guide/protocol
The IOE Cycle
POLL QUESTION #5
What is your level of experience with statistics? (select only one)
•None
•A little
•Some
•A lot
How to Analyze Survey Data
LHDs use data entry templates (DETs) created for our standard surveys to enter data and produce a basic, standardized analysis
This DET has data entered for the Network High School Survey.
How to Analyze Survey Data
• P-values are computed based on a paired t-test to show statistical significance
• Red text automatically appears to highlight statistically significant results (p-values ≤ 0.05)
• Red means change!
• Pre- and post-test means are calculated only for subjects w/ complete data for a given item
• Demographics are computed for the entire sample
The IOE Cycle
Complete a Final Report
• RES provides an Excel-based Final Report template– Questions will be more in-depth, yet similar to those in
the IOE Plan– We will ask you to interpret results from your DETs and
elaborate on how they will influence your plan for the following year
– You will be asked to share some qualitative outcomes in addition to reporting your quantitative survey findings
– These reports are submitted to USDA as part of the Network’s Annual Report
– The IOE Final Report, completed DET AND Plan for next year are due on July 31, 2013 (and annually thereafter)
The IOE Cycle
Thank you!
While it’s an exciting time for everyone, we acknowledge how much information is
coming your way…
With that in mind, Network RES staff are here to help all participating LHDs fulfill your
Impact/Outcome Evaluation objectives in FFY 13
and beyond!
POLL QUESTIONS #6 & #7
We will likely be offering in-person and/or webinar-based IOE trainings in the near future.
1. Would your key LHD evaluation staff and/or subcontractors prefer in-person or webinar-based trainings? (select only one)
•In-Person
•Webinar-based
2. When do you anticipate being ready for this training? (select only one)
•November or December
•January or February
•March or later
The Network IOE Team
Carolyn Rider, MA• IOE project coordination
• Main point of contact for technical assistance: planning and reporting; data entry templates; evaluation materials and resources; evaluation requirements
• Teleconferences
• Compendium of Surveys
Amanda Linares, MS• IOE data management and analysis
• Compilation of submitted IOE data and USDA report generation
• Teleconferences
• Compendium of Surveys
You may be referred to other staff, but your first point of contact is [email protected] or [email protected]. More info at:
http://www.cdph.ca.gov/programs/cpns/Pages/ImpactEvaluation.aspx
Presented by:
Alexis Narodovich, MPHResearch Associate IVResearch & Evaluation Section
An Orientation for Local Health Departments
Activity Tracking Form
What is an Activity Tracking Form?
• The Activity Tracking Form (ATF) is an Excel document that is used to track all events and activities you conduct as part of your FFY13 Network Scope of Work.
• Two non-cumulative ATFs will be maintained during FFY13:
1. Semi-Annual ATF: activities conducted between October 1, 2012-March 30, 2013
2. Annual ATF: activities conducted between April 1, 2013-September 30, 2013
Why do we have an ATF?
The ATF serves two functions:
1) Allows you and your Network Program Manager to know that your organization is meeting the objectives outlined in your Scope of Work (SOW).
2) Compiles the data the Network is required to report to the USDA and the State on the reach and type of activities and SNAP-Ed participants served.
ATF Basics
• The ATF will be emailed to the lead contact person on the contract by 10/31/12.
• Lead person is responsible for emailing the ATF to other staff or subcontractors who will need to keep track of Network activities.
ATF Training Dates
ATF webinar-based trainings: •Friday, October 26 10:00am-12:00pm•Monday, October 29 1:00pm-3:00pm•Tuesday, November 6 10:00am-12:00pm
Network Target Audience
• Network target audience:– ≤ 130% FPL (i.e. CalFresh recipients,
free school meals)– ≤ 185% FPL (i.e. reduced school meal,
WIC) ------------------------------------------
• Non-Target Audience:- Staff/teachers- Stakeholders (i.e. health department
leadership) - Coalitions/Collaboratives
Categories of Events
Network target audience events are recorded as:
1.Direct Educationor
2.Indirect Education
Direct Education
• Interventions where a participant is actively engaged in the learning process with an educator and/or interactive media for at least 15 minutes and demographic data is available for the participants.
• Examples: classroom lessons, planned one-on-one nutrition education, grocery store or farmers’ market tours, and cooking demonstrations.
Direct Education
• Direct education is delivered in a way that allows educators to obtain information about individual participants.
• USDA requires the CalFresh participation status, age, gender, and race/ethnicity for each direct education participant.
Direct Education
• For many adult participants, demographics are collected using data cards available in Spanish and English.
• Data cards are not needed or used for direct education conducted with students at schools, preschools, day care centers, Head Start Programs.
Participant Data Collection CardWe would like to learn about people who attend our activities to help us improve services. Your answers are combined with everyone else’s and cannot be used to identify you. Everyone here today should fill out one of these forms. Thank you for your help!
1) Do you participate in CalFresh (Food Stamps, SNAP, EBT) or the Free School Meal program?
Yes (SKIP TO QUESTION3) No
2) If you answered NO to question 1, check any programs you take part in:
California Food Assistance Program (CFAP) Reduced Price School Meal
CalWORKS Section 8 Public Housing
Child & Adult Care Food Program (CACFP) Summer Food Program
Head Start Supplemental Security Income (SSI)
Medi-Cal Women, Infants, & Children (WIC Program)
3) Please check your age range: 0-4 years 5-17 years 18-59 years 60+
4) Please check your sex: Female Male
5) Is this the first time you have filled out this form since October 1, 2012?
Yes (GO TO QUESTION 6) No/Don’t know (STOP HERE)
6) Please choose ONE GROUP that describes you best.
White, not Hispanic Native American/Alaska Native, not Hispanic
White and Hispanic/Latino Native American/Alaska Native and Hispanic
Hispanic/Latino only Hawaiian Native/Pacific Islander, not Hispanic
African American, not Hispanic/Latino Hawaiian Native/Pacific Islander, and Hispanic
African American and Hispanic/Latino More than one race, not Hispanic/Latino
Asian, not Hispanic/Latino More than one race, and Hispanic/Latino
Asian, and Hispanic/Latino
Indirect Education
• The distribution of information and resources that are designed to increase public awareness of SNAP-Ed and/or increase awareness and knowledge of food, dietary quality, food security, food safety, and food resource management/shopping behaviors.
Indirect Education
Examples:
• Mass Communications: radio, TV, billboards, posters, newspapers.
• Print Materials Distribution: flyers, facts sheets, pamphlets, newsletters, nutrition articles.
• Displays of Educational Materials: bulletin boards, posters.
• Public Events: community events, fairs, exhibits.
Non-Target Audience
• Events with the non-target audience include those with staff, colleagues, teachers, government officials, stakeholders and peers that are part of the Scope of Work.
• Such events are not direct education or indirect education because they are not directly for our target population.
Technical Assistance
Alexis Narodovich, MPH
(916) 650-6905
Evan Talmage, BA
(916) 449-5407
Policy, Systems, Environmental Change Evaluation
October 2012
Objective 9 – Required for all LHDs
Activities Deliverables Timeframe
1. Based on Plan developed in collaboration with Network Research and Evaluation consultant and Program Manager, determine, develop, and implement evaluation method appropriate to the stage of the intervention the organization has chosen to focus their work on for two (2) significant targeted interventions: one each in the areas of reducing consumption of sugar sweetened beverage and increasing access to healthy fresh food (see Baseline Objectives 8;5, 8.6 and 2.4). Mixed methods may be used.
Annual evaluation plan
An evaluation instrument (s) —structured interview, moderator’s guide, survey, etc.
Report Annually:
Year 1 Plan
and Instrument10/1/2012-2/28/2013
Successive
Years10/1/2013
10/1/2014
10/1/2015
Objective 2.4 - Access to healthy foods, beverages and/or PA• Multi-level approaches to advance and market a minimum of one nutrition
and obesity prevention intervention/year related to CNAP. Submit a CNAP plan of action for review and approval to the Network PM. Strategies may include:
– Increase access to farmer’s markets through location, Electronic Benefit Transfer (EBT) and WIC coupon acceptance to increase consumption of fresh fruits and vegetables
– Increase Farm to Fork efforts in qualified schools, work places and community organizations, etc. to increase access to fresh fruits and vegetables
– Improve access to FNS programs such as school breakfast, lunch and summer meals, CalFresh and WIC
– Promote access to physical activity facilities through joint use policies– Promote access to healthy foods and beverages– Establish gardens in eligible community sites such as schools or public
housing
Objective 8.5 – Increase and promote healthy beverage options
• Identify priorities and develop a list of environmental support strategies with local partners to increase and promote access to healthy beverage options through public health approaches. Submit summary of local strategies to PM for review and approval. Some strategies may include:
– Collaborate with local school district to update their wellness policy to reduce access to sugar-sweeten beverages (SSBs) and provide more healthy beverages option in appropriate serving sizes on campus
– Reduce SSBs from county welfare offices, public housing units, city parks and recreational facilities and/or school vending machines in eligible settings serving low-income populations; provide healthy beverage alternatives in appropriate serving sizes
– Encourage partners to provide free drinking water to the public in common areas at such eligible venues: i. city and county facilities, ii. worksites, schools, iii. preschools, iv. afterschool programs, v. community organizations
– Collaborate with local youth serving organizations working with low-income populations (such as parks and rec, sports leagues, booster clubs, etc.) to ensure that healthy beverages are available at community events for purchase
– Encourage organizations to seek healthy beverage sponsorships
Examples of Data Used to Evaluate Policy/Systems/Environmental Change• Focus group - People who have a common
interest and can be easily gathered together
• Key informant interview (KII) - People who have in-depth experience or specialized knowledge
• Public opinion poll - People who would be affected by a change
• Media activity record – Record of media activities
• Observation data – Behaviors, activities, presence or absence of something
• Policy record – Records maintained by government or other organizations or institutions
First Steps
• In Year 1, the evaluation will be formative, although as the work progress, it will be possible to generate outcomes.
• The first year’s work cannot begin until the contractor has identified the strategies for change, including the community engagement process (Objective 5).
• Objective 5: Collaborate with community groups and other organizations to engage neighborhood members to identify at least two food and beverage strategies in qualifying communities to increase access and consumption of healthy foods and beverages.
After Strategies for Change Are Identified• Work with RES staff to develop an evaluation
plan for each of the strategies:
– Determine instruments to use for baseline data collection
– Decide whether or not training will be needed for data collectors and, if yes, how to determine they are ready to go into the field
– Additional tracking measures
– Data organization
