evaluating nutrition education programs leslie a lytle, phd, rd professor division of epidemiology...

Evaluating Nutrition Education Programs

Leslie A Lytle, PhD, RD

Professor

Division of Epidemiology and Community Health

University of Minnesota

l

• [email protected]

• [email protected] (Center for Youth Health Promotion website at UMN- tools, materials available for free)

mailto:[email protected]

mailto:[email protected]

The WINNERS!!!

• How to assess behavior

• Evaluating young children

• Assessing knowledge and attitudes

• Validity and reliability

• Sample size (unit of assignment)

• Strong study design

Other questions

• How to evaluate “one-shot” nutrition ed?/minumum exposure for nutr.ed to stick?

• Evaluating parental response to interventions targeted at the child (TEENS, CATCH examples)

• Home influence on kids’ eating choices• Teacher evaluating change in kids…

Types of evaluation

• Outcome evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007):

“Outcome- which demonstrates changes that occur in the presence of an intervention but do not establish cause and effect conclusions.”

Types of evaluation

• Impact evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007):

• Impact-which indicate how effective the intervention was in changing the target population’s attitude, awareness and/or behavior. (“knowledge, skills, attitudes, intention to act, behavior, or something else” (pg 29 FTNEPG, 3/2007)

• Assumes that a causal relationship is being tested

Assessing behaviors, knowledge and attitudes

What behaviors to focus on?

How do knowledge and attitudes fit in?

What tools are out there?

Which behaviors are easier/more difficult to assess?

How does age influence the evaluation tools that we use?

Recommended Behavioral Outcomes from FNS

• Eat fruits and vegetables

• Eat whole grains

• Consume fat-free or low-fat milk products every day

• Be physically active every day

• Balance caloric intake from food and beverages with calories expended

Recommended Behavioral Outcomes from FNS

• Evaluation tools should reflect knowledge, attitudes, behaviors related to these behavioral outcomes…

Designing an intervention and evaluation plan

• START with a conceptual model or logic model

Theory of Planned Behavior

Attitude Attitude toward the toward the

behaviorbehavior

Subjective Subjective normnorm

Perceived Perceived behavioral behavioral

controlcontrol

IntentionIntention BehaviorBehavior

Examining nutrition education impact/outcomes

KNOWLEDGE

ATTITUDES

BEHAVIORS

Example of how KAB should line up in evaluation

KNOWLEDGE: Does the student know how long is needed for adequate handwashing?

ATTITUDES: Does the student believe that it is important to wash their hands?

BEHAVIORS: Does the student report frequent handwashing or is observed consistently washing hands?


KNOWLEDGE:“Why” versus “How to knowledge”

“How to” KNOWLEDGE:Knowledge that will help one make

healthier choices rather than knowledge for the sake of knowledge


WHY KNOWLEDGE:

• Which of the following organs is responsible for secreting insulin?

a. Kidneysb. Lungsc. Liverd. Pancreas


HOW TO KNOWLEDGE (Example from Univ. of Nebraska):

• Which food is a lower fat snack?a. Pretzelsb. Potato chipsc. Doughnuts


HOW TO KNOWLEDGE (Example from Univ. of Wyoming)

• Write the name of one food found in each food group from MyPyramid.

Milk groupGrain groupMeat groupFruit groupVegetable group

More “how to” knowledge questions…(Kansas State)

• What is the healthiest snack choice?a. Soda pop and chipsb. Milkshake and friesc. Fruit juice and pretzels

• Which foods would always be safe to pack in a sack lunch?a. Sliced hamb. Peanut Butterc. Sliced cheese

• How long should I wash my hands before I touch or eat food?a. As long as it takes to say my nameb. As long as it takes to sing “Happy Birthday”c. As long as it takes to eat an apple

Assessing attitudes

• Motivation

• Beliefs, opinions

• Perceived benefits/barriers

• Subjective norm (how do others feel about this behavior?)

• Self efficacy (how confident do I feel about performing some behavior?

Attitude examples (Iowa State)

• I like to eat fruits and vegetables for snacks

• My friends like to eat fruits and vegetables for snacks

• How sure are you that you can fix fruits and vegetables to eat as snacks at home?

Measuring behavior

• Preference

• Self-report behavior

• Observed behavior

Other preference type questions

• See CATCH HBS: Section A: What food would you pick?

• Fruit and veggies lists

• 5-a-day Card Sorting Task for younger kids

Behavioral measures to assess

• Eat fruits and vegetables• Eat whole grains• Consume fat-free or low-fat milk

products every day• Be physically active every day• Balance caloric intake from food and

beverages with calories expended

Eat Fruits and Vegetables: Self report measures

• 24 hour recalls (valid in youth as young as 8 years old)

• Youth Adolescent Questionnaire-YAQ food frequency (valid in youth from about 6th grade and up)

• Validated 6 item fruit and veggie screener from BRFSS (valid in youth from about 6th grade and up)

Assessing fruit and veggie intake: 6th graders and older

BRFSS tool•These questions are about the foods you usually eat or drink. Please tell us how often, over the past year, you ate or drank each one. Mark the box for the response that best describes how often you eat or drink the food. Be sure to include foods you ate or drank at home, at school, at restaurants or anywhere else.1. Over the past year, how many times did you drink fruit juices such as orange, grapefruit, or tomato? (Mark only ONE box.)

Never or less than once per MONTH 1 time per DAY

1-3 times per MONTH 2 times per DAY

1-2 times per WEEK 3 times per DAY


5-6 times per WEEK 5 or more times per DAY

• 2. Over the past year, how many times did you eat fruit (not counting juice)? Mark only ONE box.)

Never or less than once per MONTH 1 time per DAY

1-3 times per MONTH 2 times per DAY



5-6 times per WEEK 5 or more times per DAY

• Over the past year, how many times did you eat green salad? (Mark only one box.)

Never or less than once per MONTH

1 time per DAY

1-3 times per MONTH 2 times per DAY 1-2 times per WEEK 3 times per DAY 3-4 times per WEEK 4 times per DAY 5-6 times per WEEK 5 or more times per DAY

• Over the past year, how many times did you eat potatoes (not including french fries, fried potatoes, or potato chips)?


1 time per DAY


• Over the past year, how many times did you eat carrots? (Mark only ONE box.)


1 time per DAY


• Over the past year, how many times did you eat vegetables (not counting carrots, potatoes, or salad)? (Mark only ONE box)


1 time per DAY


Eat Fruits and Vegetables: Self report measures

• How about for kids younger than 8?

– Observation

– Parental reporting

Behavioral outcomes to assess

• Fruits and vegetables• Whole grain consumption: Are there valid

and reliable tools? Can youth identify whole grain foods?

• Fat free or low fat dairy• Be physically active every day• Balance caloric intake from food and



• Fruits and vegetables• Whole grain consumption• Fat free or low fat dairy: EAT Survey 10-item

scale• Be physically active every day• Balance caloric intake from food and


Assessing Calcium intakes in middle school age youth

• Project EAT (Lisa Harnack, P.I.)• 10 item scale• Reliability and validity assessed• Reliability: test:retest one week apart• Validity: 3 24-hour recalls as criterion

measure

Harnack, Lytle, Story et al, JADA 2006;106: 1790-1795

Reliability and Validity

• Reliability: will the instrument produce the same result if applied twice?

• Do respondents understand the question in a consistent manner? (test-retest)

• Are observers consistent in documenting what they see? (inter-rater reliability)

• Do the items that I have included to assess a construct relate to each other as I would expect them to? (cronbach’s alpha)

Reliability and Validity

• Criterion Validity: How well does a measure compare with a “gold standard” or criterion measure?

• Face Validity: Would others reading the item understand the concept that I am trying to assess? Does the measure “appear”to measure the desired construct?

• Content validity: Does my instrument capture all the elements that I think are important

• Construct validity: Is the measure related to other measures as one would expect?


Test-retest correlations: Reliability

Sample ICCTotal sample 0.74Females 0.79Males 0.6611-12 y.o 0.7413-14 y.o. 0.75


Assessing Validity: Comparing calcium intake using 3 methods (n=248)

• 24 hour recalls: 993mg (499)• 10 item CA FFQ: 856mg (570)• 1 YRBS question: 423mg (344)• Correlations

– Recalls and FFQ= 0.43– Recalls and YRBS= 0.37


• Fruits and vegetables

• Whole grain consumption

• Fat free or low fat dairy



TEENS Survey Questions

• Do you get some regular physical activity outside of school? By regular we mean at least three times a week for at least 20 minutes at a time. Most of the time Usually Once in a while Hardly ever Never

• When you think about how hard you work out when you are physically active, do you find that you are Breathing much harder than usual Breathing somewhat harder than usual Breathing only a little harder than usual Breathing the same as usual

• How many hours per day do you usually watch TV during the weekdays? I don’t watch TV during the weekdays Less than 1 hour per day 1-2 hours per day 3-4 hours per day More than 4 hours per day

• How many hours per day do you usually watch TV during the weekend? I don’t watch TV during the weekend Less than 1 hour per day 1-2 hours per day 3-4 hours per day More than 4 hours per day

• How many hours per day do you usually play video games (including hand-held video games and computer games) during the weekdays? I don’t play video games during the weekdays Less than 1 hour per day 1-2 hours per day 3-4 hours per day More than 4 hours per day

• How many hours per day do you usually play video games (including hand-held video games and computer games) during the weekend? I don’t play video games during the weekend Less than 1 hour per day 1-2 hours per day 3-4 hours per day More than 4 hours per day


• Fruits and vegetables

• Whole grain consumption

• Fat free or low fat dairy



Balance caloric intake from food and beverages with calories expended

• Assessing caloric intake and expenditure is very difficult

• Related behavioral messages and outcomes to assess on the energy intake side may include:– Consumption of a healthy portion size– Eating meals and few snacks– Conscious eating– Reduction of empty calories– Reduction of soft drinks

Balance caloric intake from food and beverages with calories expended

• Assessing caloric intake and expenditure is very difficult

• Related behavioral messages and outcomes to assess on the energy expenditure side include:– Reducing television time– Increased walking or biking for transportation– Small increases in activity every day– More leisure time activity

Balance caloric intake from food and beverages with calories expended:

Examples

• On a typical day, how many hours do you watch TV, view videos or work/play on the computer? (Iowa)– I don’t watch tv, etc..– Less than one hour– 1 hour/day– 2 hours/day

• I eat breakfast in the morning (UMN)– Never Sometimes Almost Always Always

Energy balance knowledge: Examples from IDEA

Question % correctYouth Parent

If someone sits all day, they doNot need to eat any calories (F) 78% 98%

Alcohol contains calories (T) 77% 99%

The sweetner used in Gatorade andOther soft drinks is healthier than the Sweetener used in sodas (F) 47% 69%

Most youth who are active need toConsume sports drinks to replace Electrolytes and minerals (F) 40% 72%

How do assess young children (less than 7 years old or 2nd grade and

younger)• Very challenging• Cannot think abstractly, limited reading and

writing skills• Will be able to perform simple knowledge

tests • Use observation assessment tools as

possible, conducted by objective observers, to assess behavior

Consider outcomes that do not rely on self report

• Plate waste (Colorado)

• Visually Monitoring Consumption

• Take versus Eat

Visually monitoring consumption

• Purpose: To develop and evaluate a simple observation method to assess student consumption of specific menu items offered in school cafeterias

• Developed the Visual Food Monitoring Form • Selected 4 menu items to estimate consumption• End of tray line, dots on trays with items (established

our denominator)• At tray return area, staff completed the VFMF

Snyder, Fee, Lytle, Hann, 1996: School Food Service Research Review, 20(2), 63-68.

Results

• Evaluation staff made 3 visits to 4 schools• Observed 1839 trays• Conducted pre-post evaluations of modified

menu items• Results:

– 81% ate at least 3/4 of higher-fat entrées– 78% ate at least 3/4 or lower-fat modifications

Can food taken be used as a proxy for food eaten?

• Purpose: If we record what foods are on students’ trays at the end of the cafeteria line, can that be used as a proxy for foods consumed?

• Rationale: Observing and recording actual consumption is labor-intensive and expensive, (approximately $40/child observed!)

Gray, Lytle, Mays, et al, 2002; JADA 102:407-409.

Can food taken be used as a proxy for food eaten?

• Validation study: n=350 5th grade students• Recorded foods present on trays at end of tray line• Recorded foods actually consumed• Used different staff to record taken and eaten data. • Results:

– Correlations of taken:eaten fruits= 0.59– Correlations of taken: eaten veggies=0.51

Gray, Lytle, Mays, et al, 2002; JADA 102:407-409.

Types of evaluation

• Outcome evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007):

“Outcome- which demonstrates changes that occur in the presence of an intervention but do not establish cause and effect conclusions.”

• Study design: Post-only with a control/comparison group

• Pre-post

Pre-post study design

• What are some of the problems with this design?


• What are some of the problems with this design?– Was the change REALLY caused by the

intervention?• Some other thing happened?• Response bias? Did they learn the “right

answer?


• What are some options? – Find a comparison/control group– Do a delayed intervention condition– Use different interventions in different

classes and use one evaluation tool that assesses behavioral objectives from both intervention

Do you need to conduct an evaluation every time you run an intervention?

Sample study design

Baselinemeasures

20 Schools

10 schools intervention

10 school waiting list

1st follow up eval

1st follow up eval

10 schools post-treament

10 school intervention

Final follow-up measures

M 1/2 I M 1/2 I M

R=randomization M=measurement period

R

Impact evaluation

• Impact evaluation as defined by USDA Food Stamp Nutrition Education Guidance (March 2007):

• Impact-which indicate how effective the intervention was in changing the target population’s attitude, awareness and/or behavior. (“knowledge, skills, attitudes, intention to act, behavior, or something else” (pg 29 FTNEPG, 3/2007)

• Assumes that a causal relationship is being tested• Study design: Randomized control/comparison

design

Establishing a causal relationship

• What is meant by a causal relationship?

A --- B

• If we introduce A, then B is more likely to occur than if A was NOT introduced– B doesn’t ALWAYS occur with A– Sometimes B occurs even without A– There is some assumption of temporality (A precedes B)

What are the best study designs to evaluate IMPACT?

• Have at least 2 time points• Clearly identify what A is • Clearly identify what B is• Anticipate potential confounding factors and

assess them• Have a control or comparison group• Have an adequate sample

Control versus Comparison group

Control groups are randomized

Comparison groups are NOT randomized

Gold standard for assessing impact

• Randomized Control Trial– Unit of assignment is unit of randomization– Clustering that naturally occurs in groups included

in the analysis plan (random effects)– Sample size calculations are based on the unit of

assignment/randomization– Baseline measures, randomization, intervention,

follow-up measures– Can be very expensive

• Randomized control trial with attention paid to unit of assignment

• What is “Unit of Assignment”?

Unit of assignment and the FNS document

• Page 7:• “Describe the unit of assignment to

intervention and control groups”• “Describe how assignment to intervention and

control groups was carried out”• “Describe how many units and individuals

were in the intervention and control groups at the start of the intervention.”

Unit of Assignment

• At what UNIT is the intervention delivered? At what UNIT does randomization into condition occur?– Out-patient nutrition counseling: Individual – Community trial: An entire community– FNS nutrition programs: A classroom, a

school, a school district, an afterschool program

What is the Unit of Assignment when you…

• Train all teachers in a school to deliver an intervention?

• Work with the school food service director at the district office to change offerings in school cafeterias?

• Recruit youth in an after school program to participate in a special nutrition education class

TEENS as an example

• 16 schools were randomized into control or treatment condition

• School is the unit of assignment and analysis

• Over 3500 students in 16 schools completed surveys

• The sample size of TEENS = 16

Unit of assignment

• Consider a classroom or a school as the unit of assignment

• One classroom/school receives an intervention

• One classroom/school does not• The mean is the aggregated scores of all kids

in the classroom/school• The analysis compares the means between

the units of assignment

• School or classroom becomes the unity of assignment and analysis

• May be able to add other elements of the intervention to the model

• The analysis model needs to include a term for the clustering of kids within the unit

• What are some potential problems in using a classroom as the unit of assignment and then comparing students in “control” and “intervention” classrooms?

Keep in mind

• Focus on specific behaviors• Attempt to focus your intervention

(content, time, type of delivery)- too much flexibility is very difficult to evaluate

• Make sure your intervention objectives, intervention strategies, and KAB measures are all lined up

Keep in mind

• Look for valid and reliable instruments in the literature

• Observation may be a cost-effective way to assess change-especially with young kids

• Keep it simple• Move beyond pre-post study designs• Look for a control/comparison group

Why evaluate?

• Fulfill an obligation or regulatory requirement• Document how your program is being

implemented• Document how your program is being

received• Document if your program is working • Provide feedback for future program planning

Types of evaluation: Qualitative and quantitative evaluation

Qualitative: – Open ended questions– Focus groups/key informant

interviews/observations– Allows for stories to be told– Allows for respondents to describe their

own experiences or opinions

Types of evaluation: Qualitative and quantitative evaluation

Quantitative: – Closed ended questions – Response can be QUANTIFIED/COUNTED– Surveys/questionnaires/observations– Allows for presentation of data in a tabular format– Allows for statistical comparison– Respondents provide responses to specific

questions

Types of evaluation

Qualitative: EXAMPLE

• Interview teachers prior to the development of a new set of classroom materials. Ask them the following question:

• “Tell me about your experiences in teaching nutrition education in the classroom. What are some things that you enjoy about teaching nutrition education to your children? What are some things that make it a challenge?”

Types of evaluation

Quantitative: EXAMPLE

• Ask teachers to fill out a survey that includes the following question:• Which of the following make teaching nutrition education

challenging? Check all that apply:– Not enough classroom time– Available materials are not culturally appropriate– Available materials are not age appropriate– No available materials– Lack of support for nutrition education from administration– Lack of cooking/food preparation facilities– Lack of interest by students– Inadequate background in nutrition

• When might you use QUALITATIVE EVALUATION?

• When might you use QUANTITATIVE EVALUATION?

Types of evaluation

• Formative

• Process

• Impact

• Outcome

Types of evaluation

• Formative: Conducting information for the intent of FORMING or developing a program or for FORMING or adapting a program

Process Evaluation

• Part of formative evaluation

• Provides documentation of what is going on in a program, how a program is being implemented and received by a variety of stakeholders

Formative evaluation: Examples

• Conducting focus groups with teachers and youth to see what types of nutrition programs that they are interested in

• Conducting a survey to document parents’ beliefs about the school food environment

• Conducting key informant interviews after the completion of a pilot program to see what teachers think needs to be changed for the next phase of programming

Process Evaluation

• May provide documentation of:– Completeness of implementation: Is all of the

intervention being implemented as planned?– Fidelity to the intervention plan: Is the intervention

being implemented as planned?– Dose of the intervention: How much of the

intervention is being received by intended parties?– Reach of the intervention: How many of the

intended participants are receiving the program? – How do stakeholders feel about the program?

Kansas State:Process Evaluation Examples

• Do you plan to spend more time on nutrition education in your classroom than you did prior to this nutrition education program?

• How was class response to the program?• What part(s) of the program did students enjoy the

most?• Would you be interested in having the program again

next year?• Do you have any requests, suggestions, ideas for

improvements, or other comments?

But isn’t it all about the environment?

Do we have a chance to positively influence behaviors when the environment is toxic?

An Ecological Model

Davison-Birch, 2001

evaluating nutrition education programs leslie a lytle, phd, rd professor division of epidemiology...

Documents

sake of knowledge slide

pancreas slide

apple slide

evaluation knowledge

knowledge example

logic model slide

food group

oneshot nutrition