1 pm 536 program evaluation & research course outline, objectives & expectations “health...
TRANSCRIPT
1
PM 536Program Evaluation & Research
• Course Outline, Objectives & Expectations
• “Health Promotion Program Evaluation”
• Program Evaluation is Fun
• Program Evaluation is Difficult
2
Course Materials
1) Text (Everything you always wanted to know about program evaluation, but were afraid to ask)
2) Supplemental readings (Some case studies and other methodology points)
3) Lab materials
Labs
SAS/SPSS/STATA
4) Overhead copies (6 slides/page)
3
What is Evaluation
• Systematic application of research procedures that assess the conceptualization, design, implementation, and effectiveness intervention programs
Did it work
How effective was it
What does it mean
4
The Evaluation Field
Evaluation has had an explosive growth since WWII as social programs have expanded
Evaluation has emerged as a distinct field of inquiry boasting its own professional association and own journals.
Three stages of evaluation: Shadish, Cook and Leviton (1991)
1. Defining scientific rigor (Campbell & Stanley; Scriven, etc.)
2. Practicalities of evaluation in the field (Weiss, Wholey, Stake)
3. Integrate lessons to construct evaluation theory that is appropriate and rigorous (Cronbach, Rossi)
5
Evaluation can be Threatening
Acknowledge that evaluation can be “value” laden.
In reality evaluation is rarely used to terminate someone’s job
However, evaluation is used to decide on where to focus resources and this causes concern
Evaluation can be empowering
6
Why Do We Evaluate Programs
• Determine whether the program met its objectives• Force programmers to set objectives• Determine how the program worked (if it did)• Determine who was influenced by the program• Often required (Fed. Gov’t Acct. Act, GRPA)• Provides research opportunities• Plan for future programs• Interesting
7
Evaluation is Difficult
Control groups hard to createa. Hard to restrict programs (mass media)
b. People move
c. People talk
Programs are often culturally-specific
Control variables are hard to specify
Cost is often perceived to unjustifiable
8
Evaluation Framework
• Formative - pre-intervention planning
• Process - intervention implementation
• Summative - post-intervention impact analysis
9
Old Figure 1-1 Communication Campaign Evaluation FrameworkN
eeds
Ass
essm
ent/
Ide
ntif
icat
ion Formative
ResearchObservations
Depth Interviews
Focus Group Discusions
Key Informants
Community Mapping
SecondaryAnalysis
Dis
sem
inat
e F
indi
ngsDesign
Stage
Design & TestMessages
Design & TestInstruments
Design & TestMonitoringMethods
Sha
re F
indi
ngs
wit
h S
take
hold
ers
AnalysisStage
DetermineImpacts
DetermineOutcomes
Conduct Program & Effects Monitoring
Col
lect
Bas
elin
e D
ata
Col
lect
Fol
low
-up
Dat
a
Impl
emen
t Pro
gram
/Cam
paig
n
10
Figure 1-1. Program Evaluation FrameworkNeeds Identification and Assessment
Formative: Observations, Interviews, FGDs, and Community Mapping
Design: Design and Test Messages,Instruments, and Monitoring Methods
Data Collection:Collect Baseline DataMonitor Implementation of ProgramCollect Follow-up Data
Analysis: Determine Impact and Outcome
Interpretation: Share Findings with Stakeholders and Interpret Results
Disseminate Findings
11
Figure 1-2: Campaign Evaluation Framework
Formative Research Process Research Summative Research
ActivitiesFocus GroupsDepth InterviewsSecondary AnalysisParticipant Observation
Design Program Launch Program Program Ends
ActivitiesImplementation Monitoring (viewer logs, broadcast schedule)Effects Monitoring (sales data, visitation data or interviews)
ActivitiesAnalyze Survey DataKey Informant Interviews
Objectives, Understand:Barriers to ActionAppropriate LanguageConstellation of Factors
Objectives, Understand:Frequency of Broadcasts Potential Audience ReachPreliminary Data on Effects
Objectives, Understand:Level of EffectDegree of Efficiency
12
Formative - pre-intervention planning
• To develop messages
• Determine barriers to behavioral adoption
• Determine baseline benchmarks
13
Process - intervention implementation
• Determine the “reach & freq”
• Make adjustments to implementation
• Determine Impacts of program (esp. for long programs)
14
Summative - post-intervention impact analysis
1) determine impact
2) determine efficiency
3) future planning
15
Impact vs. Outcomes
• Immediate impact• Long-term outcomes• Many programs designed to change
outcomes in an overall model of behavior, but waiting until those outcomes are realized is unrealistic.
• Hence indicators are developed to assess programs for immediate impact.
16
Evaluators must be versatile
• Program changes
• Evaluation changes (usually for the worst)
• Programs and evaluations change– stakeholders change
– the priorities and resources change
• Program Stages - Evaluation should be tailored to the program stage such that more resources devoted to evaluating a pilot project or innovative program than an on-going one.
17
Chapter 2: Health Promotion Interventions (What Do We Evaluate)
• Provider Training - educational materials such as manuals, flyers
• CBD/CBO
• Community Mobilization
• Social Marketing
• Mass Media Advertising - Promotional campaigns
18
Interventions (cont.)
• Entertainment-education such as dramas, Soap operas, theater, film, music (songs, music videos, concerts)
• Interactive Health Communication
• Multi-media or community-wide Programs
19
Figure 2-1: Intervention Impact SpaceTradeoff Between Impact and Reach
Impact: Effect
on Audience
Reach: Percent of Audience Exposed to Message
One-OneCounseling
Outreach
Street Theater
Soap Opera
Mass Media PromotionIneffectiveProgramsIneffectivePrograms
EffectiveProgramsEffectivePrograms
community-wide programs
CommunityMobilization
20
Tailoring
• Objectives
• Program Stage
• Impact Models
21
Conditions Needed to Show Impact (4)
• Situational: Population can adopt behaviorPromoting behaviors that cannot be adopted is
frustratingBe sure creating a health promotion program, be sure
it is appropriate
22
Conditions for impact (cont.)
• Programmatic: Program delivers appropriate message, appropriately
1) reach
2) appeal
3) tailored
4) appropriate channels
5) messages match needs
6) coordinate supply
23
Programmatic conditions (cont)
7) logos & slogans
8) source credibility
9) source trustworthy
10) replication
11) call to action
12) continuous involvement
24
Programmatic conditions (cont.)
13) Maximize IPC
14) Feedback
15) Promise a reward
16) Use fear cautiously
25
Impact Conditions (cont.)
• Theoretical: Researchers and programmers understand the population and have an articulated theory of behavior change
• Methodological: Researchers appropriately measured population characteristics, mediators and outcomes
26
Chapter 3: Behavior Change Theory
Theories• Diffusion of Innovations• Hierarchy of Effects• Attitudes Intentions and Beliefs• Social Learning Theory (Self-Efficacy)• Health Belief Model• PRECEDE/PROCEEDPerspectives• Networks• Ecological Perspectives• Social Marketing• Framework vs. Theories
27
Theories
Diffusion of Innovations• "Diffusion is the process by which an innovation is
communicated through certain channels over time among the members of a social system” (Rogers, 1995).
• Innovativeness• Stages: Knowledge, Persuasion, Decision• Implementation, Confirmation• Characteristics of the Innovation (Radical vs.
Continuous; Perceived attributes)• Mathematical Curves and Properties
28
Figure 3-1: Typical Diffusion of Innovations Curves
0
20
40
60
80
100
1 3 5 7 9 11 13 15 17 19
Time
Pe
rce
nt
Cumulative Adoption
(S-shaped)
New Adopters(Bell-shaped)
29
Figure 3-2: Knowledge Attitude and Practices Curves
0
20
40
60
80
100
1 3 5 7 9 11 13 15 17 19
Time
Cum
ulat
ive
Perc
ent
Know ledge
Attitude
Practice
30
Figure 3-3: Campaigns/Programs Can Accelerate Behavior Change
0
20
40
60
80
100
1 3 5 7 9 11 13 15 17 19
Time
Perc
ent Campaign
No Campaign
31
More TheoriesHierarchy of Effects
• Expanded diffusion stages to 12/16 steps to behavior change that are more measurable
Posits rate of progress through steps
Provides expected rates for each step (for calculating expected effects)
32
Hierarchy Model
VeryHigh High ModerateStep 80.0 60.0 40
1 80.0 60.0 40.02 64.0 48.0 32.03 51.2 38.4 25.64 41.0 30.7 20.55 32.8 24.6 16.46 26.2 19.7 13.17 21.0 15.7 10.58 16.8 12.6 8.49 13.4 10.1 6.7
10 10.7 8.1 5.411 8.6 6.4 4.312 6.9 5.2 3.4
33
More Theories
Prochaska’s Stage Model • Specifies cognitive behavioral change stages
Attitudes Intentions Motivations and Beliefs
Social Learning Theory and Self-efficacy• Bandura (1985) individuals imitate models who
are similar to the individual and likeable, and the behavior must be demonstrated and then rewarded
34
More Theories
Health Belief Model
• HBM states that individuals change health-related behaviors if their perceptions of: Risk, Severity, Susceptibility, Cost-effectiveness, Cues)
PRECEDE
• presents categories of factors that inhibit and facilitate behavior change: PREdisposing, Enabling, Reinforcing
35
Theories, Frameworks, Theories
• Frameworks guide implementation of programs and describes steps in the process.
• Theories explain behavior change and predict human behavior- they explain how and why change occurs
• Models are particular embodiments of theories • Hypotheses are testable statements derived from
theories or models.
36
Frameworks/Theories/Models (2)
• Thus, the framework guides program steps, the logic model of what is going to happen
• The theory explains why or how you expect the behavior to occur
• The model depicts the relationships expected between key variables.
37
Perspectives
• Perspectives are points of view that direct programmers and researchers to variables or activities that affect the program, but
• They do not provide a proscribed prediction on behavior change, rather they
• Specify variables or activities that affect the outcome and should be included in the models
38
Perspectives
Ecological• Ecological Perspective (emphasizes analysis and
intervention at multiple levels: individual, interpersonal, institutional, community and public policy)
• Individual behavior is conditioned on environmental factors
• Ecological levels include interpersonal, family, community, organizational, societal, policy
• Ecological levels may vary depending on behavior
Figure 3-4: Ecological Models Specify Multilevel Influences on Behavior
Individual
Community
Society
Interpersonal
Policy
40
Perspectives (cont.)
Social Network Analysis
• Relations rather than attributes; and Structural positions are important
• People have many different networks and they can both constrain or promote health behavior
• A community of scholars (INSNA) has been formed to study social networks
• Methods and techniques can be simple or quite complex
41
A Typical Social NetworkData come from one class asking students to name others whom they think would make good leaders
42
Social Marketing
“Why can’t we sell health like we sell soap?”
•Product
•Price
•Place
•Promotion
Perspectives (cont.)
43
Supply vs. Demand Perspective
• Behavior change and diffusion have both supply and demand characteristics:
• Supply refers the components of a product or service that involve making the service product available for consumption.
• For example, the supply-side of family planning services is the provision of contraceptive counseling and devices used to regulate fertility.
44
Supply vs. Demand
• Demand refers to the characteristics of the consumer population in terms of their desires and preferences for the service or product.
• For example, the degree that a given population chooses to use a contraceptive such as pill is their demand for the pill.
• Supply and demand factors interact in a dynamic way
Figure 3-5: Supply and Demand Variables and Interventions
Provision, accessibility, and quality ofservices and products
Examples
Number of locationsto get vaccines
Politeness of trained service providers
Knowledge, attitudes,and preferencesof the public
Examples
Target audience knowsof the importance of vaccines
Audience members believethat vaccines areimportant
Supply Demand
Integrated/comprehensive programsaddress both supply and demand
46
Use & Selection of Theory
• Theory helps define behavior and specify the mechanism thought to influence it
• Thus it provides the target concepts for the programs
1. Program design2. Variable measurement3. Setting Goals/objectives4. Theory vs. program failures
47
Theory for Program Design
• Theory can be used to design programs.• For example, if studies show that
adolescents are most influenced by their peers, programs should use peers to implement the program
• Theory says that people smoke because “it’s cool” programs should be designed to show that it’s not cool
48
Theory forDefining Setting Goals & Objectives
• Goals are general statements about what is to be achieved (Decrease unwanted fertility, Reduce smoking, Increase seatbelt use)
• Objectives are operational, specifications of goals and should be SMART: (Specific, Measurable, Achievable, Realistic, Timely)
49
Hints:
1. Use strong active verbs
2. State only one purpose or aim
3. Specify a single end-product or result
4. Specify timeframe
Examples:– Reduce homelessness by 10 percent– Decrease fertility to from 3.6 to 3.1– Reduce smoking by 35 percent
50
Absolute vs. Relative Objectives
• Absolute objectives specify that everyone (or no one) will have the condition - Percentage Point Change
• Relative objective establish some proportionate improvement - Percentage Change
• Modifications to objectives are often necessary
51
Figure 3-6: Weiss’ View of Program vs. Theory Failures.
ProgramSuccess
Theory Success
Desired Effect
sets in motion which leads to
ProgramSuccess
which does not lead to
ProgramFailure
does not set in motion
which may or may not lead to
Theory Failure
TheoryNot Tested
sets in motion Desired Effect
Desired Effect
52
Figure 3-7: Conceptual Model of Program Impact.
SES
Gender
Income
Education
PsychographicCharacteristics
FamilyCharacteristics
InterpersonalContacts
ProgramExposure
Knowledge
Attitude
Practice
53
Table 3-2 Behavior Change Theories
Theory Citation
Diffus ion of Innovations Rogers, 1995
Hierarchy of Effects McGuire, 1989
Stages of Change Prochaska et al., 1992
Attitude/ Behavior Fishbein & Ajzen, 1980
PRECEDE Greene,
Health Belief Model Becker, 1974
Networks Valente, 1995
Social Marketing Kotler & Roberto, 1989
Ecological Factors McElroy, 1989
54
Developing an Impact Model
• An intervention or impact model is an attempt to translate conceptual ideas regarding the behavior under study into hypotheses on which action can be based.
• There is usually a causal hypothesis which must consist of concepts that can be measured.
• Causal Hypothesis
• Intervention Hypothesis
• Action Hypothesis
• Manipulability & Feasibility
55
Theory Selection
• One may have the impression that specification of an impact model is a scholarly monastic activity involving one or two persons who cogitate about the impact of the intervention on the outcome measures and systematically sets down the relevant hypotheses.
1. Prior experience
2. Peer Information
3. Literature Reviews
4. Meta-analysis
56
Internal vs. External Evaluators:Internal Evaluators
Advantages
1. Learn from each other
2. Can work w/ programmers (to modify objectives, gather information, get cost estimates
3. Results are cumulatively compiled and fed into future programs
Disadvantages
1. Not perceived as credible
2. Not able to maintain objectivity
57
Chapter 4:Formative & Qualitative Research
• Programs should conduct considerable formative research to design strategy.
• Formative research significantly improves the likelihood that programs will be successful
• Include audience in problem definition, message design, pre-testing of materials
• Makes impact evaluation much easier
58
Good qualitative research
Consist of four characteristics (SIFT)
1) Systematic
2) Iterative
3) Flexible and
4) Triangulate
59
Depth Interviews
• Establish rapport• Maximize homophily (usually)• Use a semi-structured guide• Conducted in 3 phases: Introduction, body of
interview, closure (tie up loose ends, solicit information)
• Generally don’t try to extract quantitative data• No magic number on how many to conduct• Segment respondents (men/women, age groups, etc.)
60
Intercept Interviews
Solicit interviews at public places
1) Approval from all relevant persons
2) pre-test the instruments and be flexible to changing them based on early results
3) vary the days, times and persons collecting the data
Randomize to minimize selectivity bias
61
Focus Group Discussions
• ... a carefully planned discussion designed to obtain perceptions on a defined area of interest in a permissive, non-threatening environment” (Krueger 1994, p. 18)
• Recruit participants (4-12) in advance
• Have them be similar
• Moderator by trained expert
• Use an agreed upon guide
62
FGDs (cont.)
• Introduction phase
• Have participants introduce one another
• Usually last about 2 hours
• Provide refreshments
• Tape record (notify participants)
63
FGDs (cont.)
• Advantages: socially oriented, flexible, cheap, quick and have face validity.
• Disadvantages can be difficult to organize and analyze; require trained moderators; and their results can be unpredictable
• Used to pilot test materials and for impact evaluation.
64
Limitations to FGDs (cont.)
Can be misleading in 3 ways:
1) the group is influenced by a dominant personality,
2) the topic of study is heavily influenced by normative pressures of conformity, or
3) the topic is not of sufficient interest for participants to generate opinions.
65
Observational Studies
• Conducted to observe behavior in its natural setting.
• Declared and undeclared
• Mystery or simulated client studies, done by
1) researcher
2) hired community persons
3) service providers themselves
66
Pilot Testing of Materials
• Crucial to pilot test materials
• Can pilot test at various stages
• Can conduct experiments to pilot test
• Use qualitative techniques
67
Professional Review of Materials
It is often advantageous to have noted experts in the field review program materials:
1) To ensure accuracy
2) To act as experts in cases of controversy
68
Simulations
• Hypothetical scenarios, usually computer generated, to determine “what if” outcomes
• Important to ask: How relevant and meaningful are assumptions and ruls in the model: Isomorphic to reality?
• Used to estimate how long for behavior change to occur or project potential benefits of program expansion
69
Ratings & Coverage Data
• Ratings: % of HH saw program
• Share: % of viewing HH saw program
• Reach and Freq. measures important to have in any program evaluation
• Gather from independent sources and compare to Rs self-reports
• Hire monitors
70
Content Analysis
• Used to code and describe programmatic materials (TV shows, ad campaigns)
• Must decide on universe, sampling scheme, coding scheme, and formulate hypotheses
• Ex: Daytime soap operas portray alcohol consumption as stress reduction technique
• Code all drinking all stress behaviors
71
Case Study ResearchUsed under 4 conditions:
1) the program is unique and unrelated to other activities,
2) the program’s implementation is complicated and data collection would be unwieldily,
3) the population being addressed by the program is small and/or unique, and
4) the program does not have a clear goal or objective.
Methods:
1) Observations
2) Key informant interviews
3) Archival
72
Qualitative Techniques for Impact Evaluation
Reasons include:
1) costs constraints,
2) program is small and not expected to bring about much change,
3) quantitative data will be impossible to acquire, and
4) as a complement to quantitative impact evaluation.
Chapter 5:Process Evaluation
• Information about how a program is implemented
• Documentation for that implementation
• Program corrections mid-implementation
• Information useful for determining impact
• Information needed for replication
Literature is thin
• Literature on process research is thin because it is a relatively new preoccupation.
• Doesn’t often provide much data as outcome is still what is interesting.
• Process research is most interesting when programs fail (implement poorly).
• Contradiction in mutation
Terms
• Fidelity
• Dose
• Compatibility
• Reach & Freq
Longer the Program the Greater the Need for Process Evaluation
1) the resources devoted to the program can change,
2) the implementing agencies can change,
3) unanticipated problems with intervention delivery may (and do) occur, and
4) unanticipated problems may occur in the implementation of the evaluation.
Process Evaluation is Overlooked Because:
1. Assume uniform implementation (particularly for mass media stuff)
2. Scientists ignore issues of program & message creation
3. Assume program reception is uniform
4. Programs are often multi faceted so hard to track
78
5-1: Process Evaluation can be Conducted at 5 Points in the HP process
NeedIdentified
AudienceReception
ProgramCreated
ProgramImplemented
BehaviorChange
ServiceDelivery
Supplier
4
5
32
1
Six Points of Program Monitoring
1. Program Created
2. Program Disseminated
3. Audience Reception (different from outcome research in 3 ways)
4. Audience Effects
5. Product Supplied
6. Interorganizational relations
Uses of Process Research
1) Alert Programmers to Problems w/ Dissem.
Make mid-term corrections
2) Interference from other campaigns
3) Measure contamination
4) Replication
Challenge: Collecting and reporting data in a timely manner
81
Four Sources of Data for Monitoring (use checklists)
1. Direct observation (careful of reactivity a.k.a. Hawthorne effect)
2. Service record data (A few items of data gathered consistently and reliably are generally better for monitoring purposes than a more comprehensive set of information of doubtful reliability.)
3. Service provider data
4. Program participant data
82
Monitoring Delivery of Services
Is the actual service provided similar to that which is expected to occur?
1. No treatment or not enough treatment2. Treatment is unstandardized or uncontrolled
a. mode of delivery impairs treatmentb. treatment requires delivery system that is too
sophisticated
3. Varies across target populations - Unstandardized (e.g., head start, DARE)
How to Conduct Process Evaluation
• Can require trained researchers or audience members trained for such purposes
• Inspection of primary documents (memos, contracts, etc.)
• Interviews
A Note on the Distinction Between Research Types