asking the right questions: developing effective surveys
Post on 09-Feb-2016
42 Views
Preview:
DESCRIPTION
TRANSCRIPT
1
Asking the Right Questions: Developing Effective Surveys
An evaluation capacity-building trainingfrom the Tobacco Control Evaluation Center
by Robin Kipke & Travis SatterlundJune 10, 2011
2
What We’ll Be Covering10:00 Introductions, take knowledge pre-test10:40 End-use strategizing11:35 Question types11:50 Dillman principles for writing questions12:15 Lunch break 1:15 Dillman continued 1:35 Writing Questions -- Practice 2:25 Survey sequencing & construction 3:00 Field testing surveys 3:10 Learning recap, setting action plans 3:20 Complete exit survey 3:30 Optional consultation with TCEC associates
3
Training Objectives
Participants can explain and apply these concepts:
o A survey is like a conversation o The 1st step of developing a survey is
end-use strategizingo The aim of survey design is to reduce
non-response and measurement error
4
A survey is like a conversation in that…
o Is communication with a purposeo Begins with an introductiono Needs to capture respondent’s interesto Starts with easy-to-answer ?s, builds to more
substantialo Follows logical order, uses transitions to change topico Finds a delicate way to raise sensitive issueso Winds down with less consequential subjecto Indicates end with a sign off
Photo by Robert Thivierge
5
Where Survey Design Fits In
Source: Youth Media Evaluation Toolkit 2005
6
What Surveys Can Tell YouTo inform your project abouto Need for education or outreacho Community priorities or policy optionso Level of support or opposition among
stakeholder groupso Makeup of local populationso Extent of any change effectedo Satisfaction with services
7
Thinking about What Information to Collect
8
Using Reverse Logic to Develop QuestionsApply process to evaluation planning and
development of data collection instrumentso Purpose of datao Target audience for the datao How information to be usedo Likely criticso Credibility thresholdo Pieces of datao Data sources and formats
9
How It Works
o An example o Try it out – pair activity
o Think of the project you work on and how a survey could inform your efforts
o Discuss with your team what you might want a survey to tell you, how it could be used
o Work together to fill in each of the boxes of the End-use Strategizing worksheet
10
Questions –Information Types
o Attitudes—What one wants or prefers
o Beliefs—What one thinks to be trueo Behavior—What one does or has
doneo Attributes—What one is
11
Questions—Structure Types
o Open-ended—No answer choices are offered
o Closed-ended—Answer choices are offered
12
Open-Ended QuestionsExamples:o Short Answer
o How long have you lived in this apartment?
o Clarification (as part of skip pattern)o If you answered “yes” to the previous
question, please explain why... o Comments
o Please write any additional comments you may have about the potential smoke-free policy.
13
Open-Ended Questions
Drawbacks:o Respondents may find it difficult to
express their feelingso They take more time o Can yield inadequate answers without
probing, follow-up questionso Analysis is time consuming and
difficult
14
Closed-Ended QuestionsExamples:o Yes/No questions
o Have you used any tobacco products in the last 30 days? Yes No
o Multiple Choiceo How many bedrooms does this apartment have? Zero (Studio) One Two Three Other _________________
15
Closed-Ended Questionso Likert-type Scales
o To what degree would you favor or oppose a policy to make at least half of the individual units in this apartment complex non-smoking?
o How often do you attend coalition meetings?
Strongly Favor
Somewhat Favor
Neither Favor nor
Oppose
Somewhat
Oppose
Strongly Oppose
1 2 3 4 5
All of the time
Most of the time
Some of the time
Rarely Never
1 2 3 4 5
16
Closed-Ended QuestionsExamples (continued):o Ranking
o On a scale of 1 to 5, rank the issues that matter most to you with 1 being most important.
___ Health care___ Environmental protection___ Safe neighborhoods___ Quality of education___ Fair wages
17
“Survey design is all about motivating people adequately so they complete the cognitive steps necessary for answering
questions accurately and return the questionnaire.”
~ Don Dillman
18
The Dillman Principleso The aim of survey design is to minimize
measurement and non-response erroro Measurement error: poor question wording
or formatting leads to inaccurate answerso Non-response error: people who respond to
the survey are different from those who did not
Don A. Dillman. 2007. Mail and Internet Surveys: The Tailored Design Method, 2nd ed. Hoboken, NJ: John Wiley & Sons.
19
Creating Respondent Buy-in
o Motivate people to begin and complete surveyo Use introduction/cover lettero Give compelling reason to participateo Ask interesting first question
o Build trust by demonstrating competenceo Employ good survey designo Make it easy to understand what to do
20
Example Introductions1. The Bonanza County Public Health Department is
interested in finding out about how tenants feel about being around tobacco smoke. There are no right or wrong answers and they will be kept anonymous. We hope you will share your opinions with us.
2. To help protect Bonanza County multi-unit housing (MUH) residents from the dangers of secondhand smoke, the county Tobacco Prevention Program will be working with MUH owners and managers to adopt and implement a voluntary policy that prohibits smoking in…
3. The Bonanza County Tobacco Education Coalition is seeking public opinion on the effects of exposure to secondhand smoke and possible smoking policies. Your opinions are very important to us.
4. Smoke-Free Apartment Complexes – Tenant Survey
21
Managing Cognitive Energyo Respondents will devote only a limited
amount of mental effort to complete your survey
o Make survey easy to understand, navigate and respond to
o Cluster related topicso Cluster similar response typeso Weigh need for survey length vs.
complexity of questions
22
Keep Population of Interest in Mind
o What cultural characteristics might affect their ability to understand the survey?
o What is their literacy level? o What language do they feel most comfortable
with? o How much time will they be willing to spend on
taking the survey? o Is this a topic that interests them?
23
Dillman #1: Keep It Simpleo Choose simple rather than specialized
wordso Use as few words as possibleo (Sometimes these two rules conflict)
Use Rather ThanCustomers PatronsSmoke from other people’s cigarettes
Secondhand smoke
Apartments or condominiums
Multi-unit housing
People who live here Occupants of this household
24
Dillman #2: Say What You Mean
o Use complete sentences to ask questions(even when they seem self-evidently clear)
1. Please check one: Male Female
2. Age: ____
1. What is your gender? Male Female 2. How old are you? _____ years
Problematic:
Revised:
25
Dillman #3: It’s All about Timingo Avoid vague qualifierso Instead use more specific frames
o How often do you dine out at a restaurant?Problematic: Never Rarely Occasionally Regularly
Revised: Not at all A few times 1-2 times a month 3 or more times a month
26
Dillman #4: Balancing Acto To avoid bias, state both sides of the
attitudinal scale in the question stemo Would you favor or oppose a city policy to make
all parks non-smoking?
o Balance scales with equal number of positive and negative choices
o Very likely, somewhat likely, not very likely
o Neutral position is different from undecidedo Neither supportive nor unsupportive vs. don’t
know
Problematic:
27
Balancing Act ExampleGood Example:
To what extent do you agree or disagree with the statement “Smokers have a right to smoke”?
Strongly agree Somewhat agree Neither agree nor disagree Somewhat disagree Strongly disagree I don’t know
28
Dillman #5: Pick Me!o Beware of primacy effects in “mark all
that apply” listso Lists are OK for factual questions but not
attitudes or preferenceso Make each issue a yes/no or scalar
question1. Which type of organization do you represent
in the coalition? (mark all that apply) Local lead agency Service organization Law enforcement Educational institution Other ____________ I don’t represent an
organization
Ok for list
29
Problematic:Which of the following areas in the apartment complex do you think should be made non-smoking? (Mark all that apply) courtyard pool area barbeque areas balconies/patios mailboxes laundry roomsRevised:Would you like any of the following areas of your apartment complex to be made non-smoking?Courtyard yes noPool area yes noBarbeque areas yes noBalconies/patios yes noMailboxes yes noLaundry rooms yes no
30
Dillman #6: Framing the Issueo Use cognitive techniques to improve
recallo Guided imageryo Layer questions
Problematic:
When you have watched movies where actorswere smoking, did it make smoking seem more appealing to you?
31
Framing the RecallRevised:
Think about the last three movies you saw. 1. What type of movies were they? (mark all that apply) Action Comedy Drama Documentary 2.In any of those movies, did any of the characters
smoke? Yes No (go to question 4)3. Did seeing the characters smoking make cigarettes
seem more appealing to you? Yes No
32
Framing Sensitive Issues
o Save more sensitive questions towards the endo Once you’ve established a rapporto In case respondents refuse to proceed
o Soften the impact of potentially objectionable questionso Preface personal questions with more
general ones about the issue or ask about other people
33
Problematic:1. Have you ever shoplifted any tobacco products
from a store?
Revised Versions:The questions which follow are being asked to help us understand where young people get tobacco products when they are underage. We really appreciate your help and that of students all over the state who have been asked to complete this survey honestly.
V1. Have you ever taken any tobacco products from a store without paying for them?
V2. Have you ever obtained tobacco products from…a friend or relative? yes noa store without paying? yes no
34
Dillman #7: Six of One, Half Dozen of the Othero Make sure answer choices are mutually
exclusive (numbers, conceptual overlaps)Problematic: How old are you?
15-18 yrs. old 18-30 yrs. old 30-50 yrs. old 50+ yrs. old
When you tried to quit, where did you turn to for help? my doctor support group online service quitline clinic or hospital on my own
35
Dillman #8: It Doesn’t Add Upo Ensure that respondents can answer the
questiono Is it beyond the knowledge of
respondents?o Were you aware that the California Air
Resources Board has declared secondhand smoke to be a toxic air contaminant?
o If the city were to pass a tobacco retail ordinance, how much should a license cost?
o Avoid excessive specificityo How many cigarettes have you smoked within
the last 30 days?
36
It Doesn’t Add Up continuedo Use timeframes within memory
In the last year, how many tenants complained about drifting tobacco smoke?
o Avoid unnecessary calculationsHow long have you lived in this apartment?
______ months
37
Dillman #9: Don’t Be Negative!o Avoid asking respondents to say “yes” in order
to mean “no”Problematic:
In which areas of the apartment complex should people not be able to smoke?
Revisions:
v1 Which areas of the apartment complex would you like to be non-smoking?v2 In which areas of the apartment complex should smoking be prohibited?
38
Dillman #10: Over a Barrelo Avoid double-barreled questions
where two things are being asked in the same question
o Watch out for “and”o Instead collapse into illustrative
category, split into two questions, use “or”
39
Over a Barrel?Is either of these double-barreled?o If the downtown area was free of
secondhand smoke and cigarette litter, do you think you would be more or less likely to shop and attend events there?
o Do you think retailers should have to pay for a license to sell tobacco which would earmark a portion of the funds to cover the expense of enforcement?
40
Now You Try ito Write one question using each of these
formats:o Yes/noo Multiple choiceo Scalar o Open-ended
41
Survey Design Considerations
o Create a clear navigational patho Provide signposts to guide respondentso Use graphic elements as clues
START ❶ Circle the response
o Keep format visually uncluttered
42
Survey Design Considerationso Cluster like topics (e.g., knowledge,
preferences, support)o Also cluster similar response formats
(Likert scales, statements of agreement, etc.)
o Use consistent scale directions throughouto Organize choices vertically, not horizontallyo Beware of response set effecto Use pamphlet layout
43
44
Sequencing Questionso Remember a survey is like a
conversationo First engage interest, build trust, ensure
successo Move from easy-to-answer to complexo General to more specifico Less personal to more sensitiveo Balance open-ended vs. closed-endedo Save demographic questions for the end
45
Field Testing Your Surveyo Why it’s importanto Who to involveo How to go about ito What to look foro What to do with the feedback
46
TCEC: Your Resource Centero Recorded webinars & training modules:
o Online Surveys: Techniques & Tips (12/2/10)o Reducing Error: Designing Surveys that Work
(9/30/10)o Journey of a Survey (1/28/10)o Developing a Survey Instrument (3/26/09)o End-use Strategizing for Creating DCIs
(12/10/08)o Public Opinion Surveys (mini training)
o End Use Strategizing Checklisto Tips & Tools #2 on writing questionso Hundreds of survey instruments in repositoryo Individualized help from Evaluation Specialists
47
Reflecting on Learningo Surveys are more than a brainstormed
list of ???so Start with end-use strategizing processo A survey is like a conversationo Follow design principles to reduce erroro Need to motivate respondents o Manage finite amount of cognitive energy o Create navigational patho Make it easy to complete!
48
To reach us:Email: tobaccoeval@ucdavis.eduhttp://programeval.ucdavis.edu Main phone line: 530.752.9951
top related