MSS 905 Methods of MSS 905 Methods of Missiological ResearchMissiological Research

Survey Research

Survey Research

• The most widely used data gathering technique in the social sciences

• Can be well made and appropriate, but can also be poorly constructed and used inappropriately

• Produces information that is inherently statistical in nature

• Appropriate for self-reported beliefs or behaviors (see box 10.1, p. 273); not “why?” questions

Types of Questionnaires

• See Adler and Clark (2008), Chapter 9, p. 232-239

Good Question Writing

• Good surveys are smooth, non-threatening, unambiguous, clear

• Question writing is more an art than just a science

10 Things to Avoid10 Things to Avoid

1. Avoid jargon, slang and abbreviations• Use the language of TV and newspapers

2. Avoid ambiguity, confusion and vagueness

• “What is your income?”• Monthly? Annual? Before or after taxes?

From salary or all sources?• Accurate response categories

10 Things to Avoid10 Things to Avoid

3. Avoid emotional language and prestige bias

• Some words carry emotional baggage, “What do you think about a policy to pay murderous terrorists who threaten to steal the freedoms of peace-loving citizens?”

• Prestige bias: associating a statement with a person or group that enjoy prestige, “Most doctors say that cigarette smoke causes lung disease for those near a smoker. Do you agree?”

10 Things to Avoid10 Things to Avoid• “Do you support the president’s policy

regarding Uzbekistan?” (very few have heard of Uzbekistan and will respond on the basis of their attitude toward the president)

4. Avoid double-barreled questions• Two or more questions in the same

question• “Do you have pension and health

insurance benefits at your job?”

10 Things to Avoid10 Things to Avoid

5. Avoid leading (“loaded”) questions• Suggesting the answer in the way you

pose the question• “You go to church, don’t you?”• “Should the church spend even more

tithing income on interior decorating?”• “Should the church allocate more of

tithing income to save lost souls in our community?”

10 Things to Avoid10 Things to Avoid

6. Avoid asking questions that are beyond people’s capabilities

• People can’t always recall past details• “How did you feel about your Sunday

School teacher when you were six years old?”

• Asking church visitors, “What percentage of our tithing income should we be investing in our building fund?”

10 Things to Avoid10 Things to Avoid

7. Avoid false premises• You cannot begin with a premise which is

debatable, and then ask someone a question about it

• “Our sanctuary is already open too early on a Sunday morning. Do you want it to be open an hour earlier or an hour later?”

8. Avoid asking about future, distant intentions

• “If we opened a branch church in the north, would you attend it?”

• Stay in the present

10 Things to Avoid10 Things to Avoid

9. Avoid double negatives• If you disagree with this question,

“Students should not be required to take a comprehensive exam at the end of the course”, then you don’t agree they should not

10. Avoid overlapping or unbalanced response categories

• Categories should be mutually exclusive• “Are you working or unemployed?”

10 Things to Avoid10 Things to Avoid• Unbalanced: “Which of the Democratic

candidates do you support, Hillary Clinton or one of the others?”

Some Tips

1. Use a Likert-scale to gain responses2. I prefer a 5-pointer3. “Anchor” the two extreme positions (1

and 5) with clear opposites• “To a great extent” vs “Not at all”• “Strongly agree” vs “Strongly disagree”

4. In the question, make a clear statement to which the person needs to respond

5. Mix the response directions

Getting Good Answers

1. Be aware of social desirability bias– Respondents give the socially desirable

response (politically, ethically, etc.)

2. Avoid questions that test knowledge– Large majority cannot answer elementary

geography questions or id important political documents

– Respondents do not want to appear ignorant

% Very Topic Uneasy

Getting Good Answers

3. Make use of contingency questions – “if – then”– Saves time of those who do not qualify

4. Open-ended versus closed questions– Open is unstructured, respondents can give

any answer they wish– Closed restricts the options of the

respondent– Both have benefits and disadvantages (box

10.5)

Types of Surveys: Mail and Self-administered

• Cheap• No need for

presence of researcher

• Can be completed when convenient

• Anonymity• Avoid interviewer

bias

• Low response rates• Cannot control

conditions when questionnaire is completed

• No one to clarify• Someone else may

complete• Incomplete questions• Ill-suited for illiterate

Types of Surveys: Telephone Interviews

• 95% can be reached• Can be done long

distance as well• High response rates• Cheaper than face-to-

face, same benefits• Computer-assisted

telephone interview (CATI) & Interactive Voice Response (IVR)

• Higher cost• Limited interview

length• Not all have

telephones• Calling at

inconvenient times• Open-ended

questions are difficult to use

Types of Surveys: Face-to Face Interviews

• Highest response rates

• Permit longest format• Interviewer can

observe non-verbals• Can use visual aids• All types of questions,

complex issues, extensive probing possibilities

• Highest cost• Training, travel,

supervision, personnel deployment

• Interviewer bias potential

Types of Surveys: Web Surveys

• By 2004 75% of US homes connected to Internet

• Very fast, inexpensive• Flexible: visual or

audio inputs• Cuts out the “data

entry” step (and costs)

• International potential

• Could exclude older, less-educated, lower SES and rural people

• Sampling problems are formidable

• Privacy and security• Design issues: question

format, drop-down menus, progress indicators, move back