surveys and questionnaires research methods and data college of advancing studies brendan rapple
Post on 21-Dec-2015
216 Views
Preview:
TRANSCRIPT
Surveys and Questionnaires
Research Methods and Data
College of Advancing Studies
Brendan Rapple
Surveys Provide Important Knowledge
Economists, psychologists, health professionals, political scientists, and sociologists conduct surveys to study such topics as:
Income and expenditure patterns among households;
Voting behavior;
Effects on family life of women working outside the home, etc.
Auto manufacturers use surveys to find out how satisfied people are with their cars.
And a multitude of other topics
This presentation owes much to the American Statistical Association brochure series on survey research:
http://www.amstat.org/sections/srms/whatsurvey.html
Specific Purpose Essential
Objectives of a survey should be as
Specific
Clear-cut
Unambiguous as possible
"Men's Health Practices" is a very nebulous topic.
Better:How often do African-American males aged 40-49 visit the dentist?
Or
A survey of 50-60 year old male professors at BC about their weekly exercise habits
Steps in Conducting A Survey
Define precise purpose
Specify population
Specify appropriate sample
How to administer survey?
Draft of survey instrument
Pretest it
Revise it
Administer survey
Analyze, write it up, and communicate the results
Use results meaningfully
Decide on Mode of Data Collection
Mailed Questionnaire
Telephone
In Person Interview
Computer Questionnaire
Use Mailed/Computer Questionnaires or Not?
Perhaps “better” responses by phone -- but phone interviewing is very time-consuming.
On other hand, people tend to be more truthful with anonymous questionnaires.
Generally cheaper than one-on-one interviews.
Perhaps a mix of questionnaire and interviewing?
For All Surveys it is Essential to
explain purpose of survey very clearly and precisely.
explain any potential use the results will have for the respondents themselves.
stress voluntary nature - respondents are doing a favor to the researcher.
use letter-head – official – for a mailed/computer questionnaire.
explicitly promise confidentiality.
mention a clear expression of thanks.
Whatever Format Used . . . . . .
Important to specify a deadline for response.
Follow-up letter (e-mail) often advisable for a mailed/computer questionnaire
Always essential for PILOT STUDY or PRE-TEST
Main problems revolve about:
Question content, e.g. confusion with overall meaning of question as well as misinterpretation of individual terms or concepts
Formatting, e.g. problems with how to skip or navigate from question to question may result in missing data and frustration for both interviewers and respondents.
Population → Sample
Usually the population to be surveyed is too large.
Accordingly, one must select a smaller, representative sample.
“This sample is usually just a fraction of the population being studied.”
Samples
The quality of the sample – whether it is up-to-date and complete – is probably the dominant feature for ensuring adequate coverage of the desired population to be surveyed.
Must be representative of population.
Are the distributions of attributes, opinions, and beliefs in the sample the same as in the population?
You want to be able to make inferences about the population as a whole based on what you find to be true of the sample.
Variability
Variability is large, then sample should be large
Converse also true
2 Barrels of Apples
Barrel A (low variability) -- all apples about 5 ins. in diameter (range 5.1 to 4.8 ins.)
Barrel B (high variability) -- apples range from 2 to 6 ins. in diameter
Picking 3 apples from Barrel B might give result well below (above) average.
Still, Size of Sample Isn't Everything
Large numbers do not, in and of themselves, increase the representativeness of a sample.
Most professional survey conductors hold that a moderate sample size is enough statistically and operationally.
Representative Sample
Survey: Success of unwed teenage mothers in a specific community in raising children?
To be representative, sample must contain same proportion of unwed teenage mothers at
--each age level
--each educational level
--each socio-economic statusin the community
Population -- Sample
It is essential that you select sample in such a way that every name on the population list has an equal chance of being included in the sample.
Random Sample
Example: 500 part-time students in Advancing Studies
Sample of 20% is required
Assign each student a number from 1 to 500
Randomly select 100 numbers
Systematic Random Sampling
Example 1. 2,000 in population and you want a sample of 200, then you might
select every 10th name
Example 2. 500 part-time students in Advancing Studies Sample of 20% is required
--Randomly Select a Number from 1 to 5
--Select Every 5th Person
--002, 007, 012. 017, 022, and up to 497.
Possible Problem
Staff in govt. agency may be listed unit by unit
Each unit has 9 line-level workers and 1 supervisor.
The supervisor is the 10th person on the list.
It’s a survey of 20% -- every 5th person is selected.
If first no. selected is 1, 2, 3, or 4 then no supervisor will be selected, though they comprise 10% of population.
If first number selected is 5, then supervisors will be greatly overrepresented.
Thus, possibility of bias due to periodicity or patterns.
Stratified Sampling
Population: 2,000 (800 females; 1,200 males)
Sample required: 200
If gender is an important variable in your survey, then both females and males should be included in appropriate numbers, that is, in proportions that correspond to their presence in the population.
Strategy:
Treat both sexes as separate populations and take 10% sample from each.
OR
Make sure that all females are listed first and then take every tenth name.
Either way you will end up with 80 females and 120 males
Convenience Sampling
Could ruin an otherwise well-conceived survey.
It’s simple and cheap to select a sample of names from a phone-directory to find out which candidate people intend to vote for.
However, this sampling procedure could give incorrect results since persons without telephones or with unlisted numbers would have no chance to be reflected in the sample,
Their voting preferences might be quite different from persons who have listed telephones.
Confidentiality
Confidentiality of data supplied by respondents is of prime concern to all reputable survey organizations.
Important that individual respondents are not identified in reporting survey findings.
All of the survey’s results should be presented in totally anonymous summaries, such as statistical tables and charts.
Problems with Volunteers
Example:
TV programs asking viewers to vote.
people call who are most committed to issue.
“stuffing of ballots” by multiple calls.
Time of day is important – who’s available?
Margin of Error
Error margin of 1,000 randomly chosen individuals is said to be 3.1%.
Thus, if a random sample of 1,000 indicates that 59% will vote for Obama, the actual number could range from 55.9% to 62.1%.
Questions in a Questionnaire Should Be
Woven together
Flow smoothly
Avoid confusion
A good questionnaire forms an integrated whole.
Different Understandings
Everyone should see/understand the exact same question – no ambiguity.
But people from different backgrounds, with diverse frames of reference, may have different perceptions of the same question.
KISS PrincipleKeep It Simple, Statistician
Questions should be, as far as possible: Simple Clear, Easy to answer Personally relevant to them
Often recommended that questionnaires be written at the 5th grade reading level
Avoid Ambiguity
Do you favor governmental involvement in health care?
“What is your income?” __________________ do you mean:
weeklymonthlyannualpretaxafter taxfrom salary or from all sources
Another Example:
(A blouse manufacturer wants to ascertain what type of sleeves teenage females prefer in their blouses)
1. Do you like short sleeve blouses?YES__ NO__
2. Do you like long sleeve blouses?YES__ NO__
3. Do you like sleeveless blouses?YES__ NO__
Another Example
“Do you jog regularly?”
Problem, of course, lies with REGULARLY
Avoid jargon, slang, abbreviations
Plumbers talk about “snakes”
Psychologists about “oedipus complex”
Lawyers about “mens rea”
“NATO” usually means North Atlantic Treaty Organization
But some respondents might take it to mean:
--National Auto Tourist Organization
--Native Alaskan Trade Orbit
--North African Tea Office
Types of Questions
Open-Ended Questions:
What is your age?
What is the total turnover in your company?
Which of the four seasons do you prefer?
How would you spend a a $1,000,000 lottery win?
Disadvantages of Open-ended Questions:
Variation in answers make coding/scoring difficult
Advantages of Open-ended Questions
Do not impose researcher's opinion on respondent
Can lead to a very precise answer
Forced-Choice Questions
How many books do you read each year? Please check as appropriate below:
___ none ___ 1 to 5___ 6 to 10___ 11 to 20___ more than 20
What do you recall about the frequency of snowfall in Boston during the winter of 1994?
___ It snowed almost every day___ It snowed about once a week___ It snowed about once every two weeks___ It snowed about once a month
Importance of Wording
Take a very simple question:
How many drinks do you have each day? (Check one of the following)
____ 5 or more____ 4____ 3____ 2____ 1____ none
Better to Ask
Are your daily drinking habits reasonably consistent --i.e. do you take about the same number of alcoholic drinks each day?
_____ YES
_____ NO (if you mark "NO," skip the following question).
Checking Respondents' Consistency
Q. 4. Check one of the following:_x_ I believe that manufacturing should be increasingly
computerized, even if layoffs ensue.
___ Preserving the jobs of workers is more important than computerizing manufacturing.
Later on, the Questionnaire might ask:
Q. 30. Check one of the following:___ I support G.M.'s increasing use of robotics,
despite the resulting massive layoffs.
_x_ The government should force G.M. to curtail computerization to ensure a reduction in layoffs.
Avoid Emotional Language
“What do you think about a policy to pay murderous terrorists who threaten to steal the freedoms of peace-loving people?”
Problematic Words: murderous
freedomsstealpeace
Avoid Loaded Questions:
"Should the mayor spend even more tax money trying to keep the streets in top shape?"
"Should the mayor fix the pot-holed and dangerous streets in our city?"
What is your present marital status?
1 never married 2 married 3 divorced 4 separated 5 widowed
Arrangement Is Important
A Better Arrangement Would Be:
What is your present marital status? (circle number)
1. never married2. married3. divorced4. separated5. widowed
Danger of Overlapping
Check How Many Children are in Your Family:
a. 0-1 children
b. 1-2 children
c. 2 or more children
Avoid Prestige Bias
“Most doctors say that cigarette smoke causes lung disease for those near a smoker”. Do you agree?”
“Do you support the president’s policy regarding Zanozui?”
Avoid Double-Barreled Questions:
"Does your company have pension and health insurance benefits?"
Ranking Answers (Likert Scale Format):
Women should automatically receive three months maternity leave in your company
strongly agree partly agree disagree strongly
agree disagree
___ ___ ___ ___ ___
How often do you attend church?
Never Yearly Monthly Weekly Daily
____ ___ ___ ___ ___
Visual Analog Scales BIPOLAR scales
I see myself as:
Extremely Extremely interested in uninterested politics ______:______:______:______:______ in politics
top related