Week 2Survey Research

Experiment

SURVEY RESEARCH

Components – Questionnaires & Interviews

Surveys are the most Popular method for information gathering in Social Research

• Why?• Versatile – appropriate for studying most issues• Efficient – appropriate for use with large

populations, relatively low cost, quick, can use many variables, can be geographically diverse

• Generalizability – appropriate for probability sampling- can generalize from sample to population.

• Omnibus studies – good for large studies like General Social Survey

Problems in Conducting Social Research

• Sampling, measurement and Design issues• Use of Survey as Design – Society is dynamic – survey

is one shot deal• Two types of Data Errors in Survey 1. poor measurement of cases that are surveyed 2. omission of cases that should have been surveyed• Why? Individuals don’t respond, poor sampling frame,

sampling error (characteristics of sample members don’t match the characteristics of population)

Survey Instrument Types

• Questionnaire – Self Administered • Interview Schedule- Questions asked face-to-

face and answers recorded by researcher.• Both must be well thought out, clear, and

integrated as a whole with questions complementing each other

Questions must be asked of many people, not just one person

Must use same instrument for all participantsQuestions must be understood in the same way by

all participants

Principles of Question Writingfor Survey research

• Avoid confusion (double negatives and double-barreled questions)

• Screening – filter questions to move participants to other parts of survey based on their answers (contingency questions)

• Bias- Avoid words that trigger bias (loaded) i.e. communist take-over like Vietnam

Instead of: in a situation like Vietnam

• Don’t lead or mislead with questions• Response range deficient – response range

must reflect full range of possible answers• Continuums – number of positive and negative

categories must be balanced• Minimize fence sitting – neutral category may

attract too many – only use neutral when you need to know how many fence-sitters you have.

• Don’t Know/No Opinion – May omit if it keeps people from expressing their real opinion

People float to no opinion categories However, forcing people to choose an answer

even if they don’t know anything about the topic is also a problem

Make sure you know why you are using these categories and have a purpose for them

Response Categories for Questions

• Can be closed ended or open-ended• Closed-ended- Answer categories are provided

by researcher. Respondents check or circle their choice – Were you abused? 1.yes 2.no

• Open-ended – Respondent provides the answer in his/her own words. Can be lengthy and disorganized. Used to get at the meaning the subjects attach to their answers – How were you abused?_____

Best Response Category for Questionnaires

• Closed Ended (Fixed-choice) (one and only one possible response for everyone who is asked the question) 1. Male 2. Female 3. Other

• Categories must be exhaustive (if you can’t list all possible choices then provide an ‘other’ category

• Categories must be mutually exclusive (Ranges of ages, incomes, years of school etc. should not overlap and should not leave out any value ranges) 1. 20-30 2. 21-40.

• Check all that apply should be used with caution and kept to a minimum.

• If a variable can’t be measured with one question then create an index or scale

( Several questions to measure the same variable) Depression can’t be measured simply by asking Are you depressed?

If some questions in an index carry more weight than others then you have a scale

Guttman Scale

• It is okay for Aliens to work where I work

• It is okay for Aliens to live in my neighborhood

• It is okay for Aliens to live next door.

• It is okay to be friends with Aliens

• It is okay to marry Aliens

• It is okay to have children with Aliens

Multi-item Index

• Several questions for one concept• Use statistical testing of index to make sure all questions

fit in index (i.e. correlation or factor analysis)• Popular Likert Index (often called scale) Ordinal measure ranging from low to high. Is the subject

liberal about ethnic differences? Aliens are bad 1. Strongly Agree, Agree, Disagree, 4. Strongly Disagree Can sum the responses for individual statements to get a

summated rating scale for each subject

Design Issues

• Pay attention to the order of questions (questions can influence each other)

• Sort the questions into thematic categories

• Start with easy and interesting questions

• Maintain consistent focus on the research problem– eliminate irrelevant questions

• Have a clear conception of the population to be sampled

• Make the Questionnaire attractive

• Indicate path through the questionnaire with words, arrows or graphics

• Mailed questionnaires must include a cover letter and stamped, self-addressed return envelop

Response Categories and other issues for Interview Schedules

• Looks like a Questionnaire, except it is read to the respondent by the interviewer

• A mix of Open-ended and Fixed Choice• Interviewer negotiates through the questionnaire path for

the subject• Interviewer must seem to be asking questions from

memory and float seamlessly along• Interviewer must make the interview seem personal and

must seem engaged and interested• Must begin with an introductory statement to draw the

respondent in• May want to send a copy of questions to the interviewee

before the interview appointment

Refine and Test Questions

• All questionnaires and interview schedules must be pre-tested on a small sample

• Focus groups can be used to formulate content of questions for surveys

Most Popular 5 Designsfor Administering Surveys

• Mailed

• Group-administered

• Phone

• In-person Interview

• Electronic

Important Issues for Administering Surveys

• Manner of Administration – Is the survey completed by the respondent or does the researcher ask the questions and record the answers

• Structure – Is the instrument highly structured or relatively unstructured

• Setting – Is the survey being answered in an individual or group setting

• Cost – What are the costs and time expectations of each of the 5 basic designs?

Mailed, Self Administered Surveys

• Respondents administer the survey themselves• Low response rates • Have to do follow up, include self-addressed and

stamped envelope, maybe token reward• Mostly structured• Lower cost• Subjects must be literate• Can’t be sure who answers it

Group-Administered Surveys

• Researcher distributes or provides access and it is administered in group setting• Individual Group members administer surveys to selves• High response rate• Mostly structured• lower cost• Works well with students, employees, members of

military or institutionalized populations• Participants often feel coerced, less likely to answer

honestly, Believe researcher is not independent of sponsoring organization

Telephone Surveys

• Respondents interviewed via phone so is not self-administered but by professionals

• May be computerized and responses automatically recorded

• Can be structured or unstructured• Cost is high• Safe, efficient, large samples, fast turnaround• Validity questionable –may not reach proper

sampling frame• Lots of incomplete responses

• Most phone surveys use random digit dialing in sampling process

• CATI (Computer-assisted Telephone Interviewing software) often used

• May require multiple call backs• 10-15 time max• People have negative bias toward telemarketing• Impersonal nature of phone interviewing (need

careful training) (Also could be computer asking questions)

In-Person Interview Surveys

• Face-to-face interaction • More, Longer and more complex questions• Administered by professional• Mostly structured• Cost very high and time consuming• High response rates• Interviewer can monitor conditions• Interviewer can probe for meaning• Can use more open-ended Qs

• Order in which questions read and answered can be controlled

• But, in spite of flexibility, all respondents should have same interview experience

• Interview should seem personalized• Computer-assisted personal interviewing software

(CAPI) can be used. Can display interview schedule on laptop. The software processes the answers and checks to make sure answers fall into allowed ranges

• Difficult to record answers and appear to be engaged

Electronic Surveys• Self-Administered via computer over Internet• Delivered via Web or Email• Mostly structured• Cost very low• Email surveys usually have to be short• Web-based surveys can be longer• Require programming expertise by researcher• Web-based can include graphics, links, pull-down menus,

pictures, audio• Responses recorded directly into researcher’s database

(eliminates data entry error & quick report generation)• Could be problems with sampling frame• Fast growing technique

IVR – (Less Popular)Interactive Voice Response System• Allows Internet surveys to be achieved

with telephone-based technology

• Respondent received automated call

• Answer questions by pressing numbers on touch-tone phone or speaking numbers that are interpreted by computerized voice recognition software

• Is a very impersonal approach

Mixed Mode Surveys

• Combinations of 5 basic survey administration designs (group,mailed, interview, phone, electronic)

• Maximizes quantity and quality of data

• Allow strengths of one design to compensate for weaknesses in another

• Maximizes likelihood of securing data from different types of respondents

Comparisons

• Mailed surveys –

worst response rate

best for large, diverse populations

inexpensive

good for sensitive issues

Best alternative – Phone survey

• In-person Survey

Best in terms of possible length and complexity

Not as good for sensitive questions

Require a lot of training

Best response rate

Expensive and Time consuming

• Electronic Surveys Not everyone has Internet connection or capacity to display

complex Web pages

Conclusions: • In-person Interview is strongest design and generally

preferable• Phone surveys have many of the advantages of in-person

interview but at much less cost. Response rates a problem.• A decision about the best survey design must take into

account the unique features and goals of the particular study.• Survey research which is mostly Quantitative can be enhanced

by combining with Qualitative research such as Case Studies, Focus Groups or Observation

Ethical Issues in Survey Research

• Fewer ethical issues than Experiment or Participant Observation

• In group-administered survey the question of participation being voluntary arises

• Primary ethical concern is Confidentiality Answers to questions could be damaging to subject if

disclosed Only research personnel should have access to information

that could link subjects identity to answers. Use ID numbers not names

Anonymity – But, if you have no identifying info linking subjects to their surveys then follow-up is not possible

EXPERIMENT IN RESEARCH

• Answers questions about the effect of a treatment or intervention on some other variable whose values can be manipulated by the researcher. If Clarinex given then symptoms reduced. 1. Clarinex given 2. placebo given 3. nothing given

• Experiment is powerful design for testing hypotheses – (A research question in testable format)

• Hypothesis is explanatory - quantitative

Hypotheses

• A specific expectation deduced from a more general theory

• We test the hypotheses not the more general theories

• Variation in one variable in the hypothesis

is supposed to predict, influence or cause variation in another variable in the hypothesis

Example

• Deterrence Theory – Punishment deters crime

• Deterrence Hypothesis – If abusers are arrested on first offense then abusers will be less likely to abuse again.

or

As arrests for abuse increase, recidivism for abuse decreases

Components of Hypotheses

• Independent variable – considered to be the causal variable. It is manipulated by researcher to effect (influence or cause) change in Dependent (another) variable

• Independent variable=Cause

• Dependent variable=Effect

CAUSATION

• A cause is an explanation for some characteristic, attitude or behavior

• Causation is often shown by experiment –

A variation in the IV results in variation in the DV

Essential Components of Experimental Design

• At least two comparison groups of subjects (usually experimental and control groups)

• Variation in the independent variable occurs before assessment of change in the dependent variable

• Random assignment to the two (or more) comparison groups

• A combination of these components gives us more confidence in the validity of causal conclusions

• Confidence is increased with the addition of two other components:

1. Control over the context of an experiment

2. Identification of the causal mechanism

Conditions of a True Experiment

• Must have at least one experimental group (subjects who receive some treatment or manipulation)

• At least one comparison group (subjects to whom the experimental group can be compared) who receive a different treatment or no treatment

• If no treatment is given to the comparison group it is called a control group

• All true experiments have a post-test (measurement of the outcome in both groups after the experimental group has received the treatment)

• Does not require a pre-test but it can be advantageous

Pre-tests provide a measure of how much the experimental and comparison groups change over time.

Popular Pre-test/Post-test Control Group Design

• Two or more groups (at least one experimental and one control)

• Pre-test and post-tests

• Random Assignment to Groups (Randomization)

Note: This is not the same thing as Random Sampling to be discussed in Sampling Lecture

• Random Assignment places pre-designated subjects into two or more groups on the basis of chance

If comparison group differs from the experimental group in anyway besides not receiving treatment or receiving a different treatment, can’t determine for sure what the unique effects of the treatments are.

• In true experiment subjects must be randomly assigned to the comparison and experimental groups.

Eliminates systematic bias• When used, the odds of a difference

between the comparison and experimental groups by chance can be calculated.

• Larger the group the better this works (over 30 subjects is best)

• Matching – subjects are assigned to the different groups based on similarity of variables such as gender, age, year in school, or other important characteristics

• researcher identifies in advance all the important variables on which to make a match of the assignment to groups (should be used in conjunction with random assignment – not instead of

Limitations of True Experiment

• Difficult to isolate the actual mechanism by which treatments have their effect

• Difficult to guarantee that control has been maintained (more likely to maintain control in lab than in field)

Quasi-Experiments

• Often testing a hypothesis with a true experimental design is not feasible

• It may be too costly, time consuming or desired setting or subjects not available

• In Quasi-experiment- subjects in groups may not be randomly assigned

Popular Quasi-Experiment Designs

• Nonequivalent Control Group – Experimental and control groups are designated before the treatment occurs, not by random assignment

• Before and After Designs – Has a pretest and posttest, but no comparison group. The subjects exposed to treatment serve as own controls by comparison of pre and post tests

1. Multiple Group Before and After Designs– several before and after comparisons are made involving the same variables but different groups and then the groups are compared

2. Repeated Measures panel Designs- the same group is observed many times (30 or more), receiving many pre and post tests. Why? To study the process by which an intervention or treatment has an impact over time.

Identifying Cause

• Quasi-Experiments only partially meet the criteria for identifying a cause.

• But, the association between the IV and DV and the Context in which change occurs can still be met in this type of experiment

Validity in Experiment

• Internal Validity 1. Selection bias – characteristics of treatment

groups differ (attrition maybe) 2. Endogenous change – subjects develop or

change during the experiment (maybe experiment effect)

3. History – something occurs during treatment which influences results

4. Contamination – one of the groups is aware of the others and is influenced by this knowledge

5. Treatment misidentification – researcher is not aware of some process that affects treatment . For example the way staff administers treatment.

Hawthorn Effect – patients perform better because feel special. In Evaluation Research, clients know research findings may affect chances of further funding

Double Blind procedures can offset this (staff doesn’t know who is getting what)

• Placebo Effect- patient improves on placebo because

they think they are getting the real thing

External Validity - Irony

• The more that assignment to treatments is randomized and all experimental conditions are controlled, the less likely it is that the research subjects and setting will be representative of the larger population.

Ethics in Experiment

• Deception – subjects may be misled, but deception is a critical component of many social experiments

If used, debriefing may be a good idea

Deception can mean harm to subject, lack of voluntary participation and informed consent

• Selective Distribution of Benefits – Can cause harm to subjects (but researcher may not know that one treatment is more effective)

Advantages and Disadvantages

Positives:

• Can study change over time

• Control, Control, Control

• Can Replicate

• Can Quantify

Negatives:

Artificiality

Major Ethical issues

Small subject groups (can’t generalize)

Sampling problems (not random)

Internal Validity – (Experiment effects)

External Validity (history)