Survey Research

AD140 College of Advancing Studies

16 March, 2005 Brendan Rapple

This presentation owes much to the American Statistical Association brochure series on survey research:

http://www.amstat.org/sections/srms/whatsurvey.html

Little Cards on Restaurant Tables:Was the service good?

Telephone: Is president doing a good job?The most popular programs on public radio?

Census:How many bathrooms do you have?

Magazines:How is your romantic life?

Market Researchers: Brand X or Brand Y?How many drinks last week?

Surveys Provide Important Knowledge

Economists, psychologists, health professionals, political scientists, and sociologists conduct surveys to study such topics as:

Income and expenditure patterns among households; Roots of ethnic or racial prejudice; Implications of health problems on people’s lives; Voting behavior; Effects on family life of women working outside the home, etc.

More Examples

Auto manufacturers use surveys to find out how satisfied people are with their cars.

The U.S. Bureau of the Census conducts a survey every month to obtain information on employment and unemployment in the nation.

The National Center for Health Statistics sponsors an annual survey to determine how much people are spending for different types of medical care.

Local transportation authorities conduct surveys to acquire information on people's commuting and travel habits.

Surveys are used to ascertain what sort of people use our national parks and other recreation facilities.

Paul White (1998-09-02)

Specific Purpose Essential

Objectives of a survey should be as

Specific

Clear-cut

Unambiguous as possible

"Men's Health Practices" is a very nebulous topic.

Better: How often do African-American males aged 40-49

visit the dentist?

Objectives

Not sufficient that survey is intended to find out about “the living conditions of old people”.

One would need to know • the definition of an old person • is there a particular population of old people? • living conditions - house? rooms? furniture etc? • income? expenditure?

Paul White (1998-09-02)

The initial statement should explain • why the survey is being done • exactly what questions it intents to cover • the kinds of results expected and the type of analysis needed • how the information is to be used • the degree of accuracy required

Always remember that no amount of manipulation of the data can overcome the resultant defects of imprecise objectives.

Paul White (1998-09-02)

Steps in Conducting A Survey

Define precise purpose

Specify population

Specify appropriate sample

How to administer survey?

Draft of survey instrument

Pretest it

Revise it

Administer survey to sample

Analyze, write it up, and communicate the results

Use results meaningfully

Decide on Mode of Data Collection

Mail

Telephone

In Person Interview

Computer

Pre-Testing

Critical for identifying questionnaire problems.

Main problems revolve about:

Question content, e.g. confusion with overall meaning of question as well as misinterpretation of individual terms or concepts

Formatting, e.g. problems with how to skip or navigate from question to question may result in missing data and frustration for both interviewers and respondents.

Pre-Tests and Pilot Surveys

Pre-Tests provide guidance on

1. The adequacy of the sample. 2. The non-response rate to be expected. 3. The suitability of the method of data collection: observation, mail

questionnaires, interviewers, versus cost, accuracy, response rates. 4. The adequacy of the questionnaire – very important. 5. The efficiency of the instructions and general briefing of interviewers.

(Pilots also provide good interviewer training.)6. The probable cost and duration of the main survey and of its various

stages.

Population to be Surveyed

Individuals

Larger units, e.g. families

Sometimes Difficult to Specify Population

e.g. a survey of female faculty members at BC: do we include part-time profs?

Samples

Must be representative of population.

Are the distributions of attributes, opinions, and beliefs in the sample the same as in the population?

You want to be able to make inferences about the population as a whole based on what you find to be true of the sample.

Often difficult to find representative sample.

Always a danger of sampling error or bias.

Quality of Sample Important

The quality of the sample – whether it is up-to-date and complete – is probably the dominant feature for ensuring adequate coverage of the desired population to be surveyed.

Variability

Variability is large, then sample should be large

Converse also true

2 Barrels of Apples

Barrel A (low variability) -- all apples about 3 ins. in diameter (range 3.1 to 2.9 ins.)

Barrel B (high variability) -- apples range from 2 to 5 ins. in diameter

Picking 3 apples from Barrel B might give result well below (above) average.

Size of Sample Isn't Everything

Large numbers do not, in and of themselves, increase the representativeness of a sample.

Most professional survey conductors hold that a moderate sample size is enough statistically and operationally.

Whole Population and Sample Sometimes the Same

Example: Small companies in the paper recycling industry in LA.

Unit of Analysis: a company

You define "small company" as a private co. with turnover of less than $2,000,000 per annum

Research shows that there are 34 relevant companies

Therefore, manageable to use ALL in sample

N.B.Results will only relate to small paper recycling companies in LA -- difficult to generalize about other

types of company in other parts of the country.

Population Often Not Feasible Due to Size

Welfare Recipients

Mentally Ill

Prison Inmates

But still essential to survey a REPRESENTATIVE sample.

Early Studies of Gay Men

Sampling frame composed of men, patients of therapists participating in research

But most gay men were not patients of therapists

Representative Sample

EXAMPLE--Success of unwed teenage mothers in raising children?

To be representative, sample must contain same proportion of unwed teenage mothers at

--each age level

--each educational level

--each socio-economic statusin community

Lists May Be Very Exclusive

EXAMPLEUndocumented Aliens

--We know that many live in LA

--But relying on Govt. lists may be useless

Suppose You Have a “Population,”e.g.

all registered voters in your county

all Mercedes owners in the state

all soccer players in your school district who drive green mopeds

THEN YOU SELECT SAMPLE IN SUCH A WAY THAT EVERY NAME ON THE LIST HAS AN EQUAL CHANCE OF BEING INCLUDED IN THE SAMPLE

Random Sample

Random =Purposeful & methodical

Not reflect biases of researcher

Everyone has equal & independent chance of being selected

Random Sample

Once selected it cannot be chosen again (like lottery winners)

Example: 500 part-time students in Advancing Studies

Sample of 20% is required

Assign each student a number from 1 to 500

Randomly select 100 numbers (by computer or by table of random numbers)

Systematic Random Sampling

Example 1. 2,000 in sampling frame and you want a sample of 200, then you might

select every 10th name

Example 2. 500 part-time students in Advancing Studies Sample of 20% is required

--Randomly Select a Number from 1 to 5

--Select Every 5th Person

--002, 007, 012. 017, 022, and up to 497.

Possible Problem:

Staff in govt. agency may be listed unit by unit

Each unit has 9 line-level workers and 1 supervisor.

The supervisor is the 10th person on the list.

It’s a survey of 20% -- every 5th person is selected.

If first no. selected is 1, 2, 3, or 4 then no supervisor will be selected, though they comprise 10% of population.

If first number selected is 5, then supervisors will be greatly overrepresented.

Thus, possibility of bias due to periodicity or patterns.

Stratified Sampling

Population: 2,000 (800 females; 1,200 males)

Sample required: 200

If gender is an important variable in your survey, then both females and males should be included in appropriate numbers, that is, in proportions that correspond to their presence in the population.

Strategy:

Treat both sexes as separate populations and take 10% sample from each.

OR

Make sure that all females are listed first and then take every tenth name.

Either way you will end up with 80 females and 120 males

Convenience Sampling

Could ruin an otherwise well-conceived survey.

It’s simple and cheap to select a sample of names from a phone-directory to find out which candidate people intend to vote for.

However, this sampling procedure could give incorrect results since persons without telephones or with unlisted numbers would have no chance to be reflected in the sample,

Their voting preferences might be quite different from persons who have listed telephones.

Cluster Sampling

Often difficult to list all members of target population and select the sample from among them

e.g. 1) Population of American high school students

2) Population of U.S. postal delivery workers

3) Adult population of Atlanta

Possible Strategies

Population of American high school students

choose 50 schools randomly from entire list and include all students in those schools in the sample.

Population of U.S. postal delivery workers

choose 100 post offices randomly from all 50 States and include

all deliverers in those post offices in the sample.

Adult population of Atlanta

Randomly choose sample of 50 blocks from a city map and then poll all adults living on those blocks.

Potential Problems

Confidentiality Confidentiality of data supplied by respondents is of prime

concern to all reputable survey organizations.

Strategies: Using only number codes to link the respondent to a questionnaire. Refusing to divulge names and addresses of survey respondents. Omitting the names and addresses of survey respondents from

computer files used for analysis Presenting statistical tabulations by broad enough categories so that

individual respondents cannot be singled out.

Reporting

Important that individual respondents are not identified in reporting survey findings.

All of the survey’s results should be presented in totally anonymous summaries, such as statistical tables and charts.

Volunteers

Volunteers usually have characteristics that differentiate them from the larger population.

The fact that they volunteer makes them different from persons in the population who do not volunteer.

They tend (but not in all circumstances): to be better educated have higher social class to be more intelligent• have greater need for social approval

Volunteers (Cont.)

They tend to be • more sociable• more unconventional (especially when volunteering for

studies of sex behavior)• less authoritarian• less conforming

• Jews more likely than Protestants.

• Protestants more likely than Catholics.

• Females more likely than males.

Volunteers -- Example

TV programs asking viewers to vote.

people call who are most committed to issue.

“stuffing of ballots” by multiple calls.

Time of day is important – who’s available?

Margin of Error

Error margin of 1,000 randomly chosen individuals is said to be 3.1%.

Thus, if a random sample of 1,000 indicates that 59% will vote for Bush, the actual number could range from 55.9% to 62.1%.

In the 1984 election, the Gallup Survey (using 3,456 individual responses) missed by just +0.2 of 1% when it predicted that Ronald Reagan would win by 59.0%.