SURVEYSTrusting human beings to

accurately report on their own characteristics

WHY ARE THESE COMING UP IN THIS CLASS? Surveys (along with observations,

discussed later) are a very common technique for collecting nonexperimental data

Surveys are a systematic way of asking people to volunteer information about their attitudes, behaviors, opinions and beliefs

The success of survey research rests on how closely the answers that people give to survey questions matches reality (i.e., how people really think and act)

PROBLEMS TO ADDRESS The first problem that a survey researcher has

to tackle is how to design the survey so that it gets the right information. Is this survey necessary? Is the purpose of the survey to evaluate

people or programs? Can the data be obtained by other means? What level of detail is required?

The second problem is how accurate does the survey have to be? Is this a one-time survey or can the

researcher repeat the survey on different occasions and in different settings?

How will the results be used? How easy is it to do the survey?

WHEN TO DO A SURVEY The survey is an appropriate means of

gathering information under three conditions:

when the goals of the research call for quantitative and qualitative data

when the information sought is specific and familiar to the respondents and

the researcher has prior knowledge of the responses likely to emerge

TYPES OF SURVEY DATA Categorical data: numbers or words are

used to group thingsEX: gender, race, religion, food

group, or place of residence. Ordinal data: When the numbers are used

to order a list of thingsEX: The ranking of football or

basketball teams, A list of things to do, color of medal in the Olympics

Interval data: responses represent actual quantityEX: height, weight, age

MORE SPECIFIC TYPES OF SURVEY DATA There are six basic types of data that you might collect: Attitudes Opinions Beliefs Behavior Attributes (demographic characteristics)

Preferences

SETTING UP SURVEY QUESTIONS The way a question or statement is

worded and the response options offered determine the nature of the data received.

Types of survey questions include:Open-ended responseClosed response Semantic differential scales Agreement and rating scales Ranking scales Checklists

OPEN-ENDED RESPONSERespondent writes response in own wordsConsiderations for using open-ended

questions: Need to enter data by hand Develop a coding scheme for responses Content analysis?

Frequently used in exploratory studies to facilitate better understanding of a concept

Suggestion: it’s a good idea to always include an open-ended question giving the respondent the opportunity to add any additional comments.

OPEN-ENDED RESPONSE – CON’T

Advantages: Allows the respondent to answer the question

with few limitations Report more information than with discrete

answers Disadvantage:

Need qualitative methods or coding system to analyze the responses

Require subjective judgements Example:

What habits increase a person’s risk for being overweight?

Describe the pain you experience with walking?

CLOSED RESPONSEThese are the "multiple-choice" variety

where a person has to choose among several possible answers.

There are two types of closed response questions: Ordered answer choices represent points

along a continuum. Pain on a scale of 0 (none) to 10 (worst

pain ever) Unordered answer choices with each

choice is an independent answer. Examples: ethnicity and marital status

CLOSED RESPONSE – CON’T

Advantages: Quicker and easier to answer Easier to tabulate and analyze List of possible responses helps participant

understand the meaning of the question Suitable to multi-item scales designed to

provide a single scoreDisadvantages:

Do not allow participants to express their own answers

Set of answers may not be exhaustive Must be clear about selection of items, one

or as many as applicable

PARTIALLY OPEN RESPONSE

Require specific, short answers that do not encourage free expression.

Are a compromise between closed response and open response forms.

Provide an “Other” category where a person can provide additional information.

Example: blank spaces provided for the questions on racial background and persons living with you.

SEMANTIC DIFFERENTIAL QUESTIONS

Use a five to seven-point rating scale with each end of the scale having an adjective or phrase.

These adjectives, called bipolar adjectives, are direct opposites. Semantic differential scales have three

common factors: an Evaluative factor covering such dimensions as good-bad, pleasant-unpleasant and positive-negative;

A Potency factor representing the dimensions of strong-weak, hard-soft and heavy-light, and

An Activity factor with such scales as fast-slow, active-passive and excitable-calm

AGREEMENT SCALESEnd points are identified by adjectives or phrases

all of the steps may have an adjective or phrase associated with it

Example:A five point scale with steps labeled Strongly Agree, Agree, Neutral, Disagree, and Strongly Disagree (“Likert” scale)

MODES OF SURVEY ADMINISTRATION – FACE TO FACE ADVANTAGES:

Generally yields highest cooperation and lowest refusal rates

Allows for longer, more complex interviews High response quality Takes advantage of interviewer presence Multi-method data collection

DISADVANTAGES: Most costly mode of administration Longer data collection period Interviewer concerns

MODES OF SURVEY ADMINISTRATION - TELEPHONE ADVANTAGES:

Less expensive than personal interviews RDD samples of general population Shorter data collection period than personal interviews Interviewer administration (vs. mail) Better control and supervision of interviewers (vs.

personal) Better response rate than mail for list samples

DISADVANTAGES: Biased against households without telephones, unlisted

numbers Nonresponse Questionnaire constraints Difficult to administer questionnaires on sensitive or

complex topics

MODES OF SURVEY ADMINISTRATION - MAIL ADVANTAGES:

Generally lowest cost Can be administered by smaller team of

people (no field staff) Access to otherwise difficult to locate, busy

populations Respondents can look up information or

consult with others DISADVANTAGES:

Most difficult to obtain cooperation No interviewer involved in collection of data Need good sample More likely to need an incentive for

respondents Slower data collection period than telephone

MODES OF SURVEY ADMINISTRATION - INTERNET ADVANTAGES:

Lower cost (no paper, postage, mailing, data entry costs) Can reach international populations Time required for implementation reduced Complex skip patterns can be programmed Sample size can be greater

DISADVANTAGES: Approximately 77% homes have computer (2010 data) –

what about the other 23%? Representative samples difficult - cannot generate random

samples of general population Differences in capabilities of people's computers and

software for accessing Web surveys Different ISPs/line speeds limits extent of graphics that can

be used

WRITING “GOOD” SURVEY QUESTIONS

Are clear and use simple languageAre conciseAre specificAre possible to answerAre relevant to the respondentDo not use negativesAvoid bias termsHave only one part (not two parted

question)

READING LEVEL So that every respondent will

understand a question, it is important to keep the reading level at or below the average reading level of the population. Complex words may be replaced by simpler ones or ones more easily understood. If you are giving a survey to a particular group, you would want to use words that are common to the group.

RESPONSE OPTIONS Should reflect concepts you are

trying to measure, and fit with the wording of the questionAvoid simple “yes” or “no” answers

and attempt to measure intensity if possible

Mutually exclusive (select only one answer)

Exhaustive (all possible answers are listed, including other or not applicable or don’t know)

QUESTION ORDER Be attentive:

Initial questions affect answers to subsequent ones

Start with easy, salient, non-threatening questions near the end

Cluster questions addressing the same topic or concept together.

Avoid redundancy

RELIABILITY AND VALIDITY

WHAT ARE THEY? Reliability:

Expect to obtain the same information time after time. Assessed by correlation coefficient The concept of reliability can be applied to

sampling If we repeatedly draw random samples of equal

size from a population, we can expect to get the same sample values each time (plus or minus a certain amount due to sampling error).

Validity: Measures the concept intended to measure

Instrument is presented or used in the way for which it was intended

An IQ test is valid only if it is used to measure intelligence it is not valid if it used to assign individuals to groups.

A psychological test that is a valid measure of anxiety is not a valid measure of stress.

TYPES OF VALIDITY Face validity

Information collected appears to be what was expected (Face value)

A question that asked “Do you smoke?” would appear to have face validity as a measure of smoking behavior

Content validity A question adequately reflects the underlying

behavior or body of knowledge Content validity is established by having a panel of

experts evaluate and agree on the relevance of the test items

Concurrent validity One instrument or question is comparable to

another that has been shown to validly measure the same content or construct

Concurrent validity is established by correlating one question with another that has previously been validated

EVEN MORE TYPES OF VALIDITY Discriminant validity

A question or survey that is able to discriminate between group differences.

Example Depressions scales have discriminate validity if

individuals who are depressed score differently from those who are not clinically depressed.

Predictive validity A question can be used to predict behavior. Example:

Can you walk 5 blocks? Construct validity

A construct is a theoretical dimension like self-esteem that is measured by having several questions that all relate to how people view themselves.

Self-esteem does not exist by itself but is represented by how people respond to these questions.

In this example, construct validity measures the extent to which these responses can be called self-esteem.

TYPES OF RELIABILITY Test-retest reliability

Obtained by administering the same test on two or more successive occasions and then correlating the scores

Statistic that reflects reliability is correlation coefficient, higher is better

Internal consistencyObtained by correlating the scores on

several questions that pertain to the same content to the sum total of the scores

The average item-total correlation is a measure of how consistently people respond to related items on a test

TYPES OF RELIABILITY – CON’T

Stability How much variation exists in scores upon

repeated administrations of the instrument.Stable measures will reproduce the same

score on repeated administrations of the instrument.

This concept is similar to test-retest except that in test-retest situations there is no assumption that the absolute value of each persons test score will stay the same.

Considerations: Time between administration, if too close

may be remembering answers Learning effect on repeated administrations