experiments exploring the relationship (or lack thereof) between nonresponse error and measurement...

Experiments Exploring the Relationship (or Lack Thereof) Between Nonresponse

Error and Measurement Error Roger Tourangeau, SRC, JPSM Robert M. Groves, SRC, JPSM

Stanley Presser, JPSMChris Toppe, Points of Light Foundation

Courtney Kennedy, MPSMTing Yan, NORC

Nonresponse Error and Measurement Error

2

Acknowledgement

• The work reported here was supported by a grant from the National Science Foundation (SES 550385) to Roger Tourangeau and Robert Groves. We are grateful to NSF for its support and especially to Cheryl Eavey.

• The National Science Foundation is not responsible for the conclusions presented here.

• We thank Duston Pope for his contributions to the design, John LaFrance, and Mirta Galesic for their contributions to the studies discussed here study design.


3

Outline of Talk

• Models of the relationship between nonresponse and measurement error

• Experiment 1: A study of volunteers and volunteering

• Experiment 2: Topic and sponsorship in a two phase web surveys

• Two additional studies – Gain-loss framing and response propensities– An experiment with voters and nonvoters

• Conclusions


4

Models of the Nonresponse-Measurement Error Relationship

• Underlying model

p

epCovBiasM

p

XpCovBiasNR

),(

),(


5

Model 1: Independence Model • Nonresponse essentially motivational (interest in topic,

altruism, low opportunity costs)• Error largely cognitive (poor comprehensive of the items,

memory difficulties, poor estimation strategies)

M

pUp Ue

C

e


6

Model 2: Reluctance • Difficult to persuade respondents answer sloppily

(Bollinger and David, 2001; Fricker 2006)—Common cause model

M

pUp Ue e


7

Model 3: Topic Sensitivity

• Drug users and reports about drug use

),(

),(),(

epCov

XeCovandXpCov

X

pUp Ue e


8

Model 4: Topic, Sponsor Effects • Presentation of the survey request affects both who

responds and how they respond

Topic Interest

pUp Ue

Topic

e


9

“Framing” Effects • Why should the description of the topic or

sponsor effect the answers Rs give? • Three mechanisms

– Construal: Description of the survey may affect how they interpret the questions (National Crime Survey vs. National Survey of Violence Against Women—the latter may suggest a broader interpretation of “sexual assault” than the former)

– Priming: Topic, sponsor may serve as a retrieval cue, making some memories more accessible

– Perception of what is relevant or useful—Norenzayan and Schwarz (1999): Institute for Social Research and Institute for Personality Research

• Evidence for all three (Galesic and Tourangeau, 2007)


10

Experiment 1: The March of Dimes Study

• Survey topics– Survey of Giving and Volunteering– Survey of Jobs and Labor Force Activity

• Target populations– March of Dimes mailing list

• Donors, volunteers, other nominated by donors

– General adult sample from vendor list

• Sponsor– March of Dimes– University of Michigan


11

Recruitment Protocol

• Mailed paper questionnaire

• Advance letter

• Initial questionnaire packet

• Reminder postcard

• Followup questionnaire packet

All materials emphasized sponsor and topic

11


12

Response Rate by Topic and Population

12

Concern about sugging?


13

Commitment to MoD Predicts Preference for the MoD Survey

PredictorComparison Categories

Odds-Ratio of Participating in the SurveyMarch of

Dimes Volunteering

UM Volunteering

UM Labor Force

Volunteer Last Year

Yes vs. No 7.9 (4.0, 15.5)

2.5 (1.1, 5.4)

1.9(0.9, 3.8)

Events Last Year

Any Event vs. No Event

5.0(2.6, 9.8)

2.2(1.1, 4.4)

2.3 (1.2, 4.7)

Recency of Donation

w/in last 12 mos. Vs. longer ago

2.4 (1.6, 3.4)

1.8(1.4, 2.5)

1.6 (1.2, 2.2)

Last Year Donation

Some Money vs. None

1.8 (1.2, 2.8)

1.4 (1.0,1.9)

1.4 (1.0, 1.9)

Mean 4.3 2.0 1.813


14

Mixed Results on Commitment and Poor Match to Records

PredictorComparison Categories

Odds-Ratios

ParticipatingMatching

Record ReportVolunteer Last Year

Yes vs. No 7.9 (4.0, 15.5)

0.8(0.4, 1.5)

Events Last Year

Any Event vs. No Event

5.0(2.6, 9.8)

0.006(.002, .020)

Recency of Donation

w/in last 12 mos. Vs. longer ago

2.4 (1.6, 3.4)

7.1 (3.4, 14.9)

Last Year Donation

Some Money vs. None

1.8 (1.2, 2.8)

2.5 (1.206, 5.303)

Mean 4.3 2.6

14


15

Conclusions from Preliminary Analysis

• Volunteers and donors to March of Dimes tend to participate in all survey topics/sponsor combinations

• Those MoD donors with greater commitment more positive to the MoD sponorship than those with lesser commitment

• Mixed results on whether those with greater commitment tend to misreport their activities in support of MoD

15


16

Experiment 2: Two-Wave Web Survey

• Collected data from members of two web panels in on-line survey

• Wave 1 collected predictors of response propensities and answers to second wave

• Wave 1 respondents (n=3000) invited to take part in second survey about a month later

• Ostensible topic and sponsorship of second survey varied

• Examined how response rates varied as a function of attitudes toward topic and sponsor

• Also looked at effects on answers


17

Experiment 2: First Wave

• n=47,078 asked to complete University of Michigan survey on “attitudes and lifestyles”

• Members of two web panels: SSI (4.1% responded), e-rewards (14.8%); 3,000 respondents in all

• Wave 1 items included:– Questions on how often the respondent had responded to web

survey requests in the past;– Items assessing the respondents’ views about the importance of

a number of social issues, including issues related to the topic of the second questionnaire (crime and violence, gun control);

– Items assessing support for or opposition to gun control policies;– Relevant factual items (such as the presence of a gun in the

household and experiences using guns);– Rating scales on the prestige of various organizations, including

the purported sponsors of the second


18

Experiment 2: Second Wave

• n=2,020 (67.3 response rate conditional on wave 1)

• Varied three factors—sponsor, topic, and purported length – Sponsor:

• National Coalition of Gun Owners (pro-Gun) • National Coalition for Victims of Gun Violence (anti-gun)• National Center for the Study of Crime

– Topic• Crime prevention • Gun control policies

– Length (20 minutes, 5 minutes, no mention)


19

Our Hypotheses

• Attitudes toward the sponsor and topic to affect both the likelihood of cooperation and response behavior on questions relevant to the sponsors’ likely point of view.

• Similarly, we expected topic interest to affect response propensities and answers to questions in the survey

• Expected burden to lower response rates


20

Results—Response Rates

Predictors of Completion of Second Questionnaire Estimate (S.E.)

Predictors based theory

Burden: expected 5 min. 0.13* (0.06)

Burden: expected 20 min. -0.15* (0.06)

Burden: not mentioned (reference)

Sponsor: Gun Owners Coal. -0.12 (0.07)

Sponsor: Victims Coal. 0.03 (0.07)

Sponsor: Center for Study of Crime (reference)

Topic: Crime prevention 0.04 (0.05)

Topic: Gun control policies (reference)

Affinity for assigned sponsor 0.10 (0.07)

Interest in assigned topic -0.05 (0.05)

Interactions

Sponsor: Gun Owners Coal. * Affinity 0.06 (0.09)

Sponsor: Victims Coal. * Affinity -0.15 (0.10)

Topic: Crime prevention * Topic interest 0.03 (0.05)

Topic: Crime prevention * Sponsor: Gun Owners Coal. 0.07 (0.06)

Topic: Crime prevention * Sponsor: Victims Coal. -0.02 (0.06)

Predictors based on behavior as a web panel member

Panel: e-rewards 0.55*** (0.09)

Enrolled in five or more web panels 0.65*** (0.12)

Number of invites refused in last year -0.27*** (0.04)

The experimental variables seem to have little impact on either the response rates or nonresponse bias in estimates in the second survey.


21

Results — Answers • Created three composites

– Personal safety (4 items) — no effects– Gun control attitudes (11 items)

• Main effect for topic: More support for gun control when the topic was described as crime prevention than when it was described as gun control—context effect?

• Purported sponsor interacted with wave 1 att’s (next slide)– Involvement in issue of gun control (3 items)—topic effect

among pro-gun types: those below the first survey median in their support for gun control measures reported greater involvement in the issue when the topic was described as crime prevention (mean of 2.47) than when it was described as gun control (mean of 2.36)


22

Results — Gun Control Attitudes Wave 2 Gun Control Attitudes

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

1

Study of Crime Gun Owners Gun Victims

Purported Survey Sponsor

Wa

ve

2 M

ean

s

Above First Survey Median

Below First Survey Median


23

Results — Liking for the Sponsor and Sloppiness

% Straight-liners

% Inconsistent Answers

20%

30%

40%

50%

60%

1 (Lowest) 2 3 4 (Highest)Sponsor Affinity Quartiles


24

Two New Studies: Gain-Loss Framing • Gain-Loss Framing: Two-wave telephone study;

request to do second interview at end of first interviewWe really appreciate the help you’ve given us today. We are interested in how people’s views about the issues we discussed today change over time so it is important we talk to you again in a couple of weeks.

Then get either

The information you’ve already provided to us will be a lot more valuable if you complete the second interview (77.9%)

Or

Unfortunately, the information you’ve already provided to us will be much less valuable unless you complete the second interview (87.5%)


25

Voting Study

• Obtained sample of Maryland voters and non-voters• Vary mode of data collection (mail vs. telephone),

incentive ($5 vs. none), and topic (Health and Lifestyles vs. Politics, Elections, and Voting)

• Attempt to test “topic sensitivity” model (Model 3)• Reasons to think that dosage could be higher or

lower in mail


26

Voting Study—Mail Questionnaire HEALTH & LIFESTYLES

We mailed this package to a specific person at this address. It is important that only this person answer the questions. Participation in this survey is voluntary, and all of your answers will be kept completely confidential. 1. How would you rate the quality of life in your

community? Mark one box.

Excellent Good Fair Poor

2. How would you rate economic conditions in

your community? Mark one box.


3. How many days in the past week did you read a

daily newspaper? Please enter 0 for none. ___ days 4. Many people say they have less time these days

to do volunteer work. What about you, were you able to devote any time to volunteer work in the last 12 months? Mark one box.

Yes No

5. If you were selected to serve on a jury, would

you be happy to do it or would you rather not serve? Mark one box.

Yes, would serve No, would rather not serve

6. Many people are finding it difficult to make

contributions to church or charity as much as they used to. How about you? Were you able to contribute any money to church or charity in the last 12 months? Mark one box.

Yes No

7. In the past 12 months, have you worked with

others or joined an organization in your community to do something about some community problem? Mark one box.

Yes No

Here are some questions about people in general. 8. Would you say that most of the time people try to

be helpful, or that they are just looking out for themselves? Mark one box.

Try to be helpful Just looking out for themselves

9. Generally speaking, would you say that most

people can be trusted, or that you can’t be too careful in dealing with people? Mark one box.

Most people can be trusted Can’t be too careful

The next few questions are about activities that you may or may not have done in the past 12 months. 10. In the past 12 months, did you take a vacation

that lasted one week or longer? Mark one box.

Yes No

11. In the past 12 months, have you discussed a

political issue with friends or family? Mark one box.

Yes No


27

Voting Study—Mail Questionnaires POLITICS, ELECTIONS, & VOTING

We mailed this package to a specific person at this address. It is important that only this person answer the questions. Participation in this survey is voluntary, and all of your answers will be kept completely confidential. 1. How would you rate the quality of life in your

community? Mark one box.


2. How would you rate economic conditions in

your community? Mark one box.


3. How many days in the past week did you read a

daily newspaper? Please enter 0 for none. ___ days 4. Many people say they have less time these days

to do volunteer work. What about you, were you able to devote any time to volunteer work in the last 12 months? Mark one box.

Yes No

5. If you were selected to serve on a jury, would

you be happy to do it or would you rather not serve? Mark one box.

Yes, would serve No, would rather not serve

6. Many people are finding it difficult to make

contributions to church or charity as much as they used to. How about you? Were you able to contribute any money to church or charity in the last 12 months? Mark one box.

Yes No

7. In the past 12 months, have you worked with

others or joined an organization in your community to do something about some community problem? Mark one box.

Yes No

Here are some questions about people in general. 8. Would you say that most of the time people try to

be helpful, or that they are just looking out for themselves? Mark one box.

Try to be helpful Just looking out for themselves

9. Generally speaking, would you say that most

people can be trusted, or that you can’t be too careful in dealing with people? Mark one box.

Most people can be trusted Can’t be too careful

The next few questions are about activities that you may or may not have done in the past 12 months. 10. In the past 12 months, did you take a vacation

that lasted one week or longer? Mark one box.

Yes No

11. In the past 12 months, have you discussed a

political issue with friends or family? Mark one box.

Yes No


28

Conclusions • Every study is framed somehow

• Framing can effect response rates, though not necessarily nonresponse error – Topic effects on response rates found in Groves, Presser,

and Dipko (2004), Experiments 1 and 2 here, “Birding study” in Groves et al. (2006)

– But not always—three other studies in Groves et al. (2006) don’t find topic effects

– Sponsor can effect responses, but again not always

• Framing can effect answers, though effects may be subtle as in Experiment 2 here and in earlier experiment by Galesic and Tourangeau (2007)

• Only weak evidence that reluctant R’s, or those who dislike sponsor or topic more prone to measurement error

experiments exploring the relationship (or lack thereof) between nonresponse error and measurement...

Documents

nonresponse error

measurement error experiment

model slide

independence model nonresponse

low opportunity costs

norc slide

p upup ueue topic e

pupup ueue c e slide