examining item nonresponse through paradata and respondent ... · ana lucía córdova cazar rebecca...

Survey Research and MethodologyUNL-GALLUP Research Center


Examining item nonresponse throughparadata and respondent

characteristics: A multilevel approach

Examining item nonresponse throughparadata and respondent

characteristics: A multilevel approach

Ana Lucía Córdova CazarRebecca Powell

May 19, 2013AAPOR 68th Annual Conference

This material is based upon work supported by the National ScienceFoundation under Grant No. SES - 1132015.

Any opinions, findings, and conclusions or recommendations expressed in thismaterial are those of the author(s) and do not necessarily reflect the views of the

National Science Foundation.



Theoretical Background:Item nonresponse

• Item nonresponse:– The respondent fails to give an answer to one or more questions

asked• Determinants of item nonresponse:

– Cognitive difficulties• clarity of question and response options• question complexity

– Psychological and social costs• sensitivity to question topic

– Motivation• Interest in question topic

• Item nonresponse occurs when cognitive orpsychological/social costs exceed motivation to respond(Stocke 2006, Beatty and Hermann, 2002)

22



Theoretical Background: Paradata

• Paradata– Data about the data collection process (Couper, 1998)– Can contribute to enhance data accuracy by allowing the

examination of factors associated to difficulties in theresponse process

– Example: response times• Too short: Speeding• Too long: Cognitive difficulties

33



Focus of this research

• Do characteristics such as time spent filling out aquestionnaire, questionnaire’s topic, respondent’s levelof interest in the survey, and respondent’s characteristics(age and education), influence whether the respondentcompletes the entire questionnaire or not?

44



Hypotheses

1. The more interested the respondent is in the surveytopic, the lower the item nonresponse rate

2. Response times are associated with item nonresponserates

3. The higher the cognitive ability, the lower the itemnonresponse rate

55



Data

• This study will use paradata and survey responsescollected from the internet component of the GallupPanel in its last five waves (April 2012, June 2012, July2012, September 2012, and December 2012)

• It includes all panel participants who have at leastresponded to one of the waves– 18360 participants

66



Method (1)

Outcome variable• Item nonresponse rate: Number of failed questions divided by

the total number of questions respondents should haveanswered

Predictor variables• Average time spent per question• Questionnaire topic

– Media Usage– World Affairs and Elections– Social Issues and Elections– The Year in Review– Mobile Retailing

• The respondent’s level of interest in the survey– Question asked by Gallup – scale 1 to 5

• Cognitive ability:– Measured through respondent’s age and education

77



Method (2): Distribution of nonresponse rate

88



Method (3): Analytic Rationale

• Questionnaires are nested within respondents– Members of the panel were repeatedly asked to answer a web

questionnaire

• Correlation that exists in item nonresponse rates from eachrespondent across the different questionnaires needs to betaken into account

• Outcome variable (nonresponse rate) is zero-inflated– Two stage analysis:

1) Predicting perfect item response vs. item nonresponse(Binary distribution)

2) Predicting degree / severity of non-response (Lognormal distribution)

• Multilevel modeling is used to account for this dependencyamong responses coming from the same respondents(Hoffman, 2012)

99



Findings First Model(Logistic model predicting the probability of

having at least one missed question)

• Significant random intercept variance– Persons’ commonalities in the propensity to fail to respond to the

entire questionnaire

• 3.7% of the total variance is between persons– Effects at the respondent level are small

• Once we control for type of questionnaire, 12.9% of theproportion of total variance is between persons

• So what explains whether a person misses at least onequestion?

1010



Probability of Item Nonresponse by Topic(College Education, Average Age, Average Time per Question and

Average Interest in Survey)

1111

0.56

0.41

0.22

0.12

0.70

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Media Usage World Affairs andElections

Social Issues andElections

The Year in Review Mobile Retailing



Probability of Nonresponse by Response Time(College education, Average Age, Average Interest in Survey)

1212

Average time spent per question0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1 5 9 13 17

Media UsageWorld Affairs and ElectionsSocial Issues and ElectionsThe Year in ReviewMobile Retailing



Probability of Item Nonresponse by Age(College education, Average Time per Question and

Interest in Survey)

1313

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

20 30 40 50 60 70

Media Usage

World Affairs andElectionsSocial Issues andElectionsThe Year in Review

Mobile Retailing

Age (years)



Findings Second Model:Predicting the degree of item nonresponse

• Significant random intercept variance after controlling forquestionnaire topic– There are effects at the respondent level

• 7.27% of the variance in degree of item nonresponse isdue to effects at the respondent level

• So what explains degree of item non response?– Difficult answer

1414



Variable Estimate PIntercept 0.09696 <.0001World Affairs -0.04658 <.0001Social Issues -0.05628 <.0001The Year in Review -0.06118 <.0001Mobile Retailing 0.01877 <.0001Education: High school -0.01336 0.0015Education: Some college -0.01495 0.0002Education: College grad -0.01745 <.0001Education: Postgrad -0.01764 <.0001Gender (Men) 0.00662 <.0001Income (Average) -0.00109 <.0001Age (Centered) 0.00088 <.0001Person mean average time 0.00204 <.0001Within person average time 0.00468 <.0001Person mean interest in survey -0.01218 <.0001Within person interest in survey -0.02108 <.0001

Estimated coefficients for degree of item nonresponse rate

1515



Conclusions

• Questionnaire topic seems to play a very important rolein motivating full response to survey questions– Lower probability of item nonresponse in day-to-day topics

• Less cognitive burden?– Higher probability of item nonresponse in topics dealing with

technology purchasing behaviors• More social and cognitive costs?

• Questionnaire topic moderates the effect of age– As age increases, probability of item nonresponse increases, but

effect varies depending on topic

• Questionnaire topic moderates the effect of responsetimes– Depending on the topic, longer response times lead to

lower/higher probability of item nonresponse

1616



Conclusions

• Effects at the respondents levels only explain a smallpart of item nonresponse rates– The main source of the effect seems to be at the questionnaire

level

• More research on how to better explain “degree” of itemnonresponse

1717



Questions?Comments?

Ideas?

Thank you!

Ana Lucía Córdova CazarSRAM – Gallup Research CenterUniversity of Nebraska – Lincoln

[email protected]

1818



Variable Estimate p

Intercept 0.5493 <.0001worldaffairs -0.6484 <.0001socialissues -1.5495 <.0001yearinreview -2.1942 <.0001mobileret 0.4915 <.0001age57 0.02717 <.0001worldaffairs*age57 -0.0112 <.0001socialissues*age57 -0.0124 <.0001yearinreview*age57 -0.0137 <.0001mobileret*age57 -0.0149 <.0001worldaffairs*pm_av_time6 0.01654 0.1529socialissues*pm_av_time6 0.0399 0.0017yearinreview*pm_av_time6 -0.0116 0.436mobileret*pm_av_time6 0.09126 <.0001highschool -0.3156 0.0012somecollege -0.2992 0.0013collegegrad -0.2893 0.0018postgrad -0.2626 0.0046pm_av_time6 -0.0105 0.2083wp_av_time 0.04199 <.0001pm_interest4 -0.2616 <.0001wp_interest -0.2539 <.0001

Logistic regression model

1919



2020



2121



2222



2323

examining item nonresponse through paradata and respondent ... · ana lucía córdova cazar rebecca...

Documents