Do Monetary Incentives IncreaseBusiness Survey Response Rates?

Results from a Large Scale Experiment

Paul Biemer, RTI International and University of North Carolina

Christopher Ellis, Angela Pitts and Kimberly Robbins

RTI International

Outline of the Presentation

Essential information about the establishment survey

Design of the incentive experiment

Results

Conclusions

The O*NET Survey

Occupation Information Network Survey (O*NET)

Provides descriptive ratings on 800+ U.S. occupations

Target population is all U.S. employees in these 800+ occupations

Continuing survey since 1997

Mode of interview

Telephone contact of establishments

Point of contact (POC) is usually Human Resources Manager

PAPI or Internet questionnaires completed by selected employees

The O*NET Survey (cont’d)

Two-stage sample design

Establishments and employees within establishments in selected occupations

~24,000 establishments/year (79% RR)

~36,000 employee respondents/year (69% RR)

O*NET Incentive System

All incentives are prepaid (prior to receipt of questionnaires)

For the establishment: O*NET toolkit (personnel job description aid)

For POC: Desk clock

Framed certificate of appreciation

$20 money order (to be tested)

For employees $10 cash

POC Incentive Experiment: Motivation

Concern that POC incentives not commensurate with POC tasks

Some evidence of diminished POC effort during employee nonresponse follow-up phase

Establishment and employee nonresponse rates were low for some occupations

POC does not receive monetary incentive although employees do ($10)

Overview of Data Collection Protocol

Step 1:

Verification Call to Receptionist

Step 4:

Recruiting Call to POC

Send Information Package

Step 2:

Screening Call to the Point of Contact (POC)

Step 5:

Sampling Call to POC

Step 8:

7-Day Follow-up Call to POC

Step 11:

31-Day Follow-up Callto POC

Step 10:

21-Day Follow-up Callto POC

Step 9:

Send Thank You/Reminder Postcards

Step 12:

Send Replacement Questionnaires

Step 13:

45-Day Follow-up Callto POC

Step 1:

Step 4:

Step 3:

Step 2:

Step 5:

Step 8:

Step 11:

Step 10:

Step 9:

Step 12:

Step 13:

Step 1:

Send Questionnaire Pacakge

Step 6:

Step 7:

Send Toolkit

Step 7:

Overview of Data Collection Protocol

Step 1:

Verification Call to Receptionist

Step 4:

Recruiting Call to POC

Send Information Package

Step 2:

Screening Call to the Point of Contact (POC)

Step 5:

Sampling Call to POC

Step 8:

7-Day Follow-up Call to POC

Step 11:

31-Day Follow-up Callto POC

Step 10:

21-Day Follow-up Callto POC

Step 9:

Send Thank You/Reminder Postcards

Step 12:

Send Replacement Questionnaires

Step 13:

45-Day Follow-up Callto POC

Step 1:

Step 4:

Step 3:

Step 2:

Step 5:

Step 8:

Step 11:

Step 10:

Step 9:

Step 12:

Step 13:

Step 1:

Send Questionnaire Pacakge

Step 6:

Step 7:

Send Toolkit

Step 7:

Incentives sent here

Brochure sent here

Cover of the POC Incentive Brochure

FRONTBACK

Inside of the POC Incentive Brochure

Experimental Design

Split sample design (75/25 split)

Incentive group: POCs offered $20 money order in addition to clock and framed certificate

Control group: POCs offered only clock and framed certificate

Money orders could be made out to POC, the business or a favorite charity

Interviewers (called business liaisons or BLs) handled both incentive and non-incentive cases

Sample Allocations to the Control and Incentive Groups

Control Incentive

Establishments 2,624 7,874

Employees 7,694 22,309

Results

Unweighted Response Rates (%)

Control Trt Diff P-value

Establishment 62.0 61.3 1.7 0.12

Employee 73.6 72.7 0.8 0.45

Other Results

No effect on response rates by

size of establishment

industry grouping

urban or rural regions

occupation of employee

Some evidence of a negative effect for some sectors

Cumulative Employee Response Rates for by Week of Data Collection

Average Employee RR for Waves 1.5, 1.6, 1.7, 1.8, 2.4, 3.3, 3.4, 3.5, 4.3, 4.4, 5.2, 5.3, 6.11, 6.1

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

1 3 5 7 9 11 13 15 17 19

Week of Data Collection

Per

cen

tag

e

Control

Treatment

Summary of Results

No evidence of an effect on establishment cooperation rates

Weak evidence of an effect on employee cooperation rates Both negative and positive effects detected for small

subgroups

Overall, no significant difference

No cost advantage Follow-up costs were the same

Speed of response was the same

Incentive costs were not offset by efficiency gains

Interpreting the Findings

No establishment-level effect

$20 incentive viewed as trivial by POCs Framed certificate and clock could have much larger

perceived value

Decision to participate shared by POC and company

POC cannot accept money in many establishments

No effect at the employee-level

POC has limited ability to affect employee response

$20 ineffective for increasing follow-up intensity