show me the money? effects of preincentives, differential

University of Wisconsin Survey Center

© 2012. Materials may not be reproduced without permission of the author.

SHOW Me the Money?Effects of Preincentives, Differential Incentives,

and Envelope Messaging in an ABS Mail Survey

Kristen Cyffka1,2, Jennifer Dykema1, John Stevenson1, Kelly Elver1, Karen Jaques1

1University of Wisconsin Survey Center2Department of Statistics, University of Wisconsin-Madison

AAPORMay 18, 2012


Background

• RDD telephone surveys are no longer a viable mode for many studies• Decline in last 2 decades in response rates• Inadequate coverage due to cell phones

• Researchers are increasingly looking to alternative designs• Return to mail surveys due to the DSF and ABS

• Effective ways to increase response rates in mail surveys (Edwards et al 2002, Church 1993):• More interesting vs. less interesting questionnaire• Recorded delivery vs. standard• Monetary incentive vs. no incentive

2


Research Questions

• How much of an increase will a $5 pre-incentive yield over a $2 pre-incentive?

• Can we attract initial nonresponders with a second incentive?

• What effect will messages on the front of the envelope have on response rates?

• What is the most cost-effective incentive package we can give to increase response rates?

3


Study: Survey of the Health of Wisconsin (SHOW)

4

SHOW 1 SHOW 2Year 2011 2012Contacts Initial packet

Reminder postcardSecond packet

Initial packetReminder postcardSecond packet

Sampled cases 2,608 2,616Completed surveys 1,566 1,440Response rate (RR1) 66.9% 66.8%

Experimental conditionsPre-incentiveSecond incentiveEnvelope message



5

SHOW 1 SHOW 2Experimental conditions

Pre-incentive

Second incentive

Envelope message



6


Pre-incentive - $2- $5

Second incentive

Envelope message



7



Second incentive - $0- $2

Envelope message



8




Envelope message - No message- Monetary message



9



- $2





10



- $2


- $0- $5




11



- $2


- $0- $5


- No message- Monetary message- Health message


Envelope Message: SHOW 1

12


Response Rates: Envelope Messaging

13

57% 57%

11% 9%

0%

10%

20%

30%

40%

50%

60%

70%

No Message Money

Res

pons

e R

ate

SHOW 1

After 2nd SAQAfter 1st SAQ

67% 66%

p=0.600


Envelope Message: SHOW 2

14


Response Rates: Envelope Messaging

15

54% 52% 51%

14% 14% 15%

0%

10%

20%

30%

40%

50%

60%

70%

No Message Health Money

Res

pons

e R

ate

SHOW 269% 66%

p=0.437

66%

57% 57%

11% 9%

0%

10%

20%

30%

40%

50%

60%

70%

No Message Money

Res

pons

e R

ate

SHOW 1

67% 66%

p=0.60


Response Rates: Pre-Incentive

16

53%

61%11%

9%

0%

10%

20%

30%

40%

50%

60%

70%

$2 $5

Res

pons

e R

ate

SHOW 1

64%70%

p=0.005


Response Rates: After All Incentives

17

53% 51%

11%18%

0%

10%

20%

30%

40%

50%

60%

70%

Res

pons

e R

ate

SHOW 2

64%69%

p=0.015

$2$0

$2$5

Pre:2nd:

54%

60%

52%

62%11%

9%

10%

9%

0%

10%

20%

30%

40%

50%

60%

70%

Res

pons

e R

ate

SHOW 1

p=0.031

$2$0

$5$0

$5$2

$2$2

Pre:2nd:

70%

63%69%

66%


Response Rates: After All Incentives

18

53% 51%

11%18%

0%

10%

20%

30%

40%

50%

60%

70%

Res

pons

e R

ate

SHOW 2

64% 69%

p=0.015

$2$0

$2$5

Pre:2nd:

54% 52%

60% 62%11%

10%

9% 9%

0%

10%

20%

30%

40%

50%

60%

70%

Res

pons

e R

ate

SHOW 1

66%63%

p2nd=0.734

69% 70%

$2$0

$2$2

$5$2

$5$0

Pre:2nd:

pAll =0.030


Response Rates: Message and Incentive

19

67% 70%

62%

70%64%

68%

0%

10%

20%

30%

40%

50%

60%

70%

SHOW 2

No MessageHealthMoney

p=0.141

$2$0

$2$5

Pre:2nd:


Costs per Complete by Incentive

20

SHOW 1 Pre:2nd:

$2$0

$2$2

$5$0

$5$2

Total Number of Completes 383 371 403 423

Response Rates 65.7% 63.1% 69.0% 70.6%Cost Per Complete $23.31 $26.05 $26.67 $26.28

SHOW 2 Pre:2nd:

$2$0

$2$5

Total Number of Completes 669 741Response Rates 64.4% 69.3%Cost Per Complete $25.49 $28.37



21

SHOW 1 Pre:2nd:

$2$0

$2$2

$5$0

$5$2



SHOW 2 Pre:2nd:

$2$0

$2$5




22

SHOW 1 Pre:2nd:

$2$0

$2$2

$5$0

$5$2



SHOW 2 Pre:2nd:

$2$0

$2$5




23

SHOW 1 Pre:2nd:

$2$0

$2$2

$5$0

$5$2



SHOW 2 Pre:2nd:

$2$0

$2$5




24

SHOW 1 Pre:2nd:

$2$0

$2$2

$5$0

$5$2



SHOW 2 Pre:2nd:

$2$0

$2$5




25

SHOW 1 Pre:2nd:

$2$0

$2$2

$5$0

$5$2



SHOW 2 Pre:2nd:

$2$0

$2$5



Other Analyses in SHOW 1

Data quality• Item-missing data: no difference between treatments

Differential response• No difference between treatments in:

• Gender• Age• Race• Education• Employment• County

26


Discussion

• Envelope messaging• Effects on response rates• Future study

• Incentives• Effects on response rates

• Pre-incentives• Second incentives

• Implications on cost

27


Please visit us at:www.uwsc.wisc.edu

Thank You!

For copies of this presentation or more information, contact:

Kristen [email protected]

28


Response Rates: After All Treatments

29

53% 56%61% 59%

53% 51%

62% 61%10%13% 6% 11%

10% 10%

9% 8%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Message None Message None Message None Message None

$2 $2 $5 $5 $2 $2 $5 $5

$0 $0 $0 $0 $2 $2 $2 $2

SHOW 1


Response Rates: After All Treatments

30

55%50%

54% 53% 54%47%

12%12% 10%

17% 17%20%

0%

10%

20%

30%

40%

50%

60%

70%

80%

None Health Money None Health Money

$2 $2 $2 $2 $2 $2

$0 $0 $0 $5 $5 $5

SHOW 2


Response in Initial Nonresponders

31

0%

5%

10%

15%

20%

25%

30%

35%

40%

$0 $5

Res

pons

e R

ate

SHOW 2: Second Incentive

23%

38%

p=0.000


Cost analysis – What we included

We included:• Mailings• Postage, printing of all materials• Stuffing• Incentives• Data entry

We did not include:• Survey development and layout of SAQ• Programming of data entry instrument• Project management• Sample management• Data delivery

32


Discussion, limitations, and future directions

33

• So ... only $5 vs $2 had an effect on outcomes examined• Insight as to why the

• Additional incentive was not effective• Envelope message was not effective

• Future directions for current data• Mini- meta-analysis of effects of $5 versus $2• Look at responses to key survey questions• Use data from the sample file to try and look at

nonresponse bias• Ideas for future experimentation with ABS mail surveys?


Response rate calculation

34

MDE Code Description N rr_v11100 Complete 1580 12111 Other person refusal 1 02112 Known respondent refusal 14 02113 Blank q'naire, implicit refusal 8 02310 Deceased (including USPS: Deceased) 8 02320 Physically or mentally unable 1 03230 Refused by addressee 1 03300 Undeliverable 1 .3311 USPS: Attempted - Addressee not known 74 03313 No such address 5 .3314 Vacant 32 .3315 Not delivered as addressed 44 03320 Moved - left no address 5 04700 Ineligible Age 181 .4701 Non county resident 35 .Blank Noncontact 618 0Total 2608


Ineligible and undeliverable rates

35

11.7

4.6

119.8

10.49.2

1110.1

0

2

4

6

8

10

12

14

$5-$2-Env(Y)$5-$2-Env(N)$5-$0-Env(Y)$5-$0-Env(N)$2-$2-Env(Y)$2-$2-Env(N)$2-$0-Env(Y)$2-$0-Env(N)


Effects on data quality

• Item missing data• Weak satisficing• Responses to select survey questions• Demographics/nonresponse bias

• There were NO differences due to incentives for ANY of these measures of data quality

36


Data Quality: Item-Missing Data

37

Did treatments make a difference in the number of questions completed?

• Used a subset of 102 questions asked of everyone

91.7 92.2 92.5 91.3 91.9 92.0

0

20

40

60

80

100

Message No msg $5 pre $2 pre $2 extra No extra

Num

ber o

f Que

stio

ns C

ompl

eted Pre‐Incentive

p=0.006Second Incentive

p>0.05Messagep>0.05

show me the money? effects of preincentives, differential

Documents