![Page 1: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/1.jpg)
Response Rates and Results of the Advance Letter Experiment
2004 RRFSS Workshop
Toronto, June 23, 2004
David A. Northrup, Renée Elsbett-Koeppen and Andrea Noack
ISR, York University
![Page 2: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/2.jpg)
Outline
general comments on response rates how response rates are calculated a very brief history of response rates what strategies have/are being put in
place to deal with declining response rates
![Page 3: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/3.jpg)
Outline (continued)
response rates and RRFSS what did it take to get the 62% rate for
2003 RRFSSnumber of callsrefusal conversions
results of the advance letter experiment
![Page 4: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/4.jpg)
Calculating Response Rates
Completions / estimate of number of eligible households (HH) eligible HHs include completions, refusals,
callbacks, and a % of the “never answered”
ISR method same as BRFSS, aka “CASRO 3”
RRFSS 2003 = 62%, exclude callbacks = 71%
![Page 5: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/5.jpg)
Response Rates for American Election Study
0%
20%
40%
60%
80%
100%
1952 1956 1960 1968 1972 1976 1980 1984
![Page 6: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/6.jpg)
Response Rates for BRFSS
0%
20%
40%
60%
80%
1994 1995 1996 1997 1998 1999 2000 2001 2002
medianmean
![Page 7: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/7.jpg)
Strategies for Improving Response Rates
interviewer training increase call attempts “convert” refusalsuse advance letterspayments (as a lottery, to completers, to the
whole sample)
![Page 8: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/8.jpg)
Data Collection at ISR for RRFSS Response Rates
minimum number of 14 calls (more when there is reason to think extra calls might obtain a completion) limitation of one month sample release costs about 3 to 7 points on response rate
at least one attempt to convert almost all refusals
![Page 9: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/9.jpg)
390,106
![Page 10: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/10.jpg)
RRFSS: Fun with Numbers 1 (2003 Data)
number of calls: 390,106percent of interviews completed first call:
21number of interviews completed on the
10th or subsequent calls: 3,158number of interviews completed after a
refusal: 2,678
![Page 11: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/11.jpg)
RRFSS: Fun with Numbers 2 (2003 Data)
average number of calls per completed interview: 4.65
most calls made for a single completion: 33 (for two (different) interviews)
response rate if 10 plus calls and refusal conversions are dropped: 48.2%
number of complaints about interviewer calling registered at ISR: 13
![Page 12: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/12.jpg)
Characteristics of Refusers:2003 RRFSS Data
variable standard converted
mean age 47.99 53.86
education: > than high school (%)
university (%)
16.7
47.8
23.0
39.2
% employed 61.7 52.4
% saying health fair or poor 11.8 16.5
% doctor told high blood pressure 22.3 27.1
% smoke 100 cigarettes 52.5 55.0
# of cases: standard = 24,700, converted = 2,640all differences significant
![Page 13: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/13.jpg)
Characteristics of Easy and Hard to Reach: 2003 RRFSS Data
variable easy to reach
hard to reach
mean age 50.43 43.50
education: > than high school (%)
university (%)
18.6
45.6
12.5
52.8
% employed 54.8 73.6
% saying health fair or poor 13.8 8.1
% doctor told high blood pressure 25.0 16.8
% smoke 100 cigarettes 52.7 50.6# of cases: easy = 17,000, hard = 3,150all differences significant
![Page 14: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/14.jpg)
Letter Experiment: 1
six Health Units participated (Durham, London, Grey Bruce, Halton, Waterloo, Sudbury)
test two versions of letter: ISR and HU needed to work with our monthly target and
wanted to acknowledge random variation in response rates per HU per month
used sample “replicates” to implement experiment
![Page 15: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/15.jpg)
Letter Experiment: 2
Month one: replicate 1, ISR letter; replicate 2, HU letter; replicate 3, and 4 (when used), control group changed presentation in months 2 and 3 copy of letter at the end of this set of
handouts exactly the same text, different letterhead,
signature & envelope Except Halton
![Page 16: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/16.jpg)
Letter Experiment: 3
survey introduction exactly the same except one additional sentence “Recently, we sent a letter to your
household about an important research project.”
questions about the letter the same except Durham
![Page 17: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/17.jpg)
Why the Letter Might Improve Response Rates to RDD Surveys
reduces the possibility that the telephone call catches people by surprise
increases legitimacy of research project in the eye of the potential respondents
demonstrates social value improves the confidence of the interviewer
![Page 18: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/18.jpg)
Why Advance Letters Might Not Improve Response Rates to
RDD Surveys
letter does not reach, or is not read by, respondent
ceiling effects survey topic & subpopulations they give “timid” participants a chance to
prepare to say “no”
![Page 19: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/19.jpg)
Response Rates for Months 1 & 2 of the Experiment
66%62% 63%
70%
30%
50%
70%
letter no letter ISR letter HU letter
p value =.035 (for letter (1,200) versus no letter (1,345))p value =.025 (ISR (600) versus HU (600))
![Page 20: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/20.jpg)
Response Rates Months 1 & 2: All Six Health Units
0%
20%
40%
60%
80%
Durham London GreyBruce
Halton Waterloo Sudbury
none
ISR
HU
See next slide for numbers & p values
![Page 21: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/21.jpg)
RR by HU and Treatment treatment, RR (%) P value
HU
none
ISR
HU
none/ISR
none/ HU
ISR/ HU
Durham 58.9 63.6 65.3 .422 .275 .803
London 57.8 64.5 69.2 .254 .048 .475
Grey 70.5 73.2 78.7 .618 .128 .367
Halton 57.5 57.0 68.7 .925 .053 .087
Waterloo 64.7 60.2 65.8 .441 .851 .416
Sudbury 69.8 63.2 71.4 .255 .776 .213
Number of cases per HU: ISR = 50, HU = 50, none = 100
![Page 22: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/22.jpg)
Mean Calls per Completion
mean # of calls P value
None
ISR
HU
none/ISR
none/ HU
ISR/ HU
Durham 5.89 5.69 5.57 .825 .721 .905
London 6.41 6.06 4.43 .751 .049 .060
Grey 4.36 4.58 4.69 .763 .622 .896
Halton 5.18 6.42 6.53 .157 .124 .935
Waterloo 4.82 6.42 4.38 .049 .510 .047
Sudbury 5.95 5.14 5.42 .445 .597 .809
Number of cases per HU: ISR = 50, HU = 50, none = 100
![Page 23: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/23.jpg)
Mean Calls per Completion by Letter Status
0
1
2
3
4
5
6
# of calls
letter no letter ISR HU not seeletter
sawletter
Letter/no letter p = .890, ISR/HU p = .230, not see/saw p = .001
![Page 24: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/24.jpg)
% of First Call Attempts Leading to Completions & Refusals
0%
5%
10%
15%
letter no letter ISR letter HU letter
comps
ref
letter/no letter p =.204, ISR/HU p =.008
![Page 25: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/25.jpg)
At the Start of the Interview
35%
27%
38%
0%
20%
40%
R told Interviewerthey saw letter
R asked aboutletter
"standard" surveyintroduction
![Page 26: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/26.jpg)
Awareness of Letter
Variable (based on 602 cases) total
R indicated saw letter at intro 30
R indicated letter came to house 21
total respondents aware of letter 51
personally read the letter 40
got more info (web site, 1-800) 1
letter made a lot of difference to decision to participate
26
![Page 27: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/27.jpg)
Data Characteristics
Variable ISR(n=300)
HU(n=300)
none(n=620)
P
year of birth 1955 1954 1954 .820
male (%) 58 56 57 .800
employed (%) 59 59 60 .627
health excellent 22 22 22 .951
smoked at least 100 cigarettes (%)
50 56 56 .197
![Page 28: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/28.jpg)
Costs: Month One
cost of materials: $314; staff cost: $1,919 total: $2,233per case cost: $3.62buys 72 interviews or 12 per HUneed to estimate savings from making
fewer calls, and making fewer refusal conversion calls
![Page 29: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/29.jpg)
Conclusions HU letter (seems to) increase response and
warrants consideration as a tool to improve RRFSS response rates
affect on variable distributions minimal, but small sample size limits scope of examination
social-political distance between respondent and sender probably matters
letters may have value other than just increasing response rates
![Page 30: Response Rates and Results of the Advance Letter Experiment 2004 RRFSS Workshop Toronto, June 23, 2004 David A. Northrup, Renée Elsbett-Koeppen and Andrea](https://reader035.vdocuments.net/reader035/viewer/2022062516/56649e395503460f94b2a0c9/html5/thumbnails/30.jpg)
Questions