“Maximising Return from Cohorts” Project: Prevention of Attrition Findings

Cara Booker, Ph.D.MRC SPHSUOctober 12, 2009

Overview

• Systematic Review

• Survey of Retention Among MRC Cohort Studies• Preliminary Findings

• Future Analyses

Systematic Review

• Objectives• To determine the effectiveness of

retention strategies in improving response rates in prospective population-based cohort studies.

• To identify possible characteristics (i.e. locality, age, etc.) of studies that may have affected the retention of cohort members.

Definition, Inclusion & Exclusion Criteria

• Population-based cohort study:• “Any well-defined population defined by geographic

boundaries, membership or occupation”. Szklo M. Population-based cohort studies.

Epidemiologic Reviews. 1998;20(1):81-90.

• Exclusion Criteria:• Clinical or non-clinical

trials• Non-population-based

cohort studies • Cohorts with record

linkage as the only method of follow-up

• Studies which only looked at effectiveness of tracking methods

• Inclusion Criteria:• At least one wave of

follow-up data collection in which the participant was personally contacted by the study

• Retention rates were reported

• Some description of the retention methods was available

Review Methods

• Electronic Database Search• 13 databases searched• 5 search terms

• Manual searches of bibliographies

• Internet and website searches for technical reports and internal documents

• Contact with Principal Investigators and other study personnel for access to unpublished manuscripts and grey literature

Flowchart

Potentially relevant prospective population-based cohort studies (n=913)

Excluded: Trials & non-population-based cohort studies (n=12,596)

Papers retrieved for further evaluation (n=290)

Excluded from review (n=265)No evaluation (n=245)

Reviews (n=13) Tracing methods only (n=7)

28 Studies in 32 Papers

Total papers found from search (n=17,210)

Excluded: Recruitment (n=3,701)

Excluded (n=623)No information on strategies (n=577)No information on follow-up (x=46)

Working Papers /Handbooks/Users Guide (n=5)Unpublished papers (n=1)Book chapter (n=1)

Results

• 28 Studies were identified• Countries

• 15 in the US• 5 in the UK• 3 in Canada• 2 in Sweden• 1 in Norway, Spain & Australia

• 11 were randomized trials • Response rates ranges from 34% - 98%

Retention Strategies by Data Collection Method

Evaluated Retention Method

Data Collection Method

PostalQuestionnaire

s

Face-to-FaceInterviews

Telephone Interviews

MixedMethods

Incentives 4 2 2 2

Monetary 1 1 1 2

Non-Monetary 2 1

Mixed 1 1

Reminder Methods 9

Repeat Questionnaires/Visits/Calls

8 1 2

Alternative Methods of Data Collection

5 3 2

Other 2

Length of Questionnaire 1

Postal Method 1

Incentives

• Randomized Trials (n=8)• Incentives associated with increased response rates

• Average increase ranged from 4% to 27%• Type of incentive (i.e. gift, money, cash card) did not appear to

have an effect on response rate• Use of cash cards and percentage of checks cashed varied• Previous round responders had higher rates than non-responders

• Non-randomized studies (n=2)• Comparing monetary to gifts, monetary had increased response• Non-monetary and informational incentives compared against

each other produced no difference in response

• Greater value of incentive great response, less money spent overall

Reminder Letters/Calls

• Randomized Trials (n=1)• Higher response among participants with second

reminder of phone call• Non-randomized studies (n=8)

• Reminder letters increased response• However compared to a second questionnaire there

was lower response from the reminder letters• Overall, use of multiple reminders or methods of data

collection increased response

• Average initial response rate was 61%• The average increase of response rate via

• Reminder letters was 17%• Reminder calls was 11%

Multiple Posting/Calls/Visits

• 8 studies posted multiple questionnaires, 1 had multiple visits and 2 with multiple calls• Posting additional questionnaires increased

response by an average of 15%• Costs increase with subsequent posting

• Multiple visits to schools increased response by 34%

• More call attempts appeared to increase response rates

Alternative Data Collection Method

• 10 studies offered alternative methods• Postal studies that offered telephone interviews increased

response by an average of 5%• Face-to-face interviews which offered alternatives

increased response by 25%• Two studies started in a central location, one started with

clinic visits• Telephone studies that sent postal questionnaires

increased response by 1%• Costs are higher for postal vs. in-school

questionnaires (costs of mail & project coordinator) and telephone interviews are more expensive than post (due to personnel costs, and tracing costs)

Other

• 2 Randomized Trials• Length of questionnaire

• Received higher response with short form of questionnaire (not significant)

• Postal method• Randomized type of envelope and

certified mail vs. UPS– Certified mail had better response– Handwritten envelopes had better response

than other types of envelopes

Study Characteristics

• Different designs, sample sizes, reporting etc make it difficult to address our secondary aims. Broad pattern of:• Lowest response rates in US-based studies• Higher response rates in studies started in the 1980s

and 1990s• In general, participants who were ethnic minorities,

younger and of lower SES had lower response regardless of retention method

• Higher education associated with earlier response• Postal studies with males only had higher response

rates than mixed and female only studies• In mixed gender studies females had higher response

rates in 8 of 11 studies.• Proportion of males increased with increasing contacts

Conclusions

• Incentives were the only strategy that had a clear and positive association with response rates

• Multiple mailings of questionnaires and reminder letters also appear to increase response

• Alternative data collection methods had minimal increases for postal questionnaire studies, but large increases for in-person interview studies

• Use of multiple strategies increases overall response rate

Discussion/Limitations

• Very few cohort studies with analyses of retention methods

• Reporting of attrition is not standardized• Often have some mention in various manuscripts

detailing characteristics of attriters, which may vary by wave of data collection

• More often don’t have technical reports, or detailed manuscript about the strategies used the characteristics of the attriters across the study

• Calculation of response rates is also difficult due to eligibility criteria, definition of denominator and booster samples

Recommendations

• Pilot studies or sub-samples to evaluate retention strategies

• Focus more expensive strategies to non-responders• Most initial responders will do so without the need for

incentives or other more vigorous reminders

• Explore effectiveness and ethical issues associated with internet searches and use of social networking sites in relation to participation not just for tracking

Survey of Retention Among MRC Cohort Studies

• Objectives• To examine what retention methods are

associated with reductions in attrition• To explore what other study design features,

if any, influence minimisation of attrition

• Sample• Drawn from studies which are housed in one

of the 12 MRC Population Health Research Network (PHSRN) units• Thirty-eight studies were identified

Methods of Survey

• Data collection• Conducted between December 2007 and August 2008 • Reminder methods

• 1 Reminder telephone call• 3 Reminder emails

• Questionnaire Contents• General study design issues• Recruitment • Consent issues • Compensation/Incentives/Reimbursement • Tracking of study members• Attrition

“Maximising the return from cohort studies: The prevention of

attrition and efficient analysis” Study Questionnaire

Funded by MRC Population Health Sciences Research Network http://www.sphsu.mrc.ac.uk/sitepage.php?page=cohort_studies

Table 1: Response rates 1993 - 2002

50

55

60

65

70

75

80

85

1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

HSE

BCS

FRS

BSA

NALS

GHS

average

All Eligible 13,304

Excluded 1799

Refused 3001

Enrolled 8,504

Baseline Participants: 8504

1st Follow-up Participants: 7535 (89%)

Deceased 275

Dropped Out 1799

Successfully Traced 8119

Retention, Tracking, Tracing...

• Retention Strategies• Used to increase response at time of data collection

• Tracking/Tracing Strategies• Used to find participants between methods of data

collection

• Ambiguous Strategies• Often used between data collection but often used to

increase loyalty & not specifically to track/trace

General Study Design

• 24 studies responded (75% response rate)• 7 were dropped from analyses • One study had two distinct samples,

increasing analysed studies to 25• 4 settings

• Data presented here is from 18 studies

Data Collection

• How studies collected data• Face-to-face: Over 90% • Post: 70%• Telephone: 40%• Internet: 10%

• Studies collected between four and eight different types of data

0

10

20

30

40

50

60

70

80

90

100

Pe

rce

nt

Method

Figure A3. Method of Data Collection

Retention Strategies

• 6 studies allocated funding for retention

• 10 studies offered incentives• 1 study evaluated use of incentives

• Unconditional gift voucher found to be the most successful

• Retention Methods• 50% of the studies used 2-6 different retention methods• Most commonly used

• Email, newsletters, newspapers/magazines• GPs/CCs, schools• Key Leaders/Gatekeepers, administrative/supervisory

bodies, parent assessments

Tracking Methods

• Tracking/Tracking Methods• Minimum number of methods used was 1 & 14

was maximum• Most used:

• Change of address cards, GP records, routinely registered events, stable address of close relative

• Ambiguous methods• 80% provided findings• 40% provided holiday cards & 20% provided

birthday cards

Strategies: Numbers, Barriers & Successes

• Successful Retention Strategies• Family/friend contact, change of address card and

telephone/directory assistance

• Barriers to retention • Residential mobility, disinterest in issues covered by study,

incorrect address and people too busy to join study.

Figure S1. Number of Retention Strategies Used Per Wave

0

1

2

3

4

5

1 2 3 4 5 6 7 8 16 17

Few

est

Str

ate

gie

s

0

1

2

3

4

1 3 4 5 6 8 9 10 15 16 21 24

Most

Str

ate

gie

s

FewestStrategies Usedin a Wave

Most StrategiesUsed in a Wave

Further Analyses

• Analyze the impact of retention methods and different data collection methods on retention• What retention methods are the most significant in

reducing attrition?• What is the impact on attrition when more invasive data

collection methods are used (i.e. Vene-puncture, tissue collection, etc.)

• Use a multi-level modelling approach• Waves nested within studies

• Wave variables include: retention methods, data collection methods

• Study level variables include: setting, average age of sample at baseline, gender of sample, year study started

Future Analytic Methods

• Multivariate modelling• Combine methods into larger categories

• i.e. Radio + Internet = Media

• Examine the effects of different sample population characteristics on subsequent retention rates• Data collection types• Age• Gender• Study setting

Thank you!