Improving Improving response rates to response rates to

postal postal questionnaires in questionnaires in

clinical trialsclinical trialsRachel NakashRachel Nakash

PhD, BSc(Hons), MCSPPhD, BSc(Hons), MCSP

IntroductionIntroduction

Patient based outcome measures – Patient based outcome measures – assess various aspects of health assess various aspects of health from patient’s perspective.from patient’s perspective.

Usually take format of questionnaireUsually take format of questionnaire Telephone, face to face, postTelephone, face to face, post Post: least expensive, prevents Post: least expensive, prevents

interviewer bias, patient can interviewer bias, patient can respond in own timerespond in own time

Disadvantages: NON-RESPONSEDisadvantages: NON-RESPONSE

Non-response in a Non-response in a clinical trialclinical trial

Reduces effective sample size, Reduces effective sample size, introduces biasintroduces bias

What is acceptable loss to follow up?What is acceptable loss to follow up? Sackett et al (1997) ‘five and twenty’ Sackett et al (1997) ‘five and twenty’

rule of thumbrule of thumb Some journals refusing to publish trials Some journals refusing to publish trials

with losses > 20%with losses > 20% Obtaining losses to follow up < 20% in Obtaining losses to follow up < 20% in

a clinical trial takes considerable efforta clinical trial takes considerable effort

Previous researchPrevious research

Methods of improving response rates to Methods of improving response rates to postal questionnaires – mostly SURVEY postal questionnaires – mostly SURVEY researchresearch

No connection to health careNo connection to health care Most relevant Cochrane review: Edwards et Most relevant Cochrane review: Edwards et

al (2003): incentives, ‘user friendly’ and al (2003): incentives, ‘user friendly’ and shorter questionnaires improve response – shorter questionnaires improve response – but... only 20% of included trials had a but... only 20% of included trials had a health care connectionhealth care connection

Clinical trial participants ? different to Clinical trial participants ? different to survey participants – lack of literaturesurvey participants – lack of literature

PhD aims and objectivesPhD aims and objectives

1. Systematic literature review1. Systematic literature review – – refinement of previous reviews. Focussed refinement of previous reviews. Focussed just on health care literaturejust on health care literature

2. Randomised controlled trial2. Randomised controlled trial – to test – to test a method of improving response within a method of improving response within the context of an existing clinical trialthe context of an existing clinical trial

3. Qualitative study3. Qualitative study of clinical trial of clinical trial participantsparticipants

4. Analysis of characteristics4. Analysis of characteristics of of responders and non-respondersresponders and non-responders

Setting for the researchSetting for the research

Collaborative Ankle Support Trial – Collaborative Ankle Support Trial – CASTCAST A multi centre RCT funded by the NHS A multi centre RCT funded by the NHS

HTA HTA Designed to evaluate clinical and cost Designed to evaluate clinical and cost

effectiveness of 3 methods of mechanical effectiveness of 3 methods of mechanical support compared to tubigrip in severe support compared to tubigrip in severe ankle sprainankle sprain

Outcomes assessed at 4 weeks, 12 weeks Outcomes assessed at 4 weeks, 12 weeks and 9 months following injury using postal and 9 months following injury using postal questionnairesquestionnaires

CAST pilot phase was used to:CAST pilot phase was used to: Assess response ratesAssess response rates Clarify follow up proceduresClarify follow up procedures Identify aspects of response to subject to deeper Identify aspects of response to subject to deeper

investigationinvestigation

24 patients recruited onto CAST pilot24 patients recruited onto CAST pilot

Follow up Follow up pointpoint

Number Number respondingresponding

Response Response raterate

4 weeks4 weeks 1919 79%79%

12 weeks12 weeks 1616 67%67%

9 months9 months 1515 63%63%

A follow up protocol was needed to A follow up protocol was needed to chase responsechase response

Developed based on available lit and Developed based on available lit and time and resource constraints of CASTtime and resource constraints of CAST

Series of steps involving telephone Series of steps involving telephone contacts and repeat mailing of contacts and repeat mailing of questionnairequestionnaire

Time consuming (?soul destroying!)Time consuming (?soul destroying!) Response categories developed – Response categories developed –

summary of how much prompting was summary of how much prompting was required by each patient at each time required by each patient at each time pointpoint

ResponsResponse e

categorycategory

Questionnaire returnQuestionnaire return

11 Returned with no promptingReturned with no prompting

22 Returned after one telephone Returned after one telephone promptprompt

33 Returned after one telephone Returned after one telephone prompt and second copy of prompt and second copy of questionnaire sentquestionnaire sent

44 Returned after two telephone Returned after two telephone prompts and second copy of prompts and second copy of questionnaire sentquestionnaire sent

55 Not returned but core outcome Not returned but core outcome questions answered over questions answered over telephonetelephone

66 Not returned and unable to Not returned and unable to contact for core outcomescontact for core outcomes

Response categories grouped into Response categories grouped into response ‘types’response ‘types’

Response CategoryResponse Category Type of responderType of responder

1 and 21 and 2 KeenKeen

3 and 43 and 4 ReluctantReluctant

5 and 65 and 6 Non-responderNon-responder

Response types at each time pointResponse types at each time point

0

2

4

6

8

10

12

14

4 weeks 12 weeks 9 months

KeenReluctantNR

1. Systematic Literature 1. Systematic Literature ReviewReview

Cannot assume methods of improving Cannot assume methods of improving response in survey research will be response in survey research will be successful in clinical trial settingsuccessful in clinical trial setting

Refinement of previous reviewsRefinement of previous reviews Limited to ‘health care studies’ on Limited to ‘health care studies’ on

‘patient populations’‘patient populations’ More useful and relevant for health More useful and relevant for health

researchers using postal researchers using postal questionnairesquestionnaires

Results:Results:

13 trials identified evaluating 5 different 13 trials identified evaluating 5 different methods of improving responsemethods of improving response

Questionnaire length, incentives, Questionnaire length, incentives, question order, reminder strategies, question order, reminder strategies, information brochureinformation brochure

Similar methods of improving response Similar methods of improving response grouped for comparisongrouped for comparison

Random effects model meta-analysis Random effects model meta-analysis conductedconducted

ConclusionsConclusions

Intense follow up strategies and shorter Intense follow up strategies and shorter questionnaires improve response ratesquestionnaires improve response rates

The evidence is, however, limitedThe evidence is, however, limited

2. Randomised 2. Randomised Controlled TrialControlled Trial

Method of improving response Method of improving response nested into CASTnested into CAST

‘‘Trial Calendar’ – a tool to prompt Trial Calendar’ – a tool to prompt questionnaire returnquestionnaire return

Research question: Does the trial Research question: Does the trial calendar improve response rates and calendar improve response rates and reduce the need for promptingreduce the need for prompting

296 CAST participants randomly 296 CAST participants randomly allocated to receive a trial calendar or notallocated to receive a trial calendar or not

Effect of calendar assessed using chi-Effect of calendar assessed using chi-squared test at each follow up pointsquared test at each follow up point

No sig effect on response at any follow up No sig effect on response at any follow up pointpoint

Logistic regression performed to account Logistic regression performed to account for effect of covariates (eg age, sex)for effect of covariates (eg age, sex)

Still no effect of trial calendar at any time Still no effect of trial calendar at any time point (except females with calendar sig point (except females with calendar sig better at responding at 9 months – data better at responding at 9 months – data trawling!)trawling!)

Conclusions:Conclusions:

Trial calendar Trial calendar notnot an effective tool for an effective tool for improving response rates in a clinical improving response rates in a clinical trialtrial

Follow contact and prompting needs to Follow contact and prompting needs to be high profile and timely – trial be high profile and timely – trial calendar too subtlecalendar too subtle

Extra cost involved in implementing a Extra cost involved in implementing a tool such as a trial calendar not justifiedtool such as a trial calendar not justified

3. Qualitative Study3. Qualitative Study

To gain the participant’s perspective of To gain the participant’s perspective of response issues in clinical trialsresponse issues in clinical trials

Semi-structured interviews of a Semi-structured interviews of a purposive sample of CAST participantspurposive sample of CAST participants

22 participants interviewed – 22 participants interviewed – responders and non-respondersresponders and non-responders

Framework Analysis used to analyse Framework Analysis used to analyse data (similar to thematic content data (similar to thematic content analysis but deeper).analysis but deeper).

Common themes:Common themes: Reasons for response: Reasons for response:

Personal relevancePersonal relevance Agreed to take part – Compliance/Agreed to take part – Compliance/trial trial

understandingunderstanding Obliged to respond – Good service/reciprocationObliged to respond – Good service/reciprocation AltruismAltruism Important projectImportant project

Reasons for non-responseReasons for non-response Internal – due to aspects of trialInternal – due to aspects of trial External – beyond control of trialExternal – beyond control of trial Personal blame – Forgetful, lazy, disorganised…Personal blame – Forgetful, lazy, disorganised… Life eventsLife events (Full recovery, treatment preference)(Full recovery, treatment preference)

Typologies:Typologies: ‘‘Happy non-responders’Happy non-responders’ ‘‘Unhappy non-responders’Unhappy non-responders’ Serial non-responders – once a non-Serial non-responders – once a non-

responder, always a non-responderresponder, always a non-responder

Conclusions:Conclusions: Recruitment phase very important in Recruitment phase very important in

appealing to psychological processes of appealing to psychological processes of response and ensuring participant fully response and ensuring participant fully understands trial follow upunderstands trial follow up

? Convert happy non-responders? Convert happy non-responders

4. Characteristics of 4. Characteristics of responders and non-responders and non-

respondersresponders ? Any predictive characteristics of ? Any predictive characteristics of

responders and non-respondersresponders and non-responders Survey researchers have attempted to Survey researchers have attempted to

establish common personal establish common personal characteristics of responders and non-characteristics of responders and non-respondersresponders

Advantage of a clinical trial is the Advantage of a clinical trial is the detailed background information detailed background information collected at baselinecollected at baseline

Despite this there is a lack of work Despite this there is a lack of work looking specifically at CT participantslooking specifically at CT participants

555 participants recruited onto CAST 555 participants recruited onto CAST (not inc pilot)(not inc pilot)

Either responder or non-responder at Either responder or non-responder at each follow up pointeach follow up point

This info combined with CAST database This info combined with CAST database background info to allow for background info to allow for comparisons between responders and comparisons between responders and non-responders in terms of certain non-responders in terms of certain socio-demo characteristicssocio-demo characteristics

Age, sex, CAST treatment, employment Age, sex, CAST treatment, employment type, education level, recoverytype, education level, recovery

Analysis: chi-squared, logistic Analysis: chi-squared, logistic regressionregression

Conclusions:Conclusions: Age the only variable to sig effect Age the only variable to sig effect

response – older people respond betterresponse – older people respond better Females generally better at respondingFemales generally better at responding Better recovery leads to lower likelihood Better recovery leads to lower likelihood

of responseof response Active refusals to cooperate very low Active refusals to cooperate very low

compared to surveys (refusal in a trial compared to surveys (refusal in a trial usually occurs at recruitment stage)usually occurs at recruitment stage)

Non-contacts steadily increase at each Non-contacts steadily increase at each follow up (incorrect contact details, follow up (incorrect contact details, contact details change, house movers)contact details change, house movers)

Overall conclusions and Overall conclusions and recommendationsrecommendations

Consider carefully and Consider carefully and a priori a priori follow up follow up protocols to chase reluctant respondersprotocols to chase reluctant responders

Keep non-response as a result of non-contact Keep non-response as a result of non-contact to minimum – get as much contact info as to minimum – get as much contact info as possiblepossible

No evidence incentives effective so don’t No evidence incentives effective so don’t botherbother

Careful wording of written/verbal Careful wording of written/verbal instructions to appeal to theoretical issues of instructions to appeal to theoretical issues of response and understanding of follow up response and understanding of follow up commitmentcommitment

Exclude potential non contacts at baselineExclude potential non contacts at baseline

Thank you for listening!Thank you for listening!

ReferencesReferences Lamb SE, Marsh JL, Hutton JL, Nakash Lamb SE, Marsh JL, Hutton JL, Nakash

R, Cooke MW; Mechanical supports for R, Cooke MW; Mechanical supports for acute, severe ankle sprain: a pragmatic, acute, severe ankle sprain: a pragmatic, multicentre, randomised controlled multicentre, randomised controlled trial. trial. LancetLancet 2009; 373(9663): 575-81. 2009; 373(9663): 575-81.

Sackett D, Richardson W, Rosenberg W, Sackett D, Richardson W, Rosenberg W, Haynes R. Evidence-based medicine: Haynes R. Evidence-based medicine: how to practice and teach EBM. New how to practice and teach EBM. New York: Churchill Livingstone 1997.York: Churchill Livingstone 1997.

Groves R, Singer E, Corning A. Groves R, Singer E, Corning A. Leverage-Saliency theory of survey Leverage-Saliency theory of survey participation. Public Opinion participation. Public Opinion Quarterly. 2000;64:299-308.Quarterly. 2000;64:299-308.

Edwards P, Roberts I, Clarke M, Edwards P, Roberts I, Clarke M, DiGuiseppi C, Pratap S, Wentz R, et DiGuiseppi C, Pratap S, Wentz R, et al. Methods to increase response al. Methods to increase response rates to postal questionnaires. The rates to postal questionnaires. The Cochrane Database of Cochrane Database of Methodological Reviews. 2003(Issue Methodological Reviews. 2003(Issue 4):Art. No.: MR000008. DOI: 4):Art. No.: MR000008. DOI: