how pragmatic or explanatory is the randomized, controlled trial

Selby et al. BMC Medical Research Methodology 2012, 12:101http://www.biomedcentral.com/1471-2288/12/101

RESEARCH ARTICLE Open Access

How pragmatic or explanatory is the randomized,controlled trial? The application andenhancement of the PRECIS tool to theevaluation of a smoking cessation trialPeter Selby1,2,3*, Gerald Brosky4, Paul I Oh5,6, Vincent Raymond7 and Suzanne Ranger8

Abstract

Background: Numerous explanatory randomized trials support the efficacy of chronic disease interventions,including smoking cessation treatments. However, there is often inadequate adoption of these interventions forvarious reasons, one being the limitation of generalizability of the explanatory studies in real-world settings.Randomized controlled trials can be rated as more explanatory versus pragmatic along 10 dimensions. Pragmaticrandomized clinical trials generate more realistic estimates of effectiveness with greater relevance to clinical practiceand for health resource allocation decisions. However, there is no clear method to scale each dimension during thetrial design phase to ensure that the design matches the intended purpose of the study.

Methods: We designed a pragmatic, randomized, controlled study to maximize external validity by addressingseveral barriers to smoking cessation therapy in ambulatory care. We analyzed our design and methods using therecently published ‘Pragmatic–Explanatory Continuum Indicatory Summary (PRECIS)’ tool, a qualitative method toassess trial design across 10 domains. We added a 20-point numerical rating scale and a modified Delphi process toimprove consensus in rating these domains.

Results: After two rounds of review, there was consensus on all 10 domains of study design. No single domain wasscored as either fully pragmatic or fully explanatory; but overall, the study scored high on pragmatism.

Conclusions: This addition to the PRECIS tool may assist other trial designers working with interdisciplinaryco-investigators to rate their study design while building consensus.

Keywords: Clinical trial, Explanatory, Pragmatic, Smoking cessation, PRECIS, Varenicline, Bupropion,Nicotine replacement therapy

BackgroundSchwartz and Lellouch [1] first used the terms ‘pragmatic’to describe trials designed to help choose between optionsfor therapy, and ‘explanatory’ to describe trials designed totest causal research hypotheses – for example, whether aparticular intervention causes a specific biological effect.Randomized, double-blind, placebo-controlled trials that

* Correspondence: [email protected] Program, Centre for Addiction and Mental Health, 100 Stokes St.,33 Russell Street, Toronto, ON M6J 1H4, Canada2Departments of Family and Community Medicine and Psychiatry and theDalla Lana School of Public Health, University of Toronto, Toronto, ONCanadaFull list of author information is available at the end of the article

© 2012 Selby et al.; licensee BioMed Central LCommons Attribution License (http://creativecreproduction in any medium, provided the or

are largely explanatory are necessary to establish the safetyand efficacy of new interventions and to inform evidence-based guidelines [2]. However, explanatory trials forchronic diseases such as hypertension, diabetes, de-pression and addiction have a number of limitations [3,4].They are often conducted in tertiary centres, excludepeople with comorbid conditions that cluster with thecondition of interest, provide some incentive for participa-tion, and mandate intensive follow-up visits and contactwith research staff [5].Explanatory studies, although highly internally valid, are

often less generalizable to outpatient community settings.There has been a call for more real-world ‘practical’ or

td. This is an Open Access article distributed under the terms of the Creativeommons.org/licenses/by/2.0), which permits unrestricted use, distribution, andiginal work is properly cited.

mailto:[email protected]

http://creativecommons.org/licenses/by/2.0

Selby et al. BMC Medical Research Methodology 2012, 12:101 of 13http://www.biomedcentral.com/1471-2288/12/101

‘pragmatic’ studies to enhance generalizability, but onlyrudimentary methods for distinguishing between efficacy(explanatory) and effectiveness (pragmatic) studies havebeen employed [1,3,4,6-8]. The definition and design ofpragmatic trials vary considerably, and are derived mainlyfrom descriptive papers. These often describe observa-tional studies that, in spite of limitations in internal vali-dity and the ability to control for confounders, havefrequently been used to influence clinical practice [9,10].Health policy makers have to make resource allocationdecisions based on cost-effectiveness studies that mayhave excluded various populations of interest [3,11].Therefore, randomized clinical trials with inherent internalvalidity, but with greater ecological and external validity –pragmatic, randomized trials – are required in real-worldsettings after safety and efficacy have been established.In Canada, there is limited drug plan coverage for smo-

king cessation treatments despite their proven efficacy[12-15], ostensibly due to the lack of pragmatic, rando-mized trials. The multicentre, community-based, prag-matic, randomized, controlled ACCESSATION Study(ClinicalTrials.gov Identifier NCT00818207) was designedto determine whether smoking cessation treatment insu-rance coverage is associated with improved outcomes inclinical practice. We addressed two key barriers to smo-king cessation treatment: i) the lack of formulary coveragefor smoking cessation treatment by most governmentsand private drug plans in Canada (excepting the provinceof Quebec at the time of the study), and ii) the cost ofmedications to patients. Based on existing study designelements that favoured a pragmatic study design, wedeveloped the study protocol to make the design as prag-matic as possible. The basic study flow (Figure 1) resem-bles that of a traditional randomized, controlled trial.However, for each aspect of the trial, we attempted tosimulate real-world conditions; this was based on discus-sion among the authors.After our study was initiated, an international consor-

tium published the PRECIS (Pragmatic–ExplanatoryContinuum Indicator Summary) [16] model to help tri-alists assess the degree to which a study design fallsalong the pragmatic–explanatory continuum. The tooluses 10 key domains that qualitatively distinguish prag-matic (externally valid) from explanatory (internallyvalid) trials [16]. Although it has not been validated topredict outcomes post facto, no other instrument existedat the time and the consortium that developed this toolinvites validation and enhancements to the process.Other limitations included the absence of a quantitativerating system that would increase precision, reproduci-bility and comparability of scores and of a formalprocess to reach consensus among investigators.This paper describes the use of the PRECIS tool,

coupled with the use of a numerical scale and a modified

Delphi technique, to achieve consensus on the trial de-sign, to characterize aspects of the study that determinewhether it could be described as pragmatic.

MethodsThe 50 participating sites of the ACCESSATION Studythat could utilize a Central Ethics Committee werereviewed and approved by Institutional Review Board(IRB) Services (Suite 300, 372 Hollandview Trail, Aur-ora, ON L4G 0A5, Canada), and the seven remainingsites submitted to an IRB that reviewed and approvedthe study in the respective regions.We analysed the ACCESSATION Study trial design

elements on a continuum using the qualitative, multidi-mensional PRECIS tool. Table 1 provides an overview ofthe most important study design characteristics in rela-tion to the 10 domains of the PRECIS tool, and wasdeveloped by the primary author.

Domain rating processTo simplify the rating process, we added a quantitativeaspect to evaluations using the PRECIS tool by adding avisual 20-point numerical scale, where 1 represented ‘en-tirely explanatory’ and 20 represented ‘entirely prag-matic’. Six raters (five authors and one consultant: oneacademic family physician with an interest in smokingcessation; one cardiac rehabilitation physician with ex-pertise in pharmacoeconomics; one addiction medicinephysician and clinical scientist with a focus on tobaccodependence; one pharmacist with expertise in pharma-coeconomics; one pharmacologist with clinical researchand medical affairs experience in the pharmaceutical in-dustry; and one consultant physician with pharmacoeco-nomic and policy advice experience in Quebec) wererequested to score the trial on each domain in Table 1according to where they believed it fell on a pragmatic–explanatory trial continuum, and to provide an explan-ation for their decision. Raters were asked to reviewTable 1 and to read the manuscript: ‘A pragmatic–explanatory continuum indicator summary (PRECIS): atool to help trial designers’ [16] as a guide to scoring the10 domains of the ACCESSATION Study. A modifiedDelphi technique was then used to ensure a commonunderstanding among the raters, given their multidiscip-linary background (addiction medicine, family medicine,internal medicine, pharmacy, pharmacoeconomics):

1. Initial group discussion among the raters regarding thestudy protocol and various elements classified by the 10domains and criteria to justify their position, on a scaleof 1 to 20 along the explanatory–pragmatic continuum.

2. Round 1 scoring: raters independently scored eachdomain (see domain rating process below); anindependent assistant collated the scores and

Full SCT coverage (100%)

Pre-screeningperiod

Week 0 2 52

V1 T2

13* 26 39*

T3 V4 V7 V10

SCT reimbursementperiod

Follow-upperiod

No SCT coverage (0%)

Call centrepre-screening

Screening/randomization (1:1)

Primary endpointSelf-reported 7-daypoint prevalence of abstinenceat Week 26

*Other abstinence assessments were conducted at Weeks 13 and 39 (other secondary endpoints)

SCT: smoking cessation therapyV: office/clinic visitT: telephone contact

Key secondary endpointUrine cotinine test-confirmedcontinuous abstinence ratebetween Weeks 26 and 52

Key secondary endpointUrine cotinine test-confirmed7-day point prevalence ofabstinence at Week 26

Target quit date

Figure 1 ACCESSATION study design.


developed descriptive statistics (median, mean,standard deviation [SD]). The results wereanonymized before the face-to-face groupdiscussion.

3. A second face-to-face group discussion among theraters was held to clarify individual ratings, gain abetter understanding of each domain and reachconsensus.

4. Round 2 scoring: raters did a second round of ratingindependently and submitted their scores. Scoreswere again collated in an anonymized manner togenerate descriptive statistics (Figure 2A). Theresults of the first and second rounds of scoringwere plotted in a spider graph as described by thePRECIS tool developers [16] (Figure 2B).

ResultsTable 1 results, developed prior to the group discus-sions and domain rating process, suggested that onlythe domain related to ‘Follow-up intensity’ was consid-ered more explanatory than pragmatic. Domains relatedto ‘Primary trial outcome’, ‘Secondary trial outcome’ and‘Practitioner adherence to study protocol’ were consid-ered to have a balance of both pragmatic and expla-natory elements. However, the descriptive statisticscalculated after ratings and discussions had taken place

(Figure 2A) indicated that all 10 domains scored higherthan the midpoint of 10.5. This indicates that allthe domains were more pragmatic than explanatory –albeit some were borderline. Descriptive statistics(Figure 2A) indicated that there was less variation inscores after the second round than after the first roundof discussions for every domain. This suggests that theraters’ opinions converged, presumably as a result ofreaching a common understanding of all aspects of thetrial in relation to each of the PRECIS domains. Thespider plot (Figure 2B) demonstrates the shift in opi-nions among the raters between the first to the secondround of discussions, with the plot becoming larger –more pragmatic – after the second round ofdiscussions.

DiscussionThis paper describes the use of the PRECIS tool for themultidimensional evaluation of the ACCESSATIONStudy, and provides a thorough exploration of the studydesign that impacts its pragmatic/explanatory nature.Use of the tool highlighted study design features forwhich discrepancy of opinion existed among the authorsregarding the degree of pragmatism within the trial, andprovided a basis for discussing those areas more ex-plicitly. However, this occurred after the study was

Table 1 Pragmatic and explanatory approaches in the ACCESSATION study

PRECIS domain Favours pragmatic trial Criteria that do notfavour either

Favours explanatory trial Summary

Participants

Eligibility criteria • Adult smokers(aged 18–75 years)• Prospective subjects weremotivated to make a quitattempt within 14 days• Subjects were recruitedfrom community-basedsites across Canada• Comorbid patients included• Past compliancenot considered• No risk stratification

• Smokers excluded ifthey have existing SCT;female subjects ofchildbearing potentialcould be included if notpregnant or nursing, andpractising effectivecontraception(not eligible for within-labelpharmacotherapy due tosafety reasons)

• Smokers excluded ifthey had a quit attemptin the past 30 days, hadunstable comorbid conditions,possessed SCTs, lived withanother study participantor were currentlyparticipatingin another study• Participants smoked≥10 cigarettes per day• Eligibility had tobe determinedby a central service

• Mostlypragmatic

Intervention and expertise

Experimentalintervention –flexibility

• Physicians and patients controlledthe type of smoking cessationinterventions accordingto usual practice Any on-labelcombination of non- andpharmacological interventionwas permitted• Participants could switchmedications• Type or number ofco-interventions used was not limited• Drug reimbursement card allowed anypharmacist to fill the prescription,rather than the doctor’s office• Side effects were to be managedat the discretion of the provider• Fewer contact minutes than efficacytrials with allowance for as-neededvisits to assist the smoker

• Following randomization,a participant could not bere-randomized to a morefavourable arm

• Provision of the card bythe practitioner is not howinsurance coverage cards areprovided to patients• x`More contactthan usual practice withscheduled ratherthan as-needed visits

• Mostlypragmatic

Experimentalintervention –practitionerexpertise

• All physicians werelicensed to practice inCanada, and includedGPs/FPs (86%) andspecialists in cardiovascularmedicine, surgery,cardiology, pulmonologyand psychiatry• Most investigators (80%) hadno prior clinical research experience• Primarily treatment- seeking patientsfrom physician practice (93%)• None of the authors wereassociated with sites of recruitment

• Some investigators (20%)had prior clinicalresearch experience

• Mostlypragmatic

Comparisonintervention –flexibility

• Participants received counsellingabout smoking cessation optionsand chose their SCT priorto randomization.This reflects usual practice

• Participants were given a drugreimbursement card,which included$5 to partially offsetpharmacist dispensing fees,to enable trackingof SCT use if obtainedfrom a pharmacy

• Mostlypragmatic


Table 1 Pragmatic and explanatory approaches in the ACCESSATION study (Continued)

Comparisonintervention –practitionerexpertise

• Randomization was atthe patient level to mimic thereal world, so that within the samepractice some smokers receivedreimbursement while others did not.Practitioner expertise was thereforethe same for both arms at each site,reflecting the real-worldvariation in expertisein smoking cessation• Given the centralrandomization process,re-randomizationto a more favourable arm(i.e., SCT coverage) was not possible

• Some training of smokingcessation occurred at theinvestigator meetingto ensure all practitionershad knowledge ofevidence-based interventionsfor smoking cessation

• Mostlypragmatic

Follow-up and outcomes

Follow-upintensity

• Participants did not attendthe clinic on a weekly basis• At clinic visits/telephone contact,outcomes were measured,and AEs andmethod(s) used to quit(if any) were recorded• No exhaled CO levelswere measured toreflect real-world practice

• Randomization visit involvedmore procedures than wouldusually be included at the firstvisit to discuss quitting• Although participants attendedfewer contact visitsthan a traditionalexplanatory trial, theyreceived morefrequent contact thanusual in clinicalpractice to balance theneeds of datacollection with the clinical reality ofaccess to care• Strong focus on measuringoutcomes and AEsfrom SCT use and/or quitting• A suicide severityscale was includedbecause of recentreports of possiblecorrelation with the use of SCTs

• Favoursexplanatory

Primarytrialoutcome

• Primary endpoint:comparison of theself-reported seven-daypoint prevalence ofabstinence betweenthe full and noreimbursement groupsat Week 26 asopposed to 4 weekscontinuous abstinencein efficacy trials in smokingcessation fromthe quit date. This endpoint is used inobservational studies intelephone quitlineand NRT distribution studies• Outcome assessed locally andno central adjudication of outcome

• Scheduled time forassessment ofoutcome at 26 weeks

• Balances bothexplanatoryand pragmaticelements

Secondaryendpoint

• A key secondary endpointwas self-reported 7-day pointprevalence of abstinenceat Week 26, confirmed byurine cotinine analysis.Urine collection is common in practiceand the least intrusivemethod to confirm self-reports• Exhaled CO was not used,since this is less real-world practice

• Urine cotinine confirmation wasused as a secondaryendpoint toindependently confirmself-reported abstinence.However, results werenot shared with the practice

• Balances bothexplanatoryand pragmaticelements


Table 1 Pragmatic and explanatory approaches in the ACCESSATION study (Continued)

Compliance/adherence

Participantcompliancewithintervention

• Once randomized, participants couldchoose to use (or not) anycurrently available SCT• SCT use patterns werepassively measured bydrug reimbursementcard use across both groups• Data regarding adherencewere collected but not sharedwith prescribers

• All or a portion of the participantswere compensated for costs incurredto visit the clinic. They could havebeen compensated between $25 and$50 for each on-site visit (four on-sitevisits at V1, V4, V7 and V10).There was variation betweensome sites with the total allocationbetween $100 and $200 to accountfor local factors

• Mostlypragmatic

Practitioneradherence tostudy protocol

• There was no measure of howpractitioners provided SCT per se ordeveloped their source documentationfor the study• There were some deviations fromprotocol, including prescribingoff-label, no smokingcessation counsellingbefore unblinding, andenrolling participantswho were not motivated to quit

• Sites and physicians were visited bythe site monitor twice during thestudy but kept to a minimum• Randomization was centrallycontrolled to prevent gaming.However, participants could useany available SCT• Standard study auditingprocesses discoveredone site was not compliant with GCP(inadequate source documentation)• Study monitors would bringoff-label use of medication to theattention of the investigator

• Balances bothpragmaticversus explanatorytrials

Analysis

Analysis ofprimaryoutcome

• The main analysis will be the ITTpopulation post-randomization regardlessof use of the drug reimbursement cardor making a quit attempt, with missingdata being counted as being a smoker

• Mostlypragmatic

AE: adverse event; CO: carbon monoxide; GCP: good clinical practice; GP: general practitioner; FP: family practitioner; ITT: intent-to-treat; NRT: nicotine replacementtherapy; SCT: smoking cessation treatment.


initiated, but before data collection was completed. Thehigh variability in ratings at the first scoring round wasprimarily due to differences in interpretation of the cri-teria described in the PRECIS tool and of how the designelements of the study fitted these dimensions. Thedimensions were discussed and ratings were clarifiedbased on the PRECIS tool. Therefore, it appears thatdeliberate discussion about each dimension is neces-sary, especially when there is considerable variabilitybetween raters. Use of a Delphi method is appropriateto reach consensus on such complex and subjectivematerial.If we were to design the study to be more pragmatic,

we would reduce the frequency of visits for assessmentsand use a patient- and physician-defined primary out-come measure. For example, we would ask patients ifthey had quit or not, as opposed to using a validatedscale. To make the study completely pragmatic on theprimary outcome measure, we would use an administra-tive database to see if there was reduction in healthcareutilization in those who received coverage versus thecontrol group.Developers of the PRECIS tool [16] considered it to be

an initial attempt to identify and quantify trial character-istics that distinguish between pragmatic and explanatory

trials, and requested suggestions for its further develop-ment. Since 2010, five papers describing modificationsto the PRECIS tool have been published, all of whichemploy quantification of the ratings on each dimension[17-21]. Each paper is summarized in Table 2.In a similar analysis to ours, Riddle et al. used the

PRECIS tool to design a randomized, controlled trialof pain-coping skills [17]. The authors also used thePRECIS tool to assist with face-to-face meetings andfound the approach helpful, for similar reasons. They,too, added a semi-quantitative scale, but of 4 cm inlength, and had three rounds of discussions. Theirfinal evaluation led to greater agreement on all dimen-sions, whereby they increased the explanatory scoresof each domain. The timing of their exerciseprompted the authors to make revisions to the designof their randomized trial prior to submission for fund-ing [17].Tosh et al. [19] had three reviewers (co-authors) use a

1- to 5-point scale to review published trials in mentalhealth; they referred to this as the Pragmascope. If a di-mension could not be rated, it received a score of 0.Each trial could be allocated a total possible score of 50,with a range of 0 to 30 indicating an explanatory trial,31 to 39 indicating a balanced trial and any score >35

1

Eligibilitycriteria

Experimentalintervention–flexibility

Experimentalintervention–practitioner

expertise

Comparisonintervention–flexibility

Round 1 scoring

Comparisonintervention–practitioner

expertise

Follow-upintensity

Primary trialoutcome

Participantcompliance

withintervention

Practitioneradherence

to studyprotocol

Analysis ofprimaryoutcome

2

3

4

5

6

7

8

9

10

11

Med

ian

sco

re (

min

, max

)

12

13

14

15

16

17

18

19

20

Round 2 scoring

Comparison intervention –flexibility

Comparison intervention –practitioner expertise

Follow-up intensity

Primary trial outcome

Participant compliancewith intervention

Practitioner adherenceto study protocol

Analysis of primary outcome

Round 1 scoringRound 2 scoring

Eligibility criteria

Experimental intervention –flexibility

Experimental intervention –practitioner expertise

E

b

Figure 2 A. Author ratings for the ACCESSATION trial: median scores (min, max). B. Author ratings for the ACCESSATION trial using thepragmatic–explanatory continuum indicator summary (PRECIS).


Table 2 Studies attempting to quantify PRECIS dimensions since 2010

# Study Author Year Ref. Context of use anddevelopment

Numeric scale Delphi/iterativeprocess

Conclusions Limitations

1 The Pragmatic–ExplanatoryContinuumIndicator Summary(PRECIS) instrumentwas usefulfor refininga randomizedtrial design:experiencesfrom aninvestigativeteam

Riddle et al. 2010 [17] •Prospective use ofthe tool to modifystudy design in 3-armstudy of pain coping inpatients scheduled to havea knee replacement 1-daymeeting of 7authors/investigators:1) Pre-read Thorpe et al., 20092) Discuss goal of study3) Discuss criteria4) Initial blinded rating5) Ideal blinded rating6) 3rd rating to see ifconsensus was reached

• Yes: 4-cm lineto bemarkedindependentlyon paper in personin 5 minfollowed bydiscussion• Scores measuredby a ruler

Yes • Modificationof study• Closerconsensusand moreexplanatoryin design

• 4-cmscale usingVAS;not applicable toonline ratings

2 Alternativeapproachesto tuberculosistreatmentevaluation:the role ofpragmatic trials

Bratton et al. 2011 [18] • Retrospective• 2 reviewers rated3 published studies onTB treatmentand discussedhow to rate them oneach dimension

• No, but modificationto the dimensions made(practitionerexpertise) combinedand a new spokefor blindnessinserted

• Yes, but notdescribed in detail

• Pragmatictrials inTB mightlessen time toimplementation inreal-world settings

• No metric at all,simply subjective

3 Pragmatic vsexplanatory trials:the Pragmascope toolto help measuredifferencesin protocolsof mentalhealth RCTs

Tosh et al. 2011 [19] • Adapt (PRECIS) toassist researchers duringprotocol stage ofRCTs in mentalhealth (the Pragmascope tool)• Retrospective• 3 reviewers• Cochrane SchizophreniaGroup TrialsRegister andMedline (November 2010)for references ofRCT protocols.Chose a randomsample of10 protocols dealing withschizophrenia,depression,post-traumatic stressdisorders, andpsychiatricrehabilitation

• 1 to 5; 0 for missinginformation• Total score, 0 to 50• 0 to 30,explanatory; 31- 39,intermediate;≥35, explanatory• However: Figure 1(see main text)demonstratesan explanatorystudy investigatingwhether theexperimentalintervention willwork in idealcircumstances(total score 0–15)and a morepragmatic studyfocusing mostlyon whether,in routine practice,an interventionhas a meaningful effect

• Notdescribed

• Useful tool givenhigh inter-rater reliability

• Scoringnot clear;neither why themidpoint of25 was notchosen as abalanced trial• Does the totalscore perpetuatethe dichotomousclassificationof studies?

Selbyet

al.BMCMedicalResearch

Methodology

2012,12:101Page

8of

13http://w

ww.biom

edcentral.com/1471-2288/12/101

Table 2 Studies attempting to quantify PRECIS dimensions since 2010 (Continued)

(total score >35)• A total scorebetween 16and 35 suggestsan interimwhere trialdesign balancespragmaticand explanatorydomains

4 Applying thePRECIScriteria todescribe 3effectiveness trials ofweight loss in obesepatients withcomorbid conditions.

Glasgow et al. 2011 [20] • Practice-basedOpportunitiesfor WEight Reduction(POWER) trials(3 studies)• Reduce weightin primarycare in those withone CVD risk factor.Studies were ongoingand in the field• 9 reviewers scored eachprotocol in a4-step process.1) Read article and reviewwebinar by Sackett2) Score using 0 to4 on the dimension• 8 additional ratings;representationof participantsand settings,inclusion ofcost estimates,reporting oncontext and levelof engagementwith the primary carepractices

• 0 to 4 on thedimensionwith total compositescores also

• Not formallydescribed, butdoes mentionfrequent calls to gainconsistency inthe interpretationof the dimensions

• Requires discussionand training toclarify each criteria

• ? Rater biastowards pragmatic• Scale difficult tointerpret, butneed cognitive testingas part of aqualitative study• Hard to find areliability index

5 Pragmatic vs.explanatory: anadaptationof thePRECIS toolhelps tojudge theapplicabilityof systematicreviews

Koppenaal et al. 2011 [21] • Improve lifestylein general;should be implementedin general practice• Modified the PRECIStool (calledPRECIS Review tool[PR tool]) to gradeindividualtrials and systematic

• 1 to 5 rating• Individualstudies and thereview itself were scored• Abandoned VASwith 0 to10 due to thearbitrary distinctionsbetweenconsecutive scores• Used a Likert-type

• 2 independent scoresper study andthen discussionto reach consensus• If unable, thenconsultationwith third rater

•Useful toolfor reviewersand policymakers and trialists• Ability to detectheterogeneity instudies included insystematic reviews

• Missing informationmore likely for adimension to berated as pragmatic• Applicabilityto settingand contextneeds to consideredby policy makerseven if a trial is

Selbyet


Methodology

2012,12:101Page

9of

13http://w

ww.biom


Table 2 Studies attempting to quantify PRECIS dimensions since 2010 (Continued)

for dailypractice

reviews of trials• This should helppolicy makers,clinicians,researchers, andguideline developersto judgethe applicabilityof individualtrials andsystematic reviews• 2 systematic reviews

scale of 1 to5, with concurrent %• If 3 dimensionsunscorable,a randomizedcontrolled trialwas not to be used• 1 to 5 less arbitrary than0 to 10 for thistype of review

consideredpragmatic• Equal weighting foreach dimension

RCT: randomized controlled trial.

Selbyet


Methodology

2012,12:101Page

10of

13http://w

ww.biom



indicating a pragmatic trial. However, in Figures 2 and 3,Tosh et al use ranges of 0 to 16 to describe explanatorytrials and 16 to 35 to describe an interim trial that isbalanced. They had independent ratings and averagingof scores, but did not describe an explicit process to beused to reach consensus.Several limitations are associated with the use of the

Pragmascope at this time. For example, if the dimensioncould not be rated, the dimension would receive a scoreof 0 and as such, bias ratings towards the study beingexplanatory. Moreover, the use of cut-offs for the totalscores categorizing trials reverts to the problem of look-ing at trial design as purely explanatory or pragmatic[8,16]. Moreover, the reason for the cut-offs used isnot specified. It is not clear why they did not choose25 (the midpoint) to indicate a balanced trial and anyscore less than that would favour an explanatory study,while any score greater than 25 would favour a pragmaticstudy. Moreover, we agree with Glasgow et al. [5] andSpigt and Kotz [22] that composite scores should beavoided because widely disparate trials can receive thesame score and defeat the purpose of having a dimensionalapproach to the rating.The PRECIS Review (PR) tool was developed by

Koppenaal et al. [21] to evaluate systematic reviews andthe randomized controlled trials used in the review tohelp policy makers decide on applicable trials to in-form their work. Like us, they quickly realised that aVisual Analog Scale (VAS) scale of 0 to 10 was arbitraryand so converted it to a Likert-type scale of 1 to 5, also in-cluding a percentage score. They used two reviewers andan additional reviewer to rate the score when consensuscould not be reached. The scoring scale appeared to bevalid for the stated purpose and they acknowledged thelimitations of broader applicability. Again, given the pur-pose behind the PRECIS tool to introduce multidimen-sionality to the evaluation of a study design, scores areimportant to initiate and guide discussion, but broaderconsensus on the rating is still required to inform decisionmaking.Glasgow et al. [20] also used a 5-point (0 to 4) scale to

rate three interrelated, yet separate studies by investiga-tors from three separate institutions. They describe asimilar process of training reviewers and noted thatinvestigators tended to rate their own papers as beingpragmatic. The scoring revealed moderate levels ofvariability with most variability within 1 point on the5-point scale. However, several telephone calls wererequired to develop consensus on the meaning of eachscore. It is possible that the scale was not sufficientlysensitive to detect a difference, which would be import-ant if the group was interested in achieving consensus,but less so if trying to evaluate the study per se andcategorize the protocol dichotomously.

Our proposed refinement also identified the need for arating scale, but included a modified Delphi techniqueto reach consensus [23]. We chose a 20-point numericalscale to approximate a continuous scale. This permittedeasier, more accurate and more stable coding of the re-sponse using e-mail. A VAS with measurement is appro-priate when standardized in pen and paper format ratherthan e-mail, which distorts the dimensions. We alsoused extreme anchor points, 1 to 20, to discourage rat-ing the domains beyond the numbers provided. More-over, Likert scales have increased reliability with up to11 steps, 7 steps being the minimum. Therefore, thescale we used was most sensitive to capture inter-individual differences to better target our discussions.This may be one reason why the spider graphs do notreach the extremes, but it is also possible that the ratersappreciated that elements existed in each dimension toprevent an extreme rating. Use of the iterative techniqueprovided a sound basis for discussing the intricacies ofthe trial design and allowed individuals to provide view-points anonymously and then offer their opinions duringface-to-face meetings.Taken together, these examples demonstrate that de-

pending on the purpose of the application of the PRECIStool (study evaluation versus study design), differentscales and methods may need to be used to rate studies.However, our method may be particularly helpful to tri-alists to ensure common understanding of a study de-sign when working in teams with disparate expertise.Therefore, other investigative teams may find theseapproaches helpful.The multidimensional PRECIS tool can be implemen-

ted easily by investigators and represents a major ad-vance in the design and evaluation of clinical trials thatinform practice, as demonstrated by our own experienceand that of others. All clinical trialists need to makecompromises in their design due to a variety of practicalfactors that affect the conduct of a large study. Colla-borative research by a team requires consensus on studydesign to ensure the methods are appropriate to answerthe study question. Methods to evaluate study designand reach consensus are needed to ensure that disparateviews and perspectives can be reconciled so that the bestpossible course of action is adopted.Although most agree that the 10 dimensions are ne-

cessary to understand the explanatory–pragmatic con-tinuum, numerical scales run the risk of dichotomouslyclassifying the study and we did not provide a compositescore for the study. This required a qualitative approach.Therefore, a more structured process using the Delphitechnique that we employed, or a similar nominal grouptechnique used by Riddle et al. [17], allowed a moredemocratic process of consensus among the investigators,who hailed from different disciplines and institutions.


This process may be helpful to investigators during thedesign stage of a multicentre collaborative study to resolvedisagreements and assist in reaching a common under-standing of the design of the study.

ConclusionsThe PRECIS tool may be applicable across a variety ofhealth-related studies to help investigators design trialsmost appropriate to their study question and hypothesis.Moreover, clinicians, study reviewers, policy makers andthe so-called post-regulatory decision-makers can usethis tool to determine if a study has generalizability tothe populations of interest and the level of reasonableeffectiveness that can be expected in different ecologicalsettings versus those in explanatory trials. In these situa-tions, simpler rating systems as described by othersmight be adequate to achieve the desired outcome.

AbbreviationsIRB: Institutional Review Board; PR: PRECIS Review; PRECIS: Pragmatic–Explanatory Continuum Indicatory Summary; SD: Standard deviation;VAS: Visual Analog Scale.

Competing interestsAll authors have completed the Unified Competing Interest form at www.icmje.org/coi_disclosure.pdf (available on request from the correspondingauthor). All non-Pfizer authors declare financial compensation from Pfizer Inc.for professional services, including protocol and clinical trial materialsdevelopment, initial start-up and end-of-study activities such as Case ReportForm review, Statistical Analysis Plan review, preparation, participation andpresentation at the Investigator Meeting, and Clinical Study Report review.No payments were made by Pfizer Inc. to PS, GB or PO for authorship and/orauthorship-related activities of this paper. Travel expenses such as airfare,hotel accommodation, meals and ground transportation were reimbursed byPfizer Inc. when appropriate. During the conduct of the study, PS and GBreceived research funding from their respective employers.

AcknowledgementsThe ACCESSATION Study was sponsored by Pfizer Inc. Jacques Le Lorier,Professor in the Departments of Medicine and Pharmacology at theUniversity of Montreal and Chief of Pharmacoepidemiology andPharmacoeconomics at the Research Centre of the University of MontrealHospital Centre, kindly provided elaboration of the initial concept, the semi-quantitative scoring, contributed to the protocol design and discussion ofthe statistical analysis plan. Susan Bondy, PhD, University of Toronto DallaLana School of Public Health, provided advice regarding the use of VisualAnalog Scales. Editorial assistance in the form of proofreading, collation ofreview comments, formatting the manuscript for submission, preparation offigures and formatting of references was provided by Alexandra Bruce, TinaMorley and Helen Jones of UBC Scientific Solutions and was funded by PfizerInc. Rebecca Cowan from Med Plan Inc. provided administrative assistanceand was funded by Pfizer Inc.

Author details1Addictions Program, Centre for Addiction and Mental Health, 100 Stokes St.,33 Russell Street, Toronto, ON M6J 1H4, Canada. 2Departments of Family andCommunity Medicine and Psychiatry and the Dalla Lana School of PublicHealth, University of Toronto, Toronto, ON Canada. 3Ontario TobaccoResearch Unit, Toronto, ON Canada. 4Department of Family Medicine,Dalhousie University, Halifax, NS Canada. 5Cardiac Rehabilitation andSecondary Prevention Program, Toronto Rehabilitation Institute, Toronto, ONCanada. 6Department of Medicine, University of Toronto, Toronto, ONCanada. 7Health Economics & Outcomes Research, Pfizer Canada Inc,Kirkland, Québec Canada. 8Therapeutic Areas, Cardiovascular and Respiratory,Medical Division, Pfizer Canada Inc, Kirkland, Québec Canada.

Authors’ contributionsPS conceptualized the idea for the paper, the addition of the Delphi Paneland the semi-quantitative scale for each dimension. He also preparedTable 1 for discussion and prepared the first draft. All authors participated inand made a substantial contribution to the discussions of the conceptionand design, and acquisition analysis and interpretation of data. They alsorevised the article critically for intellectual content and gave final approval ofthe version submitted for publication. All authors read and approved thefinal manuscript.

Received: 27 October 2011 Accepted: 23 July 2012Published: 23 July 2012

References1. Schwartz D, Lellouch J: Explanatory and pragmatic attitudes in

therapeutical trials. J Chronic Dis 1967, 20:637–648.2. Guyatt GH, Sackett DL, Cook DJ: Users’ guides to the medical literature. II.

How to use an article about therapy or prevention. B. What were theresults and will they help me in caring for my patients? Evidence-BasedMedicine Working Group. JAMA 1994, 271:59–63.

3. Tunis SR, Stryer DB, Clancy CM: Practical clinical trials: increasing the valueof clinical research for decision making in clinical and health policy.JAMA 2003, 290:1624–1632.

4. March JS, Silva SG, Compton S, Shapiro M, Califf R, Krishnan R: The case forpractical clinical trials in psychiatry. Am J Psychiatry 2005, 162:836–846.

5. Glasgow RE, Emmons KM: How can we increase translation of researchinto practice? Types of evidence needed. Annu Rev Public Health 2007,28:413–433.

6. Liberati A: The relationship between clinical trials and clinical practice:the risks of underestimating its complexity. Stat Med 1994, 13:1485–1491.

7. Glasgow RE, Magid DJ, Beck A, Ritzwoller D, Estabrooks PA: Practical clinicaltrials for translating research to practice: design and measurementrecommendations. Med Care 2005, 43:551–557.

8. Gartlehner G, Hansen RA, Nissman D, Lohr KN, Carey TS: A simple and validtool distinguished efficacy from effectiveness studies. J Clin Epidemiol2006, 59:1040–1048.

9. Jepsen P, Johnsen SP, Gillman MW, Sørensen HT: Interpretation ofobservational studies. Heart 2004, 90:956–960.

10. Hannan EL: Randomized clinical trials and observational studies:guidelines for assessing respective strengths and limitations. JACCCardiovasc Interv 2008, 1:211–217.

11. Mullins CD, Whicher D, Reese ES, Tunis S: Generating evidence forcomparative effectiveness research using more pragmatic randomizedcontrolled trials. Pharmacoeconomics 2010, 28:969–976.

12. 2008 PHS Guideline Update Panel, Liaisons, and Staff: Treating tobacco useand dependence: 2008 update U.S. Public Health Service ClinicalPractice Guideline executive summary. Respir Care 2008, 53:1217–1222.

13. Cahill K, Stead LF, Lancaster T: Nicotine receptor partial agonists forsmoking cessation. Cochrane Database Syst Rev 2010, 12:CD006103.

14. Hughes JR, Stead LF, Lancaster T: Antidepressants for smoking cessation.Cochrane Database Syst Rev 2007, 1:CD000031.

15. Stead LF, Perera R, Bullen C, Mant D, Lancaster T: Nicotine replacementtherapy for smoking cessation. Cochrane Database Syst Rev 2008,1:CD000146.

16. Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG,Tunis S, Bergel E, Harvey I, Magid DJ, Chalkidou K: A pragmatic-explanatorycontinuum indicator summary (PRECIS): a tool to help trial designers.J Clin Epidemiol 2009, 62:464–475.

17. Riddle DL, Johnson RE, Jensen MP, Keefe FJ, Kroenke K, Bair MJ, Ang DC:The Pragmatic-Explanatory Continuum Indicator Summary (PRECIS)instrument was useful for refining a randomized trial design:experiences from an investigative team. J Clin Epidemiol 2010,63:1271–1275.

18. Bratton DJ, Nunn AJ: Alternative approaches to tuberculosis treatmentevaluation: the role of pragmatic trials. Int J Tuberc Lung Dis 2011,15:440–446.

19. Tosh G, Soares-Weiser K, Adams CE: Pragmatic vs explanatory trials: thepragmascope tool to help measure differences in protocols of mentalhealth randomized controlled trials. Dialogues Clin Neurosci 2011,13:209–215.


20. Glasgow RE, Gaglio B, Bennett G, Jerome GJ, Yeh HC, Sarwer DB, Appel L,Colditz G, Wadden TA, Wells B: Applying the PRECIS criteria to describethree effectiveness trials of weight loss in obese patients with comorbidconditions. Health Serv Res 2011. doi:10.1111/j.1475-6773.2011.01347.x.[Epub ahead of print]. Nov 2.

21. Koppenaal T, Linmans J, Knottnerus JA, Spigt M: Pragmatic vs. explanatory:an adaptation of the PRECIS tool helps to judge the applicability ofsystematic reviews for daily practice. J Clin Epidemiol 2011, 64:1095–1101.

22. Spigt MG, Kotz D: Comment on: ‘a simple and valid tool distinguishedefficacy from effectiveness studies’. J Clin Epidemiol 2007, 60:753–755.

23. Mullen PM: Delphi: myths and reality. J Health Organ Manag 2003, 17:37–52.

doi:10.1186/1471-2288-12-101Cite this article as: Selby et al.: How pragmatic or explanatory is therandomized, controlled trial? The application and enhancement of thePRECIS tool to the evaluation of a smoking cessation trial. BMC MedicalResearch Methodology 2012 12:101.

Submit your next manuscript to BioMed Centraland take full advantage of:

• Convenient online submission

• Thorough peer review

• No space constraints or color figure charges

• Immediate publication on acceptance

• Inclusion in PubMed, CAS, Scopus and Google Scholar

• Research which is freely available for redistribution

Submit your manuscript at www.biomedcentral.com/submit

http://dx.doi.org/10.1111/j.1475-6773.2011.01347.x

how pragmatic or explanatory is the randomized, controlled trial

Documents