British Journal of Obstetrics and Gynaecology April 1989, Vol. 96, pp. 397400

Reporting controlled trials

ADRIAN GRANT

The aim of this short paper is to make it easier for authors, refcrccs, cditors and, ultimately, rcadcrs to assess the quality of controlled trials mounted to compare alternative forms of care. The guidelines set out below are based on check- lists publishcd in other journals (e.g. the British Medical Journal 1987), and schemes suggested for assessing the quality of published random- ized controlled trials (Mahon & Daniel 1964; Mosteller et uf. 1980; McMastcr University Department of Clinical Epidemiology and Bio- statistics 1981; Chalmers et al. 1981; Der Simo- nian et ul. 1982; Zelen 1983; Emcrson et ul. 1984; Thacker 1985; Meinert 1986). The rationale for paying attention to aspects of study design and reporting is discussed in these and other publi- cations included in the bibliography (see Grant & Chalmers 1985; Chalmers 1989).

A list of 28 reasonable questions to ask in respect of reports of controlled trials is pre- sented in Table 1. Not all of these questions are relevant to all trials. Ncvertheless, cvidence that investigators have paid appropriate attention to these guidelines is likely to improve the chances that a trial report submitted to this Journal will be accepted for publication.

introduction to the report

Some of the questionr. such as whether there hac bcen a clear statement of the rationale for and the objectivc(s) of a Controlled trial (Question l), may seem hardly worth asking; yet it IS sur- prising how often thc objectives of studies arc not clearly statcd in manuscripts and even pub- lirhed reports.

Natianal Perinatal Epidemiology Unit, Radcliffe Infirmary, Oxford OX2 QHE ADRIAN GRANT

Description of materials and methods

The answers to questions about the materials and methods used in controlled trials must bc clear if readers are to judge how much faith to place in the results presented. Questions 2-13 deal with the most important of these issues. A common controversy associated with controlled trials is the area of clinical practice to which thc results of a trial can he generalized. There must, therefore, be a clear description of the source of the study participants (Qucstion 2) and of the entry and exclusion criteria (Question 3). In some trials it is possible to describe people who met the entry criteria but who did not join thc study. This information can be useful for assess- ing whether those who joincd thc trial are systematically different from those who did not,

Two factors which may influence recruitment to a trial are the method of approach to and the information given to potential participants (Question 4). There are diffcrences of opinion about the nature and timing of consent to par- ticipation in a trial and they will obviously vary depending on the naturc of the trial; both should be outlined in the report.

Random allocation is the only strategy for eliminating biased treatment assignment in con- trolled trials. Some methods of 'random' assign- mcnt are morc prone to corruption than others (Chalmers er ul. 1983). Methods of assignmcnt which allow identification of the treatment bcfore formal trial entry-for example. alter- nate allocation, or case note numbers, or date of birth-are much less satisfactory than mcthods which require formal registration of entry bcforc random allocation of treatment. This is thc reason why a brief description of the actual way in which the treatments were assigned is csscn- tial (Question 5 ) . Knowledge of thc treatment allocation by either the subject or thc investi- gator may also change the intervention (Spriet CQ Simon 1985)-see below. Chance imbalances in important prognostic variables, a problem which

397

398 A . Grant

Table 1. A chccklist for reports of controlled trials

introduction to the report 1 . Is there an adequate description of the rationale for and objective of the trial in terms of hypothesized effect(s)

of specific interventions?

Description of materials and methods

and duration of recruitment? 2. Is there an adequate description of the source of participants (hospital, outpatient clinic, etc.) and the timing

3. Is there an adcquate description of the entry and exclusion criteria?

4. Has the method of approach to potential participants and the information given to them been described?

5. Is thcrc a satisfactory description of the actual way in which the treatments were assigned, and the usc of

6. Have the forms of care compared (the treatment regimens), both expcrimental and control, been described in

7. Has the degree of masking (‘blinding’), if any, of participants and investigators been described?

8. If a placebo was used, was there an assesbent of its succcss in ‘masking’ the nature of the treatment?

9. Have the methods used to measure outcome been described, specifying whether or not the assessor knew the

prognostic stratification, if any?

sufficient detail to allow replication?

treatment allocation (‘degree of masking’)?

10. Has the objective been specified in terms of a quantified effect on a defined primary measure of outcome?

11. Is there an explanation of how the final sample size was chosen and a statement on statistical power in respect

12. If there were any interim analyses, have the arrangements and methods used been described?

13. Have all the statistical methods been identified and is there a description of any statistical techniques used

of the quantified effect on the primary measure of outcome?

which are not in common use?

Presentation of results 14.

15.

16.

17.

18.

19.

20. 21.

22.

Have the total numbers of participants entered into each of the comparison groups been presented?

Have the characteristics of the groups at entry been tabulated so that their comparability can be checked?

Has the cxtcnt of adherence to study regimens (numbers completing treatment, numbers complying with regimen, numbers ‘crossing-over’ to receive treatment opposite to that allocated-‘contamination’) been reported satisfactorily, for each trial group separately?

Have the reasons for withdrawals been reported, for each trial group scparately?

Has the comparability of the trial groups in respect of other rclcvant treatments (co-interventions) been described?

Have the primary analyses been based on all subjccts as allocated, i.e. on ‘intention-to-treat’, with non- compliers and ‘crossovers’ retained in the group to which they were originally assigned?

Have the estimated sizes of any differential treatment effects been accompanied by confidence intervais?

€Iave complications and sidc-cffccts bccn reported, for each trial group separately?

Have the implications of any imbalances in prognostic variables been considered?

Study conclusions 23. Have the conclusions been based on the analysis?

24. Have competing explanations for thc study outcome been discussed and, where appropriate, explored in

25. Has the statistical power of the completed study to detect or rule out differences been discussed?

26. Has a distinction been drawn between prior hypothcscs and post hoc findings which were not prespecified?

27. Have the study results been placed in the context of existing findings?

28. Have the implications for clinical practicc and future research been spelled out?

additional statistical analyses?

Reporting controlled trials 399

is more common in small trials, may be pro- tected against by prognostic stratification (Pocock 1983; Meinert 1986) and, if so, this should be described (Question 5) .

If the reported results of an evaluation of alternative forms of care are to be of use to its readers, the forms of care compared must be described in sufficient detail to allow the results of the study to be implemented in clinical prac- tice or replicated in a further study (Question 6).

Knowledge of the treatment to which a par- ticipant has been allocated may lead to imbalance in other treatments (‘co-intcrven- tion’) by investigators, to psychologically medi- ated effects of care in participants, and to biased observations by those assessing the outcomcs of treatment. For thcse rcasons the extent of mask- ing (‘blinding’) of participants, investigators and outcome assessors should be described (Ques- tions 7, 8, 9). Liability to biased assessment of outcomc also depends on what measurement is being used. Unambiguous measures such as death and caesarean section are less prone to bias than ‘softer’ measures (Thacker 1985).

The objective of a controlled trial should he specified in terms of a quantified effect on a defined primary measure of outcome (Question 10). The importance of statistical power to iden- tify such an effect is often not sufficiently appre- ciated (Freiman et al. 1978; Dctsky & Sackett 1985) when cvaluating a sample size in a ran- dornizcd controlled trial (Question 11). There are two major problems with trials in which the sample size is too small. Firstly, they tend falsely to ascribe real differences between treatments to chance (Type I1 error-Freiman et al. 1978). Secondly, they may grossly overestimate the true effect of an intervention when the differ- ence observed in the trial is statistically signifi- cant. A lull discussion of sample size is available elsewhere (Pocock 1983; Meinert 1986).

The use of multiple tests of statistical signifi- cance increases the risk of falsely claiming that an observed difference is real when, in truth, it solely reflects chance (Type 1 error) (Pocock et al. 1987). lnterim analyses or analyses involving multiple end-points (Question 12) therefore carry this risk unless appropriate corrections are made (Pocock 1983).

All statistical methods should be identified and unusual techniques should be described in sufficient detail to allow their replication (Ques- tion 13).

Presentation of results

Description of the trial groups at entry (Ques- tions 14 and 15), with tabulation of the major prognostic variables, should provide reassuring evidence that the allocation was indeed random. It also allows a check that the groups do not differ importantly in one or more prognostic variables.

Non-compliance, crossover to receive the alternative treatment (‘contamination’), and imbalance in other treatments (‘co-interven- tions’) may all alter the nature of the interven- tions being compared (Questions 16, 17, 18).

It is comparison of the groups characterized by the random trcatrnent allocation at entry which is free of selection bias (provided that treatment assignment was truly randomized). Withdrawals after entry, particularly if they dif- fer between the groups, or the reasons are related to a specific treatment, may introduce selection bias (Gail 1985); the likely extent of this can only he assessed if the number and rea- sons for withdrawals arc listed for each trial group separately (Question 17). This is the basis for recommending that primary analyses should be based on the groups as allocated (Question 19), regardless of subsequent management (‘intention-to-treat’ analyses). The manoeuvres compared in an ‘intention-to-treat’ analysis are thus the actual managements used in the trial (Questions 16 and 18), rather than the strict study regimens (Question 6). A possible excep- tion to the rule about ‘intcntion-to-treat‘ primary analyses is the explanatory trial (Sch- wartz et al. 1980), a small, tightly controlled trial in an ideal ‘laboratory-type’ setting. Even in these circumstances an ‘intention-to-treat’ analysis can be helpful to assess the likely size of any bias due to withdrawals.

Most comparisons of trial groups are better presented as estimations of a trcatment effect (c.g. diffcrcnccs in means for continuous vari- ables, odds ratios or relative risks for categorical data), with confidence intervals, than as a test of a hypothesis by generation of a P value (Gardner & Altman 1986) (Question 20). These estima- tions should be presented.

The outcomes assessed should include poss- ible side-effects and complications (Question

It may seem sensible to adjust for chance imbalances in prognostic variables in secondary analyses after the trial is completed (Question

21).

400 A. Grant

22). These secondary analyses should be used to check whether the imbalance makes any impor- tant difference to the conclusions drawn from the study.

Study conclusions

Questions 23-28 concern the validity of the con- clusions based on the analyses presented in the report. It goes without saying that these should be supported by the data presented (Question 23) and that a range of possible interpretations of the rcsults of thc controlled trial should have been considered (Question 24). If no clear difference is observed between the trial groups there should be comment (Question 25) on the statistical power of the study to detect or rule out the differences which were prespecified in the trial’s objectives (Detsky & Sackett 1985).

Post hoc findings which are ‘statistically sig- nificant’ should be given much less weight than prespecified hypotheses which are supported by a statistically significant difference (Question 26).

Finally, it is important that the results of the study should bc placed in the context of existing findings (Question 27), and that the implications for clinical practice and future research should be spelled out (Question 28).

References

British Medical Juurizul (1987) Guidelines for writing papers. Br Med J 294, 36-38.

Chalmers I. (1989) Evaluation of care during peg- nancy and childbirth. In ESfctive Cure in Preg- nancy arid Childbirth (Chalmers I . , Enkin M. & Keirse M. J . N.C.. eds). Oxford University Press, Oxford (in press).

Chalmers T. C., Smith H., Blackhurn H.. Silverman B.. Schroeder B., Reitmarr D. & Ambroz A. (1981) A method for assessing the quality of a randomized control trial. Controlled Clzn Trials 2, 31-49.

Chalmers T. C., Celano P., Sacks H. S. & Smith I i . (1983) Bias in treatment assignment in controlled clinical trials. N Engl J Med 309, 1358-1361.

Der Simonian R., Charette L. .I., McPeek B. &

Mosteller F. (1982) Reporting on methods in clini- cal trials. N Engl J Med 306, 1332-1337.

Detsky A. S. & Sackett D. L. (1985) When was a ‘negative’ clinical trial big enough? Arch Intern Med 145,709-712.

Emerson J . D., McPcek R. & Mostellar F. (1984) Reporting clinical trials in general surgical journals. Surgery 95, 572-579.

Frciman J. A , , Chalmers T. C . , Smith H. & Kuebler R. R. (1978) The importance of beta, the Type I1 error and sample size in the design and interprcta- tion of the randomized control trial. N Engl J Med

Gardner M. J. Pr Altman D. G. (1986) Confidence intervals rather than P values: estimation rather than hypothesis testing. Br Med J 292, 746-750.

Gail M. H. (1985) Eligibility exclusions, losses to follow-up, removal of randomized patients, and uncounted events in canccr clinical trials. Cancer Treuf Rep 69, 1107-1113.

Grant A. Sr Chalmers I. (1985) Some research strat- egies for investigating aetiology and assessing the effccts of clinical practice. In Scientific Basis of Obstetrics and Gynaecology, 3rd cdn (Macdonald R. R.. ed.), Churchill Livingstone. London. pp.49- 84.

McMaster University Department of Clinical Epi- demiology and Biostatistics (1981) How tci read clinical journals. V. To distinguish useful from use- less or even harmful therapy. Can Med Assoc J 124,

Mahon W. A. & Daniel E. E. (1963) A method for asscssmcnt of reports of drug trials. Cart Med

Meinert C. I>. (1986) Clinical Trials-Design, Conducr und Analysis. Oxford University Press, Oxford.

Mostcller F., Gilbert J . P. & McPeek B. (1980) Reporting standards and research strategies for controlled trials. Controlled Clin Triab 1, 37-58.

Pocock S. J. (1983) Clinical Trials-A Practical Approach. John Wiley & Sons. Chichester.

Pocock S. J.. Hughes M. D. Pr Lee R. J. (1987) Statis- tical problems in the reporting of clinical trials. N

Schwartz D., Flamant K. & Lellouch S. (1980) Clinical Triuls. Academic Press, London.

Spriet A. & Simon P. (eds) (1985) Single blind and double blind trials. In Meihodology of Clinical Drug Trials. Karger, Basel.

Thacker S. R . (198.5) Quality of controlled clinical trials. The case of imaging ultrasound in obstetrics: il review. Br J Ohstet Gynaecol92,437-444.

Zelen M. (1 983) Guidclincs for pu hlishing papers on cancer clinical trials: responsibilities of editors and authors. J Cliri Oricol 1, 164-169.

299, 690-694.

1156-1 162.

ASSOC I90, 565-569.

Etlgl J Med 317, 426-432.