Vaccine 22 (2004) 2073–2075

The role of editorial peer review in the evaluationof vaccine safety

Tom Jefferson∗Cochrane Vaccines Field, 00061 Anguillara Sabazia, Rome, Italy

Keywords: Peer review; Quality; Vaccine safety; Systematic reviews

1. Peer review—its strengths and weaknesses

Editorial peer review has been defined as “the systemwhereby scientific manuscripts are reviewed by outsideexperts”[1] or as a “filter for selection and a quality controlmechanism”[2]. The main purposes of editorial peer revieware aiding the selection of manuscripts of sufficient scientificmerit and the improvement of those published. Thus, peerreview should achieve its aims by identifying submissionswhich are irrelevant to the journal, trivial, methodologicallyweak, misleading, or have potentially harmful content, andby improving the clarity, transparency, accuracy, and utilityof the selected submissions[3].

Despite such laudable aims and its widespread use in sci-entific literature, there is sparse empirical evidence to sus-tain the effectiveness of peer review or its ability to fulfillits aims[4]. Systematic reviews of empirical studies assess-ing the evidence of effectiveness of the peer review systemhave found overall patchy and circumstantial evidence[3,4].Available evidence is mainly related to the assessment ofsurrogate outcomes of effectiveness such as the speed ofmanuscript turnaround or masking the identity of reviewersto authors and editors or vice versa[4]. However, two of theprocesses of peer review, the use of checklists for structuredreporting and reviewing submissions and technical editing(e.g. copy editing with extensive cross-checks of text, refer-ences and tables) were found to have reasonable empiricalevidence of effectiveness[3–5].

In summary, peer review as a mechanism for quality as-surance of content and presentation has a weak empiricalbasis with the possible exception of the use of checklists andtechnical editing.

∗ Tel.: +39-06-999-00-989; fax:+39-06-999-00-989.E-mail address: toj1@aol.com (T. Jefferson).

2. The quality of evidence of safety of vaccines

Next, I would like to examine the issue of the quality ofstudy methods and reporting of safety outcomes in publishedvaccine studies. Vaccine safety literature is a vast and het-erogeneous topic which has never been completely system-atically reviewed. Evidence of safety, usually in the guiseof reports of adverse events, may be included in a varietyof types of studies, ranging from double-blind randomisedtrials to case-series (Table 1). The studies are diverse in ob-jectives and rationale and consequently in design.

3. Methodological study quality

With such a diverse range of study designs, not surpris-ingly methodological study quality appears variable, both inrelation to the vaccine studied and study design. A recentsystematic review of evidence of safety of MMR assessedthe methods used in nine cohort studies (Table 2) [7]. Allstudies suffered from the likelihood of bias, making the in-terpretation of data and their results difficult. The commonweakness appears to be the withholding of part of the re-sults for unexplained reasons, thus undermining the possi-bility of generalising from the study cohort to the referencepopulation. Large well-funded pivotal studies of DTP vac-cines have however been well conducted and similarly welland exhaustively reported[8].

4. Reporting of adverse events

Reporting of adverse events is known to suffer from a lackof standardisation of both definitions and format. Evidencefrom a systematic review of DTP vaccines illustrates thepoint in relation to the outcome “fever” in the comparison

0264-410X/$ – see front matter © 2004 Elsevier Ltd. All rights reserved.doi:10.1016/j.vaccine.2004.01.012

2074 T. Jefferson / Vaccine 22 (2004) 2073–2075

Table 1Sources of published evidence of safety. Available evidence on the quality of methods (such as measurement) and reporting of adverse events, mainlygathered by systematic reviews and through the efforts of the Brighton Collaboration[6] points to considerable variability

Type of study Design Focus Safety data

Experimental Randomised, quasi-randomised Mainly effectiveness Often not reported or short term, frequent outcomes onlyNon-experimental

analyticalCohort, case-control,cross-sectional

Mainly adverse events,often single outcomes

Specifically collected to assess association

Non-randomisedcomparative

Case cross-over, time series (beforeand after), indirect comparisons

Adverse events Not specifically collected to assess association

Non-comparative Case reports, case-series, timetrend analysis

Single adverse events Reported as a first warning

Table 2Types of bias and their possible impact in nine cohort studies included in a review of safety of MMR (based on Jefferson, Price Demicheli and Bianco)

Study Type of bias Explanation

Weibel [9] Selection The report mentions 12 studies, one carried out in the USA (data are reported) and 11 in Costa Rica (data arenot reported). The basis for the selection is not explained

Stokes[10] Selection The study reports two comparisons: 685 children from Philadelphia with 106 unvaccinated siblings and 457children from Costa Rica with 175 ‘contact’ controls in Costa Rica. The basis for the selection of controls isnot explained and the control populations are not described

Swartz[11] Reporting Safety outcomes between the period of 7–15 days post-immunisation only are reported. It is unclear ifsigns/symptoms occurred exclusively during this period or whether it is a partial presentation of the data.

Robertson[12] Selection Allocation to either MMR or measles vaccines was on the basis of parental choice. Arms are grossly unequal innumber (263 and 56 children, respectively)

Benjamin[13] Selection Response rate adjusted for incorrect address 78% in vaccinated and 64% in non-vaccinated. No attempt toidentify non-responders are reported

Dunlop [14] Selection Allocation to either MMR or measles vaccines was on the basis of parental choice. Arms are grossly unequal innumber (319 and 16 children, respectively)

Makino [15] Reporting Inadequate description of the cohorts derivation. Clinical reactions are reported only for sub-subgroups. There isno discussion of the significance of such a choice

Miller [16] Selection andreporting

The study was carried out on 1–2 year olds and 3–4 year olds with no explanation of the basis for selection.The observations from the 2–3 year olds are not reported and no explanation is given in the text

Beck [17] Selection andreporting

The report mentions seven studies but reports sufficient data from one. There is no population description, noexplanation of how the groups were selected or assigned, no reports of the study population size or of howmeasurements were made

between two-component DTaP and DTwP vaccines (Table 3)[8].

As can be seen, definitions of raised temperature and re-porting times are highly heterogeneous. What is not appar-ent however, is the equally diverse means of reporting howtemperatures were taken, which instruments were used andhow the data were reported. The latter varied from report-ing incidence in numerical or histogram form, to separate oraggregate but incomplete reporting of numerators and de-nominators.

Variable methodology and heterogeneous and inconsistentreporting thus conspire to reduce availability of comparabledata from which inference can be made.

5. Editorial peer review and the enhancement ofquality of vaccine safety data

What can be done to attempt to address the issue of vari-ability in the quality of methods and poor reporting? Edi-torial peer review may provide some of the answers in thesame way as it has improved reporting of randomised trials

[18]. This was achieved by the development and implemen-tation of the consolidated standards of reporting of trials(CONSORT) statement. The aim of CONSORT and its at-tendant checklists and flow diagrams is to facilitate criticalappraisal and interpretation of trials by providing guidanceto authors on how to improve reporting of their trials[19].

The CONSORT output consists essentially of two parts:a detailed list of items (from the title to the explanation ofresults sections) in the reporting of trials, with a suggestedreporting format and explanation of the importance of theitem and rationale for the suggestion. The CONSORT ef-fort is supported by some of the major biomedical journals.Conformity to the CONSORT has been adopted by over300 journals as a requirement for submission. The adoptionhas proved efficacious in improving the quality of reporting[8]. Thus, the management of journals submissions broadlyknown as editorial peer review can lead to improvements inquality of evidence.

I propose the development of study design-specific guide-lines for the conducting and reporting of vaccine studieswith safety outcomes. Guidelines should be accompaniedby summary checklists similar to the CONSORT statement.

T. Jefferson / Vaccine 22 (2004) 2073–2075 2075

Table 3Definition and reporting of fever in 27 trials comparing two-componentacellular DTP with whole cell DTP vaccines

Temp >38.3◦C at 24 hTemp >38◦C at 24 hTemp >38◦C at 48 hTemp >38◦C at 72 hTemp >38◦C, <40◦C at 48 hTemp >37.5◦CTemp >37.5◦C at 7 daysTemp >38.3◦C at 72 hTemp >39◦C at 24 hTemp >39◦C at 72 hTemp >40.1◦C at 72 hTemp= 37.8–38.3◦C at 72 hTemp= 38.1–39 at 24 hTemp= 38.1–39◦C at 72 hTemp= 38.3–39◦C at 72 hTemp= 38.4–38.9◦C at 72 hTemp= 38.9◦C at 72 hTemp= 39.1–40.4◦C at 72 hTemp >37.8◦C at 48 hTemp >38.4◦C at 48 hTemp >40.5◦C at 24 hTemp<40◦C at 48 hTemp at 48 hTemp at 72 hTemp at 7days

Temp: temperature.

Guidelines should clarify and list a set of methodologicalpoints to be followed and illustrated when reporting the con-duct of a study. Standardised definitions and rules of report-ing of adverse events produced by The Brighton Collabora-tion would be suggested for use in the guidelines[20]. Someof the work has already been undertaken by the BrightonWorking Groups producing the first crop of definitions andguidelines of adverse events. Standardised definitions andtheir reporting, however, need to be developed within thecontext of standardised reporting of each study. This in turnwill depend on the study design.

6. Conclusions

Editorial peer review is the main mechanism to control thequality and format of published output. One way to addressthe variability in the quality of methods of safety assessmentand the heterogeneity of data reporting is to agree on rules forthe reporting of studies incorporating standardised adverseevent definitions.

Such agreement must come following an international col-laborative effort involving the vaccine science communityand editors of journals.

References

[1] Rennie D. Editorial peer review: its development and rationale. In:Godlee F, Jefferson TO, editors. Peer review in health sciences.London: BMJ Books; 1999. p. 3–13.

[2] Wager E, Godlee F, Jefferson TO. How to survive peer review.London: BMJ Books; 2002. p. 3.

[3] Jefferson TO, Wager E, Davidoff F. Measuring the quality of editorialpeer review. JAMA 2002;287:2786–90.

[4] Jefferson TO, Alderson P, Davidoff F, Wager E. Effects of editorialpeer review: a systematic review. JAMA 2002;287:2784–6.

[5] Wager E, Middleton P. Technical editing of research reports inbiomedical journals (Cochrane Methodology Review). In: TheCochrane Library, Issue 1; 2003. Chichester, UK: John Wiley &Sons, Ltd. MR000002.

[6] Bonhoeffer J, Kohl K, Chen R, Duclos P, Heijbel H, HeiningerU, Jefferson T, Loupi E, The Brighton Collaboration. The Brightoncollaboration enhancing vaccine safety. Vaccine [this issue].

[7] Jefferson TO, Price D, Demicheli V, Bianco E. Unintended eventsfollowing immunisation with MMR: a systematic review. Vaccine2003;21(25–26):3954–60.

[8] Jefferson TO, Rudin M, Dipietrantonj C. Systematic review of theeffects of DTP vaccines in children. Vaccine 2003;21(17–18):2012–23.

[9] Weibel RE, Carlson Jr AJ, Villarejos VM, Buynak EB, McLeanAA, Hilleman MR. Clinical and laboratory studies of combined livemeasles, mumps, and rubella vaccines using the RA 27/3 rubellavirus (40979). Proc Soc Exp Biol Med 1980;165(2):323–6.

[10] Stokes J, Weibel RE, Villarejos VM, Arguedas JA, Buynak EB,Hilleman MR. Trivalent combined measles-mumps-rubella vaccine.Findings in clinical-laboratory studies. JAMA 1971;218(1):57–61.

[11] Swartz TA, Klingberg W, Klingberg MA. Combined trivalent andbivalent measles, mumps and rubella virus vaccination. A controlledtrial. Infection 1974;2(3):115–7.

[12] Robertson CM, Bennett VJ, Jefferson N, Mayon-White RT.Serological evaluation of measles, mumps and rubella vaccine. ArchDis Child 1988;63:612–6.

[13] Benjamin CM, Chew GC, Silman AJ. Joint and limb symptomsin children after immunisation with measles mumps, and rubellavaccine. BMJ 1992;304(6834):1075–8.

[14] Dunlop JM, RaiChoudhury K, Roberts St JC, Bryett KA. Anevaluation of measles, mumps and rubella vaccine in a populationof Yorkshire infants. Public Health 1989;103(5):331–5.

[15] Makino S, Sasaki K, Nakayama T, Oka S, Urano T, Kimura M, etal. A new combined trivalent live measles (AIK-C Strain), mumps(Hoshino Strain), and rubella (Takahashi strain) vaccine. Am J DisChild 1990;144:905–10.

[16] Miller C, Miller E, Rowe K, Bowie C, Judd M, Walker D.Surveillance of symptoms following MMR vaccine in children.Practitioner 1989;233(1461):69–73.

[17] Beck M, Welsz-Malecek R, Mesko-Prejac M, Radman V, JuzbasicM, Rajninger-Miholic M, et al. Mumps vaccinel-Zagreb, preparedin chick fibroblasts. I. Production and field trials. J Biol Stand1989;17(1):85–90.

[18] Moher D, Jones A, Lepage L. Use of the CONSORT statement andquality of reports of randomized trials: a comparative before-and-afterevaluation. JAMA 2001;285(15):1992–5.

[19] http://www.consort-statement.org(accessed 24 April 2002).[20] http://www.brightoncollaboration.org(accessed 22 March 2002).