www.metcardio.org surviving the peer review process giuseppe biondi zoccai division of cardiology,...
TRANSCRIPT
www.metcardio.org
SURVIVING THE PEER REVIEW PROCESS
Giuseppe Biondi ZoccaiGiuseppe Biondi Zoccai
Division of Cardiology, University of Turin, Turin, ItalyDivision of Cardiology, University of Turin, Turin, Italy
Meta-analysis and Evidence-based medicine Training in Meta-analysis and Evidence-based medicine Training in Cardiology (METCARDIO), Ospedaletti, ItalyCardiology (METCARDIO), Ospedaletti, Italy
www.metcardio.org
LEARNING GOALS
• What is peer review, and why have we to survive it through?
• What should you do and not do when actually peer reviewing other colleagues’ works?
• What should you do and not do when surviving through peer review?
• Case studies
www.metcardio.org
WHY AM I GIVING YOU THIS LECTURE?Peer reviews I have completed since 2003
expected
www.metcardio.org
LEARNING GOALS
• What is peer review, and why have we to survive it through?
• What should you do and not do when actually peer reviewing other colleagues’ works?
• What should you do and not do when surviving through peer review?
• Case studies
www.metcardio.org
DEFINITION
Peervb intr. 1. to look intently with or as if with
difficulty. 2. to appear partially or dimly.n. 1. a person who is an equal in social standing,
rank, age, etc.
Collins Dictionary of the English Language,London & Glasgow: Collins, 1979
www.metcardio.org
ORIGINS OF PEER REVIEW• Peer review was born in the 17th century in the UK,
likely thanks to Henry Oldenburg, the founder of Philosophical Abstractions (1665)
• He originally introduced the practice of soliciting opionions on manuscripts from more knowledgeable external colleagues
www.metcardio.org
NOTABLE EXCEPTIONS• Not all manuscripts undergo thorough peer
review. Notable examples?– Albert Einstein’s “Annus Mirabilis”
papers, published in 1905 in Annalen der Physik by Max Planck (father of quantum theory and Nobel prize winner), who read the papers and decided altogether to publish them
– more humbly… Biondi-Zoccai et al, Int J Cardiol 2005;100:119-23
www.metcardio.org
WHAT’S WRONG WITH PEER REVIEW?
• Unreliable• Unfair• Fails to truly validate or authenticate• Unstandardized• Idiosyncratic• Open to every sort of bias
www.metcardio.org
WHAT’S EVEN WORSE?• Stifles innovation• Perpetuates the status quo• Rewards the prominent but punishes the
weak• Unnecessarily delays dissemination• Very expensive• Insufficiently tested
www.metcardio.org
YES, BUT…
• It has been said that democracy is the worst form of government except all those other forms that have been tried from time to time – W. Churchill
• The same applies to peer review
www.metcardio.org
WHAT ARE ITS PROS?• Filters bad from good, and better from less good,
as well as possibly identifying misleading and false research
• Protects patients• Guides authors to improve the quality of their
article, and improve their whole research approch• Authenticates work, assuring quality• Improves readability• Broadens participation and dialogue
www.metcardio.org
PARAPHRASING GROUCHO
I would never enter into a club that would accept me as a member…Groucho Marx, 1980-1977
www.metcardio.org
LEARNING GOALS
• What is peer review, and why have we to survive it through?
• What should you do and not do when actually peer reviewing other colleagues’ works?
• What should you do and not do when surviving through peer review?
• Case studies
www.metcardio.org
TYPICAL REVIEW FLOW
www.metcardio.org
ARE PEER REVIEWERS UNBIASED?
Mahoney et al, CTR 1977
www.metcardio.org
WILL PEER REVIEW LET YOU SLEEP?
McNutt et al, JAMA 1990
On average a review will take you around 3 hours
(actually I usually complete one in 45’)
www.metcardio.org
SHOULD YOU BE BLINDFOLDED?
Godlee et al, JAMA 1998
www.metcardio.org
WHO ARE THE BEST REVIEWERS?
Kliever et al, AJR 2005
www.metcardio.org
WHO ARE THE BEST REVIEWERS?
Kliever et al, AJR 2005
Other (also weak) predictors:
•Coming from good institutions
•Known to the editors
•Had methodological training (statistics & epidemiology)
www.metcardio.org
WHO ARE THE WORST REVIEWERS?
Opthof et al, Cardiovasc Res 2002
Italian peer reviewers don’t like Italian
manuscripts!
www.metcardio.org
THREE PIVOTAL QUESTIONS FOR PEER REVIEW
• Do I understand it? Are the question and the methods clearly explained?
• Do I believe it? Are the conclusions justified by the data and are the methods valid?
• Do I care? Is the question important and interesting?
Wager et al, How to survive peer review. BMJ Books 2002
www.metcardio.org
CHECKLIST FOR PEER REVIEWERS: ISSUES FOR COMMENT
• Importance of research question• Originality of work• Delineation of strengths and weaknesses of
methodology/experimental/statistical approach/interpretation of results
• Writing style and figure/table presentation• Ethical concerns (animal/human)
Benos et al, Advan Physiol Educ 2003
www.metcardio.org
SUBTLETIES OF PEER REVIEW: PRIORITY – i.e. how to kill a paper
• Priority means novelty, originality, and likelihood of generating interest, irrespective from quality, validity, and methodology
• Manuscript can be judged as low, mid, high, or top priority
• Some journals (e.g. the Journal of Cardiovascular Medicine) use scores, e.g. ranging from 0 to 100, with 100 meaning top priority
www.metcardio.org
• Decision means recommending a specific editorial handling of the manuscript, and can be distinguished in:– accept as is– accept/reconsider after minor revisions– accept/reconsider after major revisions– reject but reconsider on a de novo basis– reject
SUBTLETIES OF PEER REVIEW: DECISION
www.metcardio.org
SUBTLETIES OF PEER REVIEW: DECISION
• Accept as is: it can be published as it stands• Accept after minor revisions: it can be published
EVEN IF not all my comments are taken into account• Accept/reconsider after major revisions: it can be
published ONLY IF all my comments are taken into account
• Reject but reconsider on a de novo basis: it must be changed altogether, and priority also reappraised after resubmission
• Reject: just send it back, it ain’t worth it
www.metcardio.org
SUBTLETIES OF PEER REVIEW: DECISION
• Accept as is: it can be published as it stands• Accept after minor revisions: it can be published
EVEN IF not all my comments are taken into account• Accept/reconsider after major revisions: it can be
published ONLY IF all my comments are taken into account
• Reject but reconsider on a de novo basis: it must be changed altogether, and priority also reappraised after resubmission
• Reject: just send it back, it ain’t worth it
www.metcardio.org
TYPICAL EDITOR’S EVALUATION OF PEER REVIEW AND PEER REVIEWER• Thoroughness and comprehensiveness• Timeliness• Citing appropriate evidence to support comments
made to author• Providing constructive criticism• Objectivity• Clear statement to editor as to the
appropriateness and priority of research for publication
Benos et al, Advan Physiol Educ 2003
www.metcardio.org
LEARNING GOALS
• What is peer review, and why have we to survive it through?
• What should you do and not do when actually peer reviewing other colleagues’ works?
• What should you do and not do when surviving through peer review?
• Case studies
www.metcardio.org
DON’T PANIC
www.metcardio.org
SHOULD YOU TRUST PEER REVIEWERS?
Opthof et al, Cardiovasc Res 2002
www.metcardio.org
IDENTIFYING THE MOST APPROPRIATE TARGET
Wager et al, How to survive peer review. BMJ Books 2002
www.metcardio.org
IDENTIFYING THE MOST APPROPRIATE TARGET
Wager et al, How to survive peer review. BMJ Books 2002
www.metcardio.org
COVER LETTER
www.metcardio.org
AFTER SUBMITTING PAPER
• Store all data and documentation surrounding data analysis in durable and appropriately referenced form
• Store the original data—questionnaires, data collection sheets, CD’s
• Keep as long as readers may reasonably expect to ask questions where you need to reference this data
www.metcardio.org
AFTER SUBMITTING PAPER
• Store 5-10 years--normal• All references to where data held and how
archived logged in handbook• Include all details so study can be repeated• Names, locations of electronic data files• Data bases, data recodes, data analysis
www.metcardio.org
AFTER SUBMITTING PAPER
• Once paper submitted now property of journal
• Editor has total discretion over who has access
• Peer editors sometimes pass on papers to colleagues for review
• Confidentiality not always maintained
www.metcardio.org
REPLYING TO REVIEWERS COMMENTS
• Do not ignore comments, but make most if not all of changes
• Be calm, objective, and polite even if reviewer’s comments were harsh
• Deconstruct each of the messages into individual items
• Respond to each item thoughtfully• Make responses clear
www.metcardio.org
REPLYING TO REVIEWERS COMMENTS
• Don’t have to fully accept suggestions but must give reasons that will convince editor your opinion is reasonable
• Be pragmatic and not dismissive of reviewer’s work
• Explain just enough to enable you to survive• Benefit from it, and learn also how to become
a competent reviewer
www.metcardio.org
REBUTTAL LETTER
www.metcardio.org
AFTER RESUBMISSION OF REVISION• No guarantee will be published• Editor will consider new version and your replies
to comments• Editorial process can be subjective, and
sometimes downgrade priority -> rejection• Reviewer’s comments only one factor• Editor may reject paper even if reviewer’s
comments were minor• Editor has absolute discretion
www.metcardio.org
IF REJECTED CAN APPEAL
• If you think reviewer’s overlooked or misunderstood something important
• Appeal by writing a letter stating your case –> rebuttal/complaint letter
• Rare decision overturned but it does happen• If appealing—send new copy of paper—
rejected papers do not remain on file (i.e. de novo submission)
www.metcardio.org
WHAT TO DO AFTER REJECTION• Reviewers critical of basic methods—may
need to rethink study and do further data analysis
• Reviewers critical on style and presentation—fix problems before resubmitting to another journal
• Three repeat rejections—completely reassess entire approach or search for appropriate target
www.metcardio.org
WHEN ACCEPTED
• Receive page proofs—typeset copy of work—how looks in journal
• May take several months to receive• Time for final check• Journal usually send proof reading instructions
you must follow• Usually standard proofreading marks
www.metcardio.org
TYPICAL REASONS FOR FINAL REJECTION
1. Fundamentally weak hypothesis2. Lack of clinical relevance3. Old knowledge with no new or useful material4. Two or three of reasons 1–35. Reasonable text, but images are of very poor quality, are
inappropriate, or are incorrectly interpreted6. Too many methodologic errors7. Hypothesis adequate, but poor study design, methodology, or
statistics8. Lacking in logic; initial premise not logically supported by methods
and results
Ehara et al, AJR 2007
www.metcardio.org
TYPICAL REASONS FOR FINAL REJECTION (CONT.)
9. None of the other reasons, but reviewers do not like the article10. Previously published11. Sample population too small or biased to justify results and
conclusion12. Well written but better suited for another journal13. Major language problems; English not primary language of author14. Too poorly written, phrased, or presented15. Failure to follow journal guidelines16. Lack of correlation between purpose and results17. Poor statistics, beyond salvage
Ehara et al, AJR 2007
www.metcardio.org
TEN TIPS TO FOR SURVIVAL1. Properly organize the manuscript2. Clearly state the study question and rationale3. Explain methods in a systematic fashion4. Structure methods and results in a similar manner5. Make the discussion session concise6. Explain if – and why – your study results are important7. Avoid overinterpreting8. Explain the limitations9. Account for unexpected findings10. Fully incorporate reviewers’ suggestions
Provenzale et al, AJR 2007
www.metcardio.org
LEARNING GOALS
• What is peer review, and why have we to survive it through?
• What should you do and not do when actually peer reviewing other colleagues’ works?
• What should you do and not do when surviving through peer review?
• Case studies
www.metcardio.org
THE LOVERBOY REVIEWER
Lotrionte et al, Am J Cardiol 2008
www.metcardio.org
THE LOVERBOY REVIEWER
Lotrionte et al, Am J Cardiol 2008
www.metcardio.org
THE INQUIRING REVIEWER
Biondi-Zoccai et al, Am Heart J 2008
www.metcardio.org
THE INQUIRING REVIEWER
Biondi-Zoccai et al, Am Heart J 2008
www.metcardio.org
THE DELUSIONAL REVIEWER
Sheiban et al, J Am Coll Cardiol 2009
www.metcardio.org
THE DELUSIONAL REVIEWER
Sheiban et al, J Am Coll Cardiol 2009
www.metcardio.org
THE NASTY REVIEWER
Sheiban et al, J Interven Cardiol 2008
www.metcardio.org
THE NASTY REVIEWER
Sheiban et al, J Interven Cardiol 2008
www.metcardio.org
THE PESSIMISTIC REVIEWER
Biondi-Zoccai et al, Am Heart J 2008
www.metcardio.org
THE PESSIMISTIC REVIEWER
Biondi-Zoccai et al, Am Heart J 2008
www.metcardio.org
THE ASK-TOO-MUCH REVIEWER
27 COMMENTS JUST FROM
THE STATISTICAL REVIEWER!!!
HOW CAN YOU TACKLE THIS?
Biondi-Zoccai et al, BMJ 2006
www.metcardio.org
THE ASK-TOO-MUCH REVIEWER
27 COMMENTS JUST FROM
THE STATISTICAL REVIEWER!!!
HOW CAN YOU TACKLE THIS?WITH A LOT OF PATIENCE!!!
Biondi-Zoccai et al, BMJ 2006
www.metcardio.org
“There is no way to get experience except
through experience.”
www.metcardio.org
TAKE HOME MESSAGESWhat a very bad reviewer once told Amir Lerman (Mayo Clinic, USA):
Delete the first phrase of the second section of the Discussion (“It remains to be speculated that…”.Better, print it, put it in a frame and hang it in front of your desk to remind you never to repeat such mistake….
www.metcardio.org
TAKE HOME MESSAGES
• Active or passive peer review remains a time-consuming and painful process, but still the best available
• Active peer review is best summarized by Matthew 19, 16-19: Thou shalt love thy neighbor as thyself
• Passive peer review should be based on patience, perserverance, and sound methods
www.metcardio.org
Thank you for your attention
For any correspondence: [email protected]
For these and further slides on these topics feel free to visit the metcardio.org website:
http://www.metcardio.org/slides.html