scholarly communications: changes to peer review

Scholarly Communications:Changes to Peer Review

• Bradley Hemminger

• School of Information and Library Science

• University of North Carolina at Chapel Hill

Scholarly Communications Process

Idea

V1

Present to colleagues

V2

Present at conference

V3

Submit to journal V4

Referees Revision for journal V5

Journal Final Revision

V6

Revision to update analysis

V7

Revision to include additional new results V8

Scholarly Communications Process

Idea

V1


V2


V3




V6

Revision to correct analysis

V7


formulate discussion discussion,revision

Two peer reviews

CopyproofingCriticisms, new thoughts,revision

new results,revision

commentscommentscomments

Author revision

Scholarly Communications Process: What’s Produced


V6

Scholarly Communications Process:What I’d like to see saved!Idea

V1


V2


V3




V6


V7



Two peer reviews




Author revision

Peer Review Output

Review

(Peer)

With Respect to XYZ…Accept reject revise

Comments to Author

Qualitative Grade

Qualitative Comments

Article

Generalized Review Model

Review (open, peer, machine)

Accept,Reject,Revise,With respect to XYZ

Comments to Author

Qualitative Grade

Quantitative Grades

Score (1-10)


Article

Overview of Peer Review

Review

Peer, Open, Machine

Accept, reject, revise with respect to XYZ standards

Comments to Author


Quantitative Grade

Published Article

Article submitted

Send elsewhere

Filter

Reject

Score (1-10)

Qualitative Grade

General Review Model Parallels

• In general, you have sample (material) which is judged/scored quantitatively and qualitatively by an identified observer.

• Current Peer Review

• Automated Scoring Systems (lab tests)

• Moderated email lists (announce)

• Moderated Eprints servers (arXiv)

Peer Review Options• Human Judgement

– Expert peer review (status quo)– Certified expert peer review– Open Peer Review BMJ, BioMed– Open comment review pyscprints

• Computer Judgement– Computer peer review

• Human Usage– Citation-based (CiteSeer)– Usage counts (CiteSeer) Example– Quantity of discussion

• Coarse Categorization– Two Tier (grey/gold)– Moderator (current arXiv)– No review (old arXiv)

Quantitative

Score (1-10)Score (1-10)Score (1-10)Score (1-10)

Score (1-10)

#citations#hits#number of

related discussions

QualitativeRel Yes/No GroupRel Yes/No GroupAbsolute Absolute

Rel Yes/No GroupRel Yes/No Group

YYYYY

?

http://www.ama-assn.org/public/peer/arev.htm

http://citebase.eprints.org/

Judgment based on some combination of reviews/commentsIdea

V1


V2


V3




V6


V7



Two peer reviews




Author revision

What areas of improvement?

• Review Process Change

• Search, Retrieval Process Change

• Service Provider Process Change

Review Process Changes

• Include open reviews and comments to get additional feedback.

• Support normal scholarly discourse, allowing give and take with author responding.

• Add quantitative scores to allow better filtering based on quality during retrieval

• Add machine (automated) reviews

Search and Retrieval Changes

• Universal Archive: all material freely available.• Universal Searching: standardized metadata (

Dublin Core) for general searching.• Automated agents to bring material of interest to

your attention.• Use additional review scores (public reviews,

machine) to help filter search.• Example: article scores > 7.0, refereed,

citation count above X, type=research article, search terms = schizophrenia, geneX)

http://www.dublincore.org/documents/dces/

Provider Service Change

• What is worth paying for?– Quality review (Faculty of 1000)– Proofing, citation linking, professional presentation (

CiteSeer, Cite-base)– Archival (JStor)

• Who hosts material:– Society (arXiv)– Commerical Publishers (Elesiever,BioMedCentral)– University Library (MIT Dspace)

http://www.facultyof1000.com/start.asp

http://citeseer.nj.nec.com/cs

http://citebase.eprints.org/cgi-bin/search

http://www.jstor.org/search

http://arxiv.org/

http://www.biomedcentral.com/info/about/apcfaq

http://www.dspace.org/

New Frameworks for Peer Review

• As an enabling technology: frameworks like NeoRef supports all of the above models in any combination at the same time, while eliminating many of the costs.

• Requirements: Based on OAI and Dublin Core, and expectation of logical universal archive, an universal unique object IDs (URL, DOIs) and person IDs.

Example Model (NeoRef)• All material and metadata are author contributed to a

public OAI archive (author retains ownership).• All materials universally available via search engines that

harvest metadata from OAI archives.• OAI archives have automated or manual moderator to

filter out “junk”.• Everything--articles, reviews, comments, indexings, etc.,

are stored as digital content on archive using the same mechanism. Reviews contain quantitative score, qualitative grade, qualitative comments. Logically (although not physically), a two tier (Grey & Gold) system for materials– High quality keep forever material reviewed by known entity– Grey material (everything else)

NeoRef for Movies, Dates, DocSouth

• The same process used by NeoRef to support Scholarly Communication could be used for almost any purpose. All that is required is storage of Digital Content Items, and linking of reviews, comments, etc to them.

• Movies: Grey is everyone’s reviews; Gold is Siskel and Ebert reviews

• DocSouth: self cataloged and indexed items are Grey; librarian/archivist cataloged and indexed items are Gold.

Can we save the Gold and Grey?

Idea

V1


V2


V3




V6


V7



Two peer reviews

Author revision

Criticisms, new thoughts,revision



Copyproofing

NeoRef Storage Model

Conference paper (v3)

Comments on V6

Journal Submission V4

Journal Final Revision V6

Revision to include additional results and analyses V8

Auto-indexing

Material expressing content

Two peer reviewsLocal powerpoint Presentation v2

Comments on V3

Automated

Author Indexing

Recognized Expert

Open (anyone)Top Tier (Keep Forever)

Filter (Moderate)

Grey Literature

Author

Machine Review

What do users want?

The Association of Learned and Professional Society Publishers (ALPSP) Survey

Authors and Electronic Publishing• Scholarly research communication has seen far-

reaching developments in recent years. • Most journals are now available online as well as

in print, and numerous electronic-only journals have been launched;

• the Internet opens up new ways for journals to operate.

• Authors have also become conscious of alternative ways to communicate their findings, and much has been written about what they ought to think.

ALPSP felt that it would be timely to discover what they actually thought and what they actually did. This survey aimed to discover the views of academics, both as authors and as readers. Some 14,000 scholars were contacted across all disciplines and all parts of the world, and nearly 9% responded; their detailed comments make thought-provoking reading.

Alma Swan and Sheridan Brown. Authors and Electronic Publishing: The ALPSP Research Study on Authors' and Readers’ Views of Electronic Research Communication. (West Sussex, UK: The Association of Learned

and Professional Society Publishers, 2002).http://www.alpsp.org/pub5.htm

Importance of the Peer Review Process

0102030405060708090

100Peer-reviewed

Refs' commentspublished

Referees identified

Public commentary oneprints

Post-publication publiccommentary

Ability to submitcomments

http://www.alpsp.org/pub5.ppt

Importance of journal features

010

20

30

40

50

60

70

80

90

Citation links Additionaldata

Addit/colourimages

Manipulablecontent

Video/sound

Importance of the peer review process

0102030405060708090

100Peer-reviewed

Refs' commentspublished

Referees identified

Public commentary oneprints

Post-publication publiccommentary

Ability to submitcomments

Importance of publishers’ rolesFactor Responses as authors Responses as readers

Peer review 81 80

Gathering articles together to enable browsing of content

64 49

Selection of relevant and quality-controlled content

71 54

Content editing and improvement of articles

60 39

Language or copy editing 50 34

Checking of citations/adding links

46 28

Marketing (maximising visibility of journal)

44 20

Importance of future dissemination channels

Dissemination method Very important plus important

categories

Ranking

Traditional print + electronic journal 91 1

Discipline-based electronic reprint archive 78 2

Traditional print journal 77 3

Traditional electronic-only journal 66 4

Institution-based electronic reprint archive 60 5

New forms of electronic-only journal 49 6

Discipline-based electronic preprint archive

44 7

Institution-based electronic preprint archive

33 8

http://www.update-software.com/Cochrane/MR000016.pdf

Cochrane Methodology Review

• Despite its widespread use and costs, little hard evidence exists that peer review improves the quality of published biomedical research.

• There had never even been any consensus on its aims and that it would be more appropriate to refer to it as ‘competitive review’.

Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,”BMJ 326 (February 1, 2003): 241

http://bmj.com/cgi/reprint/326/7383/241/a.pdf

Cochrane Methodology Review• On the basis of the current evidence, ‘the

practice of peer review is based on faith in its effects, rather than on facts,' state the authors, who call for large, government funded research programmes to test the effectiveness of the [classic peer review] system and investigate possible

alternatives. Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,”

BMJ 326 (February 1, 2003): 241http://bmj.com/cgi/reprint/326/7383/241/a.pdf

Cochrane Methodology Review• The use of peer-review is usually

assumed to raise the quality of the end-product (i.e. the journal or scientific meeting) and to provide a mechanism for rational, fair and objective decision-making. However, these assumptions have rarely been tested.

Tom O. Jefferson, Phil Alderson, Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of

Biomedical Studies. (Middle Way, Oxford:Update Software Ltd, 2003).


Cochrane Methodology Review• The available research has not clearly

identified or assessed the impact of peer-review on the more important outcomes (importance, usefulness, relevance, and quality of published reports)

• … [G]iven the widespread use of peer-review and its importance, it is surprising that so little is known of its effects

Tom O. Jefferson, Phil Alderson,Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of

Biomedical Studies. (Middle Way, Oxford:Update Software Ltd, 2003).


FURTHERMORE …• 16% said that the referees would no

longer be anonymous• 27% said that traditional peer review

would be supplemented by post-publication commentary

• 45% expected to see some changes in the peer-review system within the next five years

Fytton Rowland, “The Peer-Review Process,” Learned Publishing 15 no. 4 (October 2002): 247-258.

Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdf

scholarly communications: changes to peer review

Documents