how to referee a technical paper saul greenberg university of calgary is this paper any good?

44
How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Upload: edward-holmes

Post on 16-Dec-2015

226 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

How to Referee a Technical PaperSaul Greenberg

University of Calgary

is this paper any good?

Page 2: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

The Message

Refereeing is excellent practice for

• developing critical appraisal skills

• understanding what makes good & bad papers

• understanding how others evaluate your work

Page 3: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Many articles on refereeing

• A Guide for New Referee in Theoretical Computer Science (Parberry)

• Rules for Referees (Forscher)

• Guidelines that accompany referee requests

Page 4: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Purpose of Refereeing

Quality control• eliminate bad papers

• choose best papers from a good set

• constraints dues tocompetition for space

Submissions295 papers

Proceedings(33 papers)

Page 5: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Referee Process

a peer review process

Page 6: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Source no longer known

Page 7: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

People involved

Author • correctness of argumentation and results • sound grounding in the literature• good quality of presentation• appropriate for the intended audience/venue

Page 8: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

People involved

Editor • whether paper should go out for review• choosing appropriate referees• acceptance/rejection decision• explanation letter

Page 9: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

People involved

Editorial support• good record keeping (copies of paper, reports)• tracking and distributing referee reports,

reminding referees

Page 10: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

People involved

Referee• critical appraisal of the paper’s contents• opinion, rationale, changes, suggested action • usually 3 referees/paper

A paperto referee

Page 11: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Referee types

• topic specialistso is/has worked on similar problemo knows literature, other work very wello understands methodologieso considers nuances of your work/contribution

• area specialistso knows general area, and how topic fits within ito considers contribution of work to the general areao evaluates comprehensibility by non-specialists

Page 12: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Why Referee?

Fairness• all your papers will be refereed• expected duty of all researchers/academics

Page 13: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Why Referee?

Upsides• good practice• enhance reputation • expedites processing of your own papers • get on editorial board or program committee • 'previews' to the state of the art

Downside• more work!

Page 14: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Quality Control: Research

Correctness • of argument/method/algorithm/

mathematics/proof... (is a fix necessary?)

Significance • does it work on a valid problem?• will these results make a difference?• is it significant to area/journal, etc...? • will it stimulate further work in the area?• is it more than an obvious/trivial solution?

Page 15: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Quality Control: Research

Innovation • original, creative, novel, inventive • not trivial extensions, or combination of old work with no

added value

Interesting • well motivated • relevant (when & where & to whom)

Page 16: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Quality Control: Research

Replication• can the work be reproduced from the description by an

experienced person in the area?

Timeliness • of current interest to the community • but account for:

o publication delay

Page 17: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Quality Control: Research

Previous publications• by other authors• by this author

o www publication, o minor conferenceso minor variations of the theme...

Page 18: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Quality Control: Writing

Succinct • message and arguments should be

o clear, compelling, to the point

• not o hand-waving o obscure/hidden behind jargon, etc.

Page 19: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Quality Control: …Writing

Accessible • is it appropriate to the audience?

o specialists & range of generalists o is there something for both?

Language & organization• readable, good grammar/structure reflects care

o people don’t have time to read badly written papers

Page 20: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Quality Control: …Writing

Use of figures/tables• supports the story

Title & abstract• indicates content, summarizes main points

English as a 2nd language?

Page 21: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Ethics: Professionalism

Act in the best interest of the author & paper

Page 22: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Ethics: Professionalism

Constructive critique• if acceptable, explain how it can be improved • if paper is unacceptable, explain why & where, but politely

Specific rather than vague criticism, e.g. • 'what' is wrong with the algorithm, rather than 'the

algorithm is wrong' • what related work is missing • key examples of numerous errors

Page 23: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Ethics: Speed

Speed • fast turnover • you are on a critical path! • affects timeliness & publication delays • turnover times:

o conferences: deadlines o journals: approx. 3-6 weeks

Page 24: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Ethics: Objectivity

Fairness • author may use point of view/methodology/

arguments different from your own • judge from their school of thought• remove personal prejudice

o e.g. field, institution, author, nationality, author, association (colleague, friend, rival)

Conflict of interest • discuss with editor • if you cannot be objective, return the paper

Page 25: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Ethics: Confidentiality

Do not circulate submitted papers • except for other reviews/comments (publication-

dependent)

Never use/discuss results • but can ask for permission from the authors

Protecting your identity• anonymous reviewing the norm• you may reveal your identity if you wish...

Page 26: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Ethics: Honesty & Courtesy

Honesty • judge your own expertise, • give your own confidence in your appraisal

Courtesy • constructive criticism • non-inflammatory language • no put-downs

I was rejected

Page 27: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Ethics: Dilemmas

How many papers to submit/referee? • 1 paper submitted -> 3 referees (minimum)

How much time should I spend reviewing • enough to give fair treatment • don't rush, the author deserves a fair hearing

Page 28: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Ethics: Dilemmas

What if a similar paper has been published? • journal papers can be reasonable expansions of

conference papers • can be republished if obscure (eg, workshop)

What if I am working on the same problem? • be honest & open -> consult with the editor • be aware of the race for independent co-discovery

Page 29: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

A Template for Reviewing

Paper TitleAuthor(s)Manuscript Number

Page 30: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

A Template for Reviewing

Briefly summarize the paper (2-3 lines)• can you extract a main message from your

paper?o lets author know if you understood the main message

• “If you can’t, there is probably something wrong with the paper”

– --- CHI FAQ

Page 31: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

A Template for Reviewing

What is NEW and SIGNIFICANT in the work reported?

• New:o has it been done before?o is it a rehash / republication of old stuff (yours or

others)?

• Significanceo in five years time, would the work have an identifiable

impact? (rare)

Page 32: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

A Template for Reviewing

...What is NEW and SIGNIFICANT • Survey/discussion piece

o does it add value?

• Would it stimulate further work in this area?o is it a reasonable increment that keeps the research

area going (frequent)?o does it have innovations?o is it interesting?o is it timely to the community?

Page 33: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

A Template for Reviewing

Describe the QUALITY of the RESEARCH

• How sound is the work?o quality of algorithms, analyses, evaluation methods,

etc.

• How appropriate/reliable are the methods used?o are they adequate to support the conclusionso is it correct?

Page 34: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

A Template for Reviewing

Describe the QUALITY of the RESEARCH

• How reasonable are the interpretations?o good argumentso alternative interpretations explored/left out

• How does it relate to existing work?o bibliographies, background, important omissions…

Page 35: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

A Template for Reviewing

Describe the quality of the research

• Can an experienced practitioner in the field duplicate the results from the paper and the references?

o are there details sufficient?

Page 36: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

A Template for Reviewing

Describe the quality of the writing• is the message clear?• is the paper easy to follow and understand?• is its style exciting or boring?• is it well organized?• is there a good flow of logic/argumentation?

• is it grammatically correct?• are figures and tables used well and integrated into the

text?• if it is a foreign writer, how can it be improved?

Page 37: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

A Template for Reviewing

How relevant is the work to the expected readers?• domain • depth of treatment• degree of specialization• accessible to expected range of expertise of readership

Page 38: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

A Template for Reviewing

Provide any other comments you believe would be useful to the author

• constructive suggestions on repairing problems• pointers to missing / relevant work• minor typos/flaws

• If revisions were possible, what should the author do to make this paper publishable?

o concrete, very specific suggestions on what – must be done– optional work

Page 39: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

A Template for Reviewing.

Rate the papers acceptibility andsummarize why you gave this rating

• Conference:o Definitely rejecto Probably rejecto Could go either way Note: equivalence class!o Probably accepto Definitely accept

Page 40: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

A Template for Reviewing

Rate the papers acceptibility• Journal:

o Definitely reject o Major revisions

– additional work, major reworking of arguments– subject to a careful check by editor/reviewers

o Minor revisions– typos, minor changes

o Accept as is

Page 41: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

A Template for Reviewing

Rate your expertise in the area addressed by the paper 1 = Know virtually nothing about this area 2 = Not too knowledgeable, but I know a bit 3 = Know a moderate amount, about average 4 = Not my main area of work, but I know a lot about it 5 = My area of work, I know it well

Page 42: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

A Template for Reviewing

Provide comments that you feel are relevant to the review process but that you do NOT want forwarded to the author(s)

• try to avoid using this• conflict of interests• pointers to things that would reveal identity• harsher things that would be mis-interpreted• suggestions on how to phrase acceptance/rejection letters...

Page 43: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

The Message

Refereeing is excellent practice for

• developing critical appraisal skills

• understanding what makes good & bad papers

• understanding how others evaluate your work

Page 44: How to Referee a Technical Paper Saul Greenberg University of Calgary is this paper any good?

Permissions

You are free:• to Share — to copy, distribute and transmit the work• to Remix — to adapt the work

Under the following conditions:Attribution — You must attribute the work in the manner specified by the author (but not in any way that suggests that they endorse you or your use of the work) by citing:

“Saul Greenberg , University of Calgary, AB, Canada: Grad Tips , http://saul.cpsc.ucalgary.ca/saul/Noncommercial — You may not use this work for commercial purposes, except to assist one’s own teaching and training within commercial organizations.Share Alike — If you alter, transform, or build upon this work, you may distribute the resulting work only under the same or similar license to this one.

With the understanding that:Not all material have transferable rights — materials from other sources which are included here are cited Waiver — Any of the above conditions can be waived if you get permission from the copyright holder.Public Domain — Where the work or any of its elements is in the public domain under applicable law, that status is in no way affected by the license.Other Rights — In no way are any of the following rights affected by the license:

• Your fair dealing or fair use rights, or other applicable copyright exceptions and limitations;• The author's moral rights;• Rights other persons may have either in the work itself or in how the work is used, such as publicity or privacy rights.

Notice — For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to this web page.