How do I referee a paper?

Question

Updated below

We all know the critical importance of peer-review. It is the main form of quality control and feedback on research. However, to an early-stage researcher (like me), it can sometimes be a confusing system/process.

Accordingly, there are several treatises on the scientific refereeing process that give guidance. Two (very different) examples from computer science -- this 1994 article by Parberry and a more recent one by Cormode -- offer great advice (though the latter might be a shade mischievous).

Here, I'd like to solicit broader advice from the more experienced members of this community about the review process, with particular regard to the peculiarities of theoretical computer science.

1. What are the main criteria for determining the significance of a paper's results? How do I judge whether a paper should be accepted to the conference/journal? Is it important to verify correctness?
2. What are the main elements of a referee report, and which parts are most important? Is it always necessary to give a recommendation of (non)acceptance? What goes in the report and what goes solely to the editor?
3. How does assessment for conferences differ from that in journals? How do reports for conferences differ from those in journals? (How on earth do I rate my "confidence" in my recommendation?) Should the journal version be significantly different from the conference paper?
4. What if I don't understand the paper? ...the proof? (Is it my fault or theirs?)
5. What about typographical/grammatical mistakes? What if there are a lot of them?
6. How much time should I spend on a report?
7. How many reports a year am I expected to write? When is it acceptable to refuse a request to referee?

Of course, any other relevant questions and answers on this topic are encouraged, since this is CW.

This question is inspired by (stolen from) a similar post at MathOverflow.

Update 15/02/2011:

I am still very interested in getting more input on this question especially with regard to reviewing conference papers and program committee membership. (These two roles are themselves different beasts, and both very unlike being a referee for a journal article, IMO.) Granted, program committee membership is rarer than refereeing or reviewing (and it hasn't been my privilege yet), but is a responsibility that every researcher in theoretical computer science must take on eventually.

• Time. How much time am I expected to spend as a committee member or conference reviewer? Given the probability that I could get ten or perhaps many more to handle in the space of a few weeks, how do I avoid running out of time? What are the most important things to spend time on?

• Confidence. What if the paper is too far from my area of expertise? What factors should go into nominating/asking someone else to review a submission? If it is not too far from my area of expertise and I elect to review it, when is it permissible to give a confidence rating of 1?

• Criteria. There are critical differences between journals and conferences. Some very important papers are not published in journals. Some very important papers did not previously appear in conferences. What are the most significant distinctions in criteria on which to assess papers in these settings?

• Recommendations. Inherently, there are fewer recommendations that can be offered to the authors of a conference paper, primarily due to space and time constraints. Also, there is usually only one round of review. Another consideration is that my report becomes public to the entire strong committee. What is the scope of suggestions/directives that I can offer?

As before, if you think I've missed out on asking any particular questions, do let me know, or edit directly. This is CW, after all.

These new thoughts are partly motivated by reading a paper that Suresh mentioned on his blog.

Answer 1

1. To the best of your knowledge, does the paper make a significant, well-presented, and correct contribution to the state of the art? If the paper fails any of the three criteria, it's fair to reject it for that reason alone, regardless of the other two.

2. Here's what I think a report should contain. Everything should be visible to the author, except possibly for serious accusations of misconduct.

   a. A quick summary of the paper, to help the editor judge the quality of the results, and to help convince both the author and the editor that you actually read and understood the paper. Place the result in its larger context. Include a history of prior versions, even if the authors include it in the submission. Be respectful, but brutally honest.

   b. A discussion of the strengths and weaknesses of the paper, in terms of correctness, novelty, clarity, importance, generality, potential impact, elegance, technical depth, robustness, etc. If you suspect unethical behavior (plagiarism, parallel submission, cooked data), describe your suspicions. Be respectful, but brutally honest.

   c. A recommendation to the editor for further action — accept, accept with minor revision, ask for a second round of reviewing, or reject outright. Keep in mind that you are making a recommendation, not a decision; if you can't make up your mind, just say so. Be respectful, but brutally honest.

   d. More detailed feedback to the author — more detailed justification for your recommendation, requests for clarification in the final version, missing references, bugs in the proofs, simplifications, generalizations, typos, etc. Be respectful, but brutally honest.

3. Conference reports should be shorter; program committees have hundreds of papers to consider at once. Whether there should be a difference between conference and journal papers is up to the journal (and indirectly, up to the community). Most theoretical computer science journals do not insist on a significant difference; it is quite common for the conference and journal versions of a theory paper to be essentially identical. When in doubt, ask the editor!

4. If you still don't understand the paper after making a good-faith effort, it's the author's fault, or possibly the editor's, but certainly not yours. The author's primary responsibility is to effectively communicate their result to their audience, and a good editor will send you a paper to referee only if they think you're a good representative of the paper's intended audience. But you do have to make a good-faith effort; do not expect to immediately understand everything (anything?) immediately on your first reading.

5. If there are a lot of errors, don't even read the paper; just recommend rejection on the grounds that the paper is not professionally written. Otherwise, if you really want to be thorough, include a representative list of grammar, spelling, and punctuation mistakes, but don't knock yourself out finding every last bug. Be respectful, but brutally honest.

6. Expect to spend about an hour per page, mostly on internalizing the paper's results and techniques. Be pleasantly surprised when it doesn't actually take that long. (If it takes significantly less time than that, either the paper is either exceedingly elegant and well-written, you know the area extremely well, or the paper is technically shallow. Don't confuse these three possibilities.)

7. You should write at least as many referee reports as other people write for you. If this takes more time than writing your own papers, you're not spending enough time on your own papers.

Answer 2

A lot depends upon the conference/journal, as each community has developed its own style, so knowing what's expected of the conference/journal certainly will help a little.

1. What are the main criteria for determining the significance of a paper's results? How do I judge whether a paper should be accepted to the conference/journal? Is it important to verify correctness?

Criteria: Novelty/originality, expected impact, correctness/validity, extensiveness (how much of the problem is studied? + theorems? + implementation? +experimental results?), quality of presentation. The criteria for journal acceptance are much higher than for conferences. For a conference you give a score based on the previous criteria. Verifying correctness as much as is possible is very important, especially for journal articles.

2. What are the main elements of a referee report, and which parts are most important? Is it always necessary to give a recommendation of (non)acceptance? What goes in the report and what goes solely to the editor?

The referees report contains at least 4 main elements: A summary of the paper and its contributions; points in favour of accepting the paper; points against the paper; major comments, including points to be addressed; minor comments (typos etc). You should always give an indication of acceptance or rejection or the degree of revision required. Comments to the editor could include: a short, perhaps blunt assessment of the paper; any statement of doubt; details of possible plagiarism or parallel submission of the paper; ...

3. How does assessment for conferences differ from those in journals? How do reports for conferences differ from those in journals? (How on earth do I rate my "confidence" in my recommendation?) Should the journal version be significantly different from the conference paper?

Journal reviews tend to be more extensive and can include a lot of things for the authors to do to bring their paper into an acceptable state, such as "implement and evaluate these ideas". Your confidence should be based on how confident you feel. Generally this will come with experience, so start out being conservative. It it's out of your field, even though you may understand the paper, being conservative is also a good idea –– it's difficult to assess originality of a paper outside of your field. Some journals say that journal submissions must contain a substantial amount of new material compared to the conference version. 30% is a figure I've heard.

4. What if I don't understand the paper? ...the proof? (Is it my fault or theirs?)

This can vary. Sometimes it will be your fault, sometimes it will be theirs. Use your judgement. Maybe ask a colleague to take a look at the paper. If you totally cannot understand it and it's not due to poor formatting, then perhaps contact the PC chair and explain that this is the case. In any case, this should be reflected in your confidence score. If the paper cannot be understood due to poor writing or language or because it has been prematurely submitted, then this should be written in your report.

5. What about typographical/grammatical mistakes? What if there are a lot of them?

When I was younger I'd report every single typo and grammatical error. Now I just don't have the time. Reporting some is always helpful: pick the most serious ones. Also recommend that the paper be proofread (by a native speaker). For journal papers be more thorough. If it has way too many errors, then is shouldn't have been submitted to a journal, which alone is reason for rejection (IMHO).

6. How much time should I spend on a report?

One day maximum for a conference paper, including reading time. For journal papers, especially long ones, reading the paper carefully might take a whole week.

7. How many reports a year am I expected to write? When is it acceptable to refuse a request to referee?

The expected amount could vary between 5 and 30. As a junior researcher, you'll be given a few for practice. As you advance through the grades, you'll take on more reviewing as you become a member of PCs. Then as you advance even further, you'll have an army of PhD students to do your reviewing for you. More than 30 in a year is quite onerous.

As I said above, I'd say it is appropriate to refuse a refereeing request when either (1) you do not feel that you have sufficient expertise to do a good review, or (2) you have too many reviewing (or other) commitments at the moment. It is also considered good form to suggest a number of possible alternative reviewers, in the case that you refuse.

Answer 3

I give here my 2 cents, and I add a seventh point:

1. How many reports do I have to do each year?

My answers

1. Of course the criteria are not the same for conferences and journals. For both correctness is probably the most important criterion. Then, my second criterion is novelty of the results for conferences, while it is impact of the results for journals. For a conference, it is also important to judge whether the paper is of interest for the audience, but this is a judgement call that can be done by PC chairs rather than reviewers.

2. First of all, except if you suspect a fraud or bad practice and are not sure (then you should talk solely to the editor), my opinion is that you should tell everything to the authors. I think it is important to be honest with the authors and thus to always give a recommendation. In a report, you should describe the topic, the results (so that authors can see that you understand the paper), then you should assess the merits of the results (novelty w.r.t. the state of the art, technical deepness, correctness, clarity) and point out every problem. Finally, you should give your recommendation, possibly with advice to the authors (such as "I think that this result is nice, but is more fitted for conference A rather than this one"). In a very few cases, you will be given papers that does not deserve to be even submitted, it is not admissible so it must be stated, politely but clearly.

3. I think that, except in the case of a journal invitation for a special issue related to a conference paper, there should be significant difference.

4. two cases: it is not understandable (their fault) so you can reject the paper for that reason or you are not the good reviewer for that paper. When it happens you know what case it is, if not you are in the second case!

5. I am not a native english speaker, so there is a bias in my answer. But I think it is not a reason for rejecting a paper (except in some extreme cases).

6. The time needed. It can range from 1 hour (a bad paper in a workshop) to several days (a very involved paper in a top journal).

7. Personally, I expect to do 3 reports per paper I submitted (and yes, it's a lot, in fact reviewing probably one of my most time consuming task - before teaching maybe).