Updated below
We all know the critical importance of peer-review. It is the main form of quality control and feedback on research. However, to an early-stage researcher (like me), it can sometimes be a confusing system/process.
Accordingly, there are several treatises on the scientific refereeing process that give guidance. Two (very different) examples from computer science -- this 1994 article by Parberry and a more recent one by Cormode -- offer great advice (though the latter might be a shade mischievous).
Here, I'd like to solicit broader advice from the more experienced members of this community about the review process, with particular regard to the peculiarities of theoretical computer science.
- What are the main criteria for determining the significance of a paper's results? How do I judge whether a paper should be accepted to the conference/journal? Is it important to verify correctness?
- What are the main elements of a referee report, and which parts are most important? Is it always necessary to give a recommendation of (non)acceptance? What goes in the report and what goes solely to the editor?
- How does assessment for conferences differ from that in journals? How do reports for conferences differ from those in journals? (How on earth do I rate my "confidence" in my recommendation?) Should the journal version be significantly different from the conference paper?
- What if I don't understand the paper? ...the proof? (Is it my fault or theirs?)
- What about typographical/grammatical mistakes? What if there are a lot of them?
- How much time should I spend on a report?
- How many reports a year am I expected to write? When is it acceptable to refuse a request to referee?
Of course, any other relevant questions and answers on this topic are encouraged, since this is CW.
This question is inspired by (stolen from) a similar post at MathOverflow.
Update 15/02/2011:
I am still very interested in getting more input on this question especially with regard to reviewing conference papers and program committee membership. (These two roles are themselves different beasts, and both very unlike being a referee for a journal article, IMO.) Granted, program committee membership is rarer than refereeing or reviewing (and it hasn't been my privilege yet), but is a responsibility that every researcher in theoretical computer science must take on eventually.
Time. How much time am I expected to spend as a committee member or conference reviewer? Given the probability that I could get ten or perhaps many more to handle in the space of a few weeks, how do I avoid running out of time? What are the most important things to spend time on?
Confidence. What if the paper is too far from my area of expertise? What factors should go into nominating/asking someone else to review a submission? If it is not too far from my area of expertise and I elect to review it, when is it permissible to give a confidence rating of 1?
Criteria. There are critical differences between journals and conferences. Some very important papers are not published in journals. Some very important papers did not previously appear in conferences. What are the most significant distinctions in criteria on which to assess papers in these settings?
Recommendations. Inherently, there are fewer recommendations that can be offered to the authors of a conference paper, primarily due to space and time constraints. Also, there is usually only one round of review. Another consideration is that my report becomes public to the entire strong committee. What is the scope of suggestions/directives that I can offer?
As before, if you think I've missed out on asking any particular questions, do let me know, or edit directly. This is CW, after all.
These new thoughts are partly motivated by reading a paper that Suresh mentioned on his blog.