How to write a negative review for a conference paper?

Question

This is related to the general question of "How do I referee a paper?". I am reviewing a paper for a conference, and this paper should be rejected, because it is not significant enough for publication and has flaws in some of its technical details. The paper is not wrong, but the ways in which it is right are not very interesting. How do I write a negative review in such a settings? More importantly, how do I do this as a non-senior researcher?

As an example, they make part of their argument through numerics, but I can prove the same result analytically. The analytic treatment is longer than what I can include in a review. I know the author's emails (the review process is not blind both ways), should I email them? This is a conference, so there is no time for revisions, will they be mad at me for rejecting their paper? Should I email them before or after they receive the reviews?

The paper is also sparsely cited, and does not connect strongly to existing literature. I am familiar with many aspects of the relevant literature, how detailed should I make my recommendations on further reading/references?

To extract the general questions:

• How does a junior researcher write a negative review?
• What should you do if you have specific technical comments/improvements that are too long for a standard review?

Answer 1

I think this is too much of a duplicate of the other question, but my comment got too long.

• Firstly, the review process is blind, so you simply have to do the best that you can. Be objective, be polite, give good reasons – see other question.

• There is probably no such thing has having technical comments/improvements that are too long for the standard review, unless you will end up rewriting the paper. Authors tend to appreciate comments, even if they mean more work. If the paper is borderline and is worth improving, then give as many comments as you have time for.

• If you have additional comments that you want to send to the authors, consult with the chair/editor before contacting the authors. Perhaps your comments can be sent anonymously via the chair/editor.

• Papers get rejected, we learn to live with it. The best you can do is provide good comments to help the authors improve the paper for next time round. Authors may get mad, but they don't get even.

• If the paper is terrible, just save your time and don't bother with all the comments. Write a sufficiently convincing review giving reasons why the paper should be rejected, point out that there are many typos and other mistakes, but don't enumerate them all.

• On the other hand, if you are presenting and proving an alternative axiomatisation of their results to the degree that you are not only rewriting their paper, but lifting their results to a whole new level, then you should consult the chair/editor to see what to do. Maybe you should get credit for your contribution. You certainly should never write up your results independently, not until the paper has appeared in print.

Answer 2

How does a junior researcher write a negative review?

By writing a long, thorough, honest, but polite review.

It helps if you address the paper, not the authors. Make it as non-personal as possible. Say how the paper should be revised, not what the authors should do. Say that Theorem 3 is wrong, not that the authors are wrong.

What should you do if you have specific technical comments/improvements that are too long for a standard review?

There is no such thing as "too long for a review".

Sure, senior researchers write one-line reviews, but that does not mean that a four-page review is too long for a conference paper. If you write a long review, of course anyone will immediately guess that this is written by a junior researcher, but I do not see any reason to avoid it.

---

However, if you write a long review, it is extremely important that it is well-structured and that anyone can find your recommendation (i.e., to publish or not, and very briefly why) in two seconds. Make your recommendation stand out, and make sure this part (usually very near the beginning of your review) is very brief.

Answer 3

The job of a reviewer is to evaluate whether the paper is worthy of publication. A nice reviewer also helps the authors make their research better. Fixing the paper is not the job of the reviewer. There should be no difference between a review by a junior or senior researchers, though in practice, a senior researcher might spend much less time and effort on it, and will feel much more secure in judging a paper as unworthy.

I agree with @Jukka Suomela that the review needs to be well structured. You want it to be useful to the editor, as well as to the authors, and you want to make it such that the authors cannot weasel their way out of doing what is necessary by "forgetting" to address an issue that is somewhere buried in your long-winded explanation.

This is how I write reviews:

The manuscript [title here] presents [2-3 sentence summary of the paper to demonstrate that you get what the paper is about. This demonstrates that you actually took the time to read and understand.] This is [say something good about the paper, even if you reject it. The authors will feel better]. [Say whether there are major flaws, if necessary]. Thus, I recommend [state your recommendation - unless you're supposed to communicate it separately to the editor, as you may see on some electronic review submission sites (this gives the editor more freedom to override one of the reviewers).]

Major issues

1. [start with a one-sentence "summary" of the issue before you go into detail]

2. [By the way, remember that you don't want to rewrite their paper. That's their job. Also, you do not recommend any future work they should do on this project. The issues they need to address are those that allow them to actually make the points that they claim they do. Check your summary if you're not sure]

Minor Issues

[keep this short, unless you specifically want to send the message to the editor that the paper is full of little flaws and thus bad. Also, leave out the trivial stuff like typos - instead write e.g. "the paper would benefit from thorough proofreading" - it costs you a lot of time and won't really make a difference in getting the paper published. If you do have a bit of additional time, re-read the major issues again to make sure you're explaining clearly what is wrong and how you expect it to improve. This is a lot more important to both the editor and the authors]

Note that I would not write out an analytical proof for them. It is their job to write their paper. State "this result should be be proven analytically, for example by following the XX approach". Once you see the paper published somewhere else, and they still don't have an analytical solution, you can contact the authors, saying "Hey, I just saw your paper, and I had an idea for an analytical solution. Are you interested in a collaboration?"