Code in Academic Papers

Question

In my academic career, I've read quite a few academic papers on various computer science topics. Many of which involve an implementation and some assessment of that implementation, yet I have found that very few of them actually publish the code they used.

To me, the benefits of including the actual implementation would be significant, namely:

• Extension of trust or reproducability (just test it yourself!)
• Clarification of ambiguities (particularly for papers written by non-native speakers)
• Reuse of code for applications

So why is it that so few papers actually include any code?

I suppose that it might be the intention of the organization behind the paper to utilize the implementation in their own applications, and thus would not want to release it, but if that's the case, why even write the paper?

Answer 1

Here is a well argued article by David Donoho and Jonathan Buckheit that I read in grad school which touches on exactly this topic from the point of view of wavelet researchers:

"WaveLab and Reproducible Research"

Their idea was even more ambitious, to provide code for reproducing all the figures in their papers in a convenient Matlab package.

I really like their idea but I think the issues are obvious.

(1) It is extra work (cleaning up the code, making at least a rudimentary user interface, writing some documentation, providing some support when people inevitably run into problems)

(2) It is not really required/expected by most conferences/reviewers

But I can't help but feel that the CS research community would benefit if there was an expectation of making the code and data used in any publication publicly available in a usable format. I admit that I haven't done so myself even when the amount of work involved would have been manageable. I think it is just hard to make yourself put in the extra effort when there is no external push.

Answer 2

If you work for an industrial lab, it can be much, much easier to get a paper approved for release than to get code approved for release (even if the paper contains all the information needed to rewrite the code). Blame bureaucracy.

Answer 3

Migrated and expanded from a comment:

I think this must vary by subfield. Nearly all the Theory B stuff I'm familiar with (and especially Haskell, Agda, and sometimes Coq -related) includes published code, sometimes even as an appendix or better yet inlined within the paper. A fair number of papers from, e.g., ICFP are written as literate programs to begin with, and their source in its entirety is published by the authors. A fair amount of those in turn have resulted in extracted libraries for distribution.

Of the remaining papers, a fair amount never had code to begin with. Of those, there's probably two main reasons. First are the papers whose main content is proof trees, typing rules with associated soundness proofs and the like. Of those, advances in mechanized metatheory have encouraged at least some authors to provide code in their theorem prover of choice (see Weirich's slides on POPLmark: http://www.seas.upenn.edu/~sweirich/talks/cambridge-09.pdf). Second are those which are descended from the Bird-Merteens stuff (banannas & co.). These are generally translatable into a functional language without too much work. However, I suspect that there's both typically a loss of generality, and that dealng with concrete issues of syntax and typing needlessly complicates things and makes it harder to follow the equational reasoning.

I wanted to substantiate my observations a bit, so did a rough count of the first two days of ICFP 2010. Of standard papers (i.e. not experience reports or invited talks), 12 out of 21 provided code of some sort. Three provided Coq (a fourth claimed a partial proof but did not publish it). Three proided Haskell. Three provided Agda. One provided Scheme, one provided Caml, and one provided Twelf. (Note that some provided code for more than one proof assistant, or for both a formalization and an implementation). Of the remaining papers, a few did work at a high enough level of abstraction that implementing it in a proof assistant would be a new paper in itself, and a fair number more did work that I suspect could have been implemented in a proof assistant using standard techniques, but which certainly would have taken a fair amount of work to do so. A few further papers claimed implementations/releases as future work.

Answer 4

You believe that code should be published, but you ask why papers do not include code. These are two different things.

Most of the time, there is simply not enough room to publish a significant amount of code. In my research field (image processing) pseudocode or architecture information is often far more valuable and I have never found myself stuck due to the lack of code in a paper. It's often left as an exercise to the reader who grasped the article.

Yet there is a lot of code available to illustrate papers. Authors usually have a webpage and even if the reviewer doesn't get a chance to try and check the code itself, natural selection appears to work pretty well and authors who do not publish code are a lot less cited.

Answer 5

This might have been asked sometime ago already, however I have always felt strongly about this so I will give my two cents. I have worked for years (not anymore) within the SAT community. Most researchers rarely publish their code. The paper is published along with the algorithm but it is very rare to see the actual code of the SAT solver (MAXSAT solver) etc, published along with the paper.

And the reality is that with the just the code published in the paper you will never have the chance to reproduce the author's experiments. Not only because the published code is not complete (of course) but also because even the published pseudo-code rarely translates semi-directly into what is actually implemented.

The reason behind this is hard to know and it might depend from researcher to researcher but mostly it is two fold.

• First, the researcher tends to work continually in a single solver publishing papers after papers on the same solver and incrementally adding new features which translate to new versions of the solver. There is an unhealthy obsession that competition will use your solver to further their careers by extending it and publishing papers without giving you due credit (meaning, co-authorship).

• Secondly, some code really is (as with all software) written in a rush. Half-baked scripts. Untested features, etc. By publishing this code the researcher would feel he would be embarrassing themselves and damaging their reputation.

I leave you with a recent reference on this from ACM: http://cacm.acm.org/magazines/2011/5/107698-the-importance-of-reviewing-the-code/fulltext

Answer 6

Historically, scientific articles had to be printed on paper, and journals were shipped internationally. Each extra page used to add a significant cost, thus articles were subject to length limitations, and even simple working code usually takes way much space than an informal description.

Today there is no good reason not to include code in any kind of article that references an algorithm.

It may also useful to abandon print-oriented formats like pdf and postscript in favor of more semantically-aware formats (HTML with MathML or perhaps an opensource variation of Mathematica).