35
$\begingroup$

In my academic career, I've read quite a few academic papers on various computer science topics. Many of which involve an implementation and some assessment of that implementation, yet I have found that very few of them actually publish the code they used.

To me, the benefits of including the actual implementation would be significant, namely:

  • Extension of trust or reproducability (just test it yourself!)
  • Clarification of ambiguities (particularly for papers written by non-native speakers)
  • Reuse of code for applications

So why is it that so few papers actually include any code?

I suppose that it might be the intention of the organization behind the paper to utilize the implementation in their own applications, and thus would not want to release it, but if that's the case, why even write the paper?

$\endgroup$
9
  • 14
    $\begingroup$ So why is it that so few papers actually include any code? Because the reviewers tolerate it for some reason, even if there is no good reason to hide the implementation. We are the reviewers, we can change it. $\endgroup$ Mar 8, 2011 at 22:11
  • 14
    $\begingroup$ I think this must vary by subfield. Nearly all the Theory B stuff I'm familiar with (and especially Haskell, Agda, and sometimes Coq -related) includes published code, sometimes even as an appendix or better yet inlined within the paper. A fair number of papers from ICFP are written as literate programs to begin with, and their source in its entirety is published by the authors. A fair amount of those in turn have resulted in extracted libraries for distribution. Of the remaining papers, a fair amount never had code to begin with. $\endgroup$
    – sclv
    Mar 8, 2011 at 22:24
  • 8
    $\begingroup$ Research results should be open (and free), and so should code. By some definitions, science only happens when hypotheses are falsifyable and experiments reproducible, so you could argue that publications basing on code that is not published are not scientific work. $\endgroup$
    – Raphael
    Mar 9, 2011 at 12:06
  • 4
    $\begingroup$ @Raphael Releasing the same code would only help establish repeatability, not reproducibility. Running the exact same code in the exact same experiment is hardly independent. Science requires reproducibility, not repeatability. $\endgroup$ Mar 9, 2011 at 14:23
  • 2
    $\begingroup$ I was thinking of results that talk about properties of the code rather than its results (in which case you are right, of course). Say, the authors conduct experiments on some implementation of the presented algorithm and compare runtimes by examining graphs (they call it "Algorithm Engineering"). Here, code provides reproducibility. $\endgroup$
    – Raphael
    Mar 9, 2011 at 19:56

6 Answers 6

17
$\begingroup$

Here is a well argued article by David Donoho and Jonathan Buckheit that I read in grad school which touches on exactly this topic from the point of view of wavelet researchers:

"WaveLab and Reproducible Research"

Their idea was even more ambitious, to provide code for reproducing all the figures in their papers in a convenient Matlab package.

I really like their idea but I think the issues are obvious.

(1) It is extra work (cleaning up the code, making at least a rudimentary user interface, writing some documentation, providing some support when people inevitably run into problems)

(2) It is not really required/expected by most conferences/reviewers

But I can't help but feel that the CS research community would benefit if there was an expectation of making the code and data used in any publication publicly available in a usable format. I admit that I haven't done so myself even when the amount of work involved would have been manageable. I think it is just hard to make yourself put in the extra effort when there is no external push.

$\endgroup$
3
  • 7
    $\begingroup$ I think an expectation of either (a) pseudocode or (b) code available, at least on websites, would be very useful for the CS research community. Not all research code has high useability, and I'm not sure whether making buggy code available would be a good thing or not. $\endgroup$ Mar 9, 2011 at 13:51
  • 1
    $\begingroup$ I've talked to paper authors who haven't released their code for similar reasons -- in the state it's in, they'd feel guilty if anybody actually tried to do anything with it. $\endgroup$
    – sclv
    Mar 9, 2011 at 20:30
  • 6
    $\begingroup$ code does not need to be good, clean, or reusable. but it does need to be reproducible in order to be considered 'science'. There was a good article in nature by Nick Barnes (2010) Publish your computer code: it is good enough arguing this point. $\endgroup$ Apr 17, 2011 at 2:46
16
$\begingroup$

If you work for an industrial lab, it can be much, much easier to get a paper approved for release than to get code approved for release (even if the paper contains all the information needed to rewrite the code). Blame bureaucracy.

$\endgroup$
4
  • 1
    $\begingroup$ while this is of course true, I wonder if an appropriate change in culture driven by academic researchers would enable industrial researchers to pressure management to allow code release. After all, I suspect even getting a policy for paper release must have taken some doing $\endgroup$ Mar 9, 2011 at 6:22
  • 6
    $\begingroup$ At least at MSR, it's very easy to release research code. It's actually harder to use outside code, since MS is super-careful about respecting software licenses. Since many paper-related code drops come without an explicit license file, this can be a little annoying. Usually it just takes an email to the paper author to clear things up, but you can make it easier for industrial researchers by remembering to stick an OSI-approved license on your public code release. $\endgroup$ Mar 9, 2011 at 12:11
  • 2
    $\begingroup$ @Neel: Do you mean "OSI-approved, but not GPL"? $\endgroup$ Mar 9, 2011 at 13:07
  • 3
    $\begingroup$ The GPL is usually ok -- lots of us use Emacs! :) We're not allowed to use Affero GPL software, since its reciprocity conditions extend to anyone who interacts with the software (ie, it closes the web service loophole), and MS doesn't want to risk the possibility that some internal server running AGPL code might accidentally be made public-facing. But even the AGPL is better than no license at all, because picking a license actually makes the terms of sharing unambiguous. $\endgroup$ Mar 9, 2011 at 16:01
13
$\begingroup$

Migrated and expanded from a comment:

I think this must vary by subfield. Nearly all the Theory B stuff I'm familiar with (and especially Haskell, Agda, and sometimes Coq -related) includes published code, sometimes even as an appendix or better yet inlined within the paper. A fair number of papers from, e.g., ICFP are written as literate programs to begin with, and their source in its entirety is published by the authors. A fair amount of those in turn have resulted in extracted libraries for distribution.

Of the remaining papers, a fair amount never had code to begin with. Of those, there's probably two main reasons. First are the papers whose main content is proof trees, typing rules with associated soundness proofs and the like. Of those, advances in mechanized metatheory have encouraged at least some authors to provide code in their theorem prover of choice (see Weirich's slides on POPLmark: http://www.seas.upenn.edu/~sweirich/talks/cambridge-09.pdf). Second are those which are descended from the Bird-Merteens stuff (banannas & co.). These are generally translatable into a functional language without too much work. However, I suspect that there's both typically a loss of generality, and that dealng with concrete issues of syntax and typing needlessly complicates things and makes it harder to follow the equational reasoning.

I wanted to substantiate my observations a bit, so did a rough count of the first two days of ICFP 2010. Of standard papers (i.e. not experience reports or invited talks), 12 out of 21 provided code of some sort. Three provided Coq (a fourth claimed a partial proof but did not publish it). Three proided Haskell. Three provided Agda. One provided Scheme, one provided Caml, and one provided Twelf. (Note that some provided code for more than one proof assistant, or for both a formalization and an implementation). Of the remaining papers, a few did work at a high enough level of abstraction that implementing it in a proof assistant would be a new paper in itself, and a fair number more did work that I suspect could have been implemented in a proof assistant using standard techniques, but which certainly would have taken a fair amount of work to do so. A few further papers claimed implementations/releases as future work.

$\endgroup$
12
$\begingroup$

You believe that code should be published, but you ask why papers do not include code. These are two different things.

Most of the time, there is simply not enough room to publish a significant amount of code. In my research field (image processing) pseudocode or architecture information is often far more valuable and I have never found myself stuck due to the lack of code in a paper. It's often left as an exercise to the reader who grasped the article.

Yet there is a lot of code available to illustrate papers. Authors usually have a webpage and even if the reviewer doesn't get a chance to try and check the code itself, natural selection appears to work pretty well and authors who do not publish code are a lot less cited.

$\endgroup$
8
$\begingroup$

This might have been asked sometime ago already, however I have always felt strongly about this so I will give my two cents. I have worked for years (not anymore) within the SAT community. Most researchers rarely publish their code. The paper is published along with the algorithm but it is very rare to see the actual code of the SAT solver (MAXSAT solver) etc, published along with the paper.

And the reality is that with the just the code published in the paper you will never have the chance to reproduce the author's experiments. Not only because the published code is not complete (of course) but also because even the published pseudo-code rarely translates semi-directly into what is actually implemented.

The reason behind this is hard to know and it might depend from researcher to researcher but mostly it is two fold.

  • First, the researcher tends to work continually in a single solver publishing papers after papers on the same solver and incrementally adding new features which translate to new versions of the solver. There is an unhealthy obsession that competition will use your solver to further their careers by extending it and publishing papers without giving you due credit (meaning, co-authorship).

  • Secondly, some code really is (as with all software) written in a rush. Half-baked scripts. Untested features, etc. By publishing this code the researcher would feel he would be embarrassing themselves and damaging their reputation.

I leave you with a recent reference on this from ACM: http://cacm.acm.org/magazines/2011/5/107698-the-importance-of-reviewing-the-code/fulltext

$\endgroup$
7
$\begingroup$

Historically, scientific articles had to be printed on paper, and journals were shipped internationally. Each extra page used to add a significant cost, thus articles were subject to length limitations, and even simple working code usually takes way much space than an informal description.

Today there is no good reason not to include code in any kind of article that references an algorithm.

It may also useful to abandon print-oriented formats like pdf and postscript in favor of more semantically-aware formats (HTML with MathML or perhaps an opensource variation of Mathematica).

$\endgroup$
2
  • 8
    $\begingroup$ +1 for first two paragraphs, -1 for last paragraph. You can take away my LaTeX when you pry it from my cold dead hands. $\endgroup$
    – Jeffε
    Mar 9, 2011 at 16:56
  • 2
    $\begingroup$ There are now plenty of tools available to enable literate LaTeX programming... $\endgroup$
    – sclv
    Mar 9, 2011 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.