6
$\begingroup$

This is what I'm looking for. Take a recursive algorithm:

def fib(n):
  if n == 0 or n == 1:
    return n
  else:
    return fib(n-1) + fib(n-2)

and turn it into this:

def fib(n):
  a, b = 0, 1
  for _ in range(n):
    a, b = b, a + b
  return a

...in a way that's fully automated.


My motivation is a hypothesis about automatic differentiation. The independent researcher Sandro Magi has had the following idea:

  • The dual numbers can be used to express forward mode automatic differentiation in very little code. This fact is well known.

  • Magi has shown that the dual numbers have a mathematically similar counterpart which he calls the codual numbers. On some level, there is no difference between the dual numbers and codual numbers; they are simply two different ways of representing the same mathematical object on a computer.

  • The codual numbers naturally express something similar to reverse-mode autodiff, which is of major interest to applications.

Unfortunately, Magi's simple algorithm exhibits exponential time behaviour on some instances. A good implementation of autodiff shouldn't behave that way. Simple experiments show that all known examples of exponential time behaviour can be solved using memoisation (as I've done here). The introduction of memoisation makes the time complexity of the resulting algorithm difficult to gauge.

Whenever exponential time behaviour gets fixed using memoisation, it indicates a connection to dynamic programming. In fact, this can be considered an instance of top-down dynamic programming, instead of the usually preferable bottom-up dynamic programming. I conjecture therefore that tape-based autodiff is the result of applying a transformation on Magi's codual numbers, turning it into an instance of bottom-up dynamic programming. This makes the time complexity legible.

$\endgroup$
4
  • $\begingroup$ Such an automatic system seems to be unlikely to exist... Specific toy problems (like the above) can of course be handled. Automatic memoization (which is already enough to reduce the running time to polynomial) are supported by some languages like lisp and scheme. A nice description of this is in en.wikipedia.org/wiki/… . A more "standard" description how to do this transformation is in Jeff Erickson algorithms book. Maybe ask in compilation/programming-languages forum about this... $\endgroup$ Jan 13 at 22:29
  • $\begingroup$ There are many posts on eliminating recursion, such as: softwareengineering.stackexchange.com/questions/347582/… $\endgroup$
    – naasking
    Jan 14 at 2:20
  • $\begingroup$ Related are compiler optimizations like "fusion" (stackoverflow.com/questions/38905369/…), and supercompilation (stackoverflow.com/questions/9067545/what-is-supercompilation), although those are pretty advanced compared to basic recursion elimination. I mention them because it's not always possible to eliminate recursion without adding allocation, but this can sometimes be done via fusion/supercompilation passes. $\endgroup$
    – naasking
    Jan 14 at 2:25
  • $\begingroup$ @SarielHar-Peled I've already mentioned memoisation in my question. In Python, it's even easier than you said: docs.python.org/3/library/functools.html - This results in a "top-down" style of dynamic programming instead of a (loosely defined) "bottom-up" style. I think the tape-based approach to autodiff is an instance of bottom-up style, and so memoisation alone is not enough. $\endgroup$
    – wlad
    Jan 14 at 8:33
2
$\begingroup$

There's actually two questions here!

  1. The transformation you ask about is called the tupling transformation. Basically, if your recursive calls follow a fixed pattern of overlap, a memo-table can be turned into some extra arguments (as in the fibonacci function)

    For references, see

    This is one of my favourite program transformations, and I wish more people knew about it, and more compilers did it!

  2. Your intuition that memoisation is enough to make reverse-mode AD work is a very good one. For a nice (IMO!) example of this, see my recent (POPL 2022) paper:

    What we do is take a naive computation of reverse derivatives via repeated evaluation of forward-mode AD with dual numbers, and incrementally optimise it until we get an asymptotically efficient algorithm for reverse derivatives. One of the key steps is to maintain sharing, precisely to avoid exponential slowdowns from loss of sharing.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.