2 of 10
deleted 1 character in body
wlad
  • 303
  • 1
  • 6

Literature on automatic derivation of dynamic programming algorithms from recursive algorithms?

This is what I'm looking for. Take a recursive algorithm:

def fib(n):
  if n < 1:
    return n
  else:
    return fib(n-1) + fib(n-2)

and turn it into this:

def fib(n):
  a, b = 0, 1
  for _ in range(n):
    a, b = b, a + b
  return a

...in a way that's fully automated.


My motivation is a hypothesis about automatic differentiation. The independent researcher Sandro Magi had the following idea:

  • The dual numbers (a hypercomplex number system very similar to the complex numbers) can be used to express forward mode automatic differentiation in very little code. This fact is well known.

  • The dual numbers have a mathematically similar counterpart which Magi calls the codual numbers.

  • The codual numbers naturally express something similar to reverse-mode autodiff, which is of critical interest to applications.

Unfortunately, Magi's simple algorithm exhibits exponential time behaviour on some instances. A good implementation of autodiff shouldn't behave that way. Simple experiments show that all known examples of exponential time behaviour can be solved using memoisation. The introduction of memoisation makes the time complexity of the resulting algorithm difficult to gauge.

Whenever exponential time behaviour gets fixed using memoisation, it indicates a connection to dynamic programming. In fact, this can be considered an instance of top-down dynamic programming, instead of the usually preferable bottom-up dynamic programming. I conjecture therefore that tape-based autodiff is the result of applying a transformation on Magi's codual numbers, turning it into an instance of bottom-up dynamic programming. This makes the time complexity legible.

wlad
  • 303
  • 1
  • 6