Hidden Constants in Complexity of Algorithms

Question

For many problems, the algorithm with the best asymptotic complexity has a very large constant factor that is hidden by big O notation. This occurs in matrix multiplication, integer multiplication (specifically, the recent O(n log n) integer multiplication algorithm of Harvey and van der Hoeven), low-depth sorting networks and finding graph minors, to make a few. Such algorithms are sometimes called Galactic algorithms.

Note that for other algorithms, such as general sorting and integer addition, algorithms are known with optimal asymptotic complexity and small constant factors.

What research has been done in separating the former algorithms from the latter algorithms, from a theoretical perspective?

I am aware that hidden constants are often omitted to hide the distinction between different models of computation. However, I am confident that under a wide variety of different models, these Galactic algorithms will be slower than asymptotically worse algorithms for inputs of size one billion, for instance. The distinction is not subtle, in some cases. Has it been made rigorous?

For instance, one could invent a very simple model of computation, such as a von Neumann machine with a very simple ISA, and then implement the algorithms and bound their running times with explicit constants. Has this been done for a variety of algorithms?

Answer 1

One place where this is approached in an interesting way for a certain class of algorithms and combinatorial problems is in analytic combinatorics. The main approach described is similar to what you suggest: you start with some concrete implementation of an algorithm and identify some repeated operation (typically the heaviest one) that you'll use to associate an explicitly countable complexity for the execution of input of a given size $N$ in the form of the number $C_N$ that the chosen operation is performed.

The methodology does not require to fix any specific model of computation, although that could be useful of course. Also note that you could either try to compute the worst case behavior or the expected behavior, or still something else.

The most important ingredient in this methodology is the analysis of the generating functions of these values. You can sometimes obtain very precise asymptotic approximations using methods from complex analysis.

A simple example that is treated in the book is quicksort. This has a quadratic worst case running time, but in practice outperforms most $\mathcal O(n\log n)$ algorithms. When making a precise analysis of the expected cost of quicksort and you compare it to mergesort, you see that it is expected to outperform the latter from a size of around 10, if I remember correctly. To be able to compute this kind of things you cannot disregard the hidden constants of course.

In fact, in quicksort you sort a list by recursively sorting sublists, so that you would get an improvement for all sizes if you use mergesort on lists smaller than size 10. An interesting note in the book mentions that in some open sourced Microsoft library, the generic sort algorithm is implemented as quicksort until you get down to a size of 10, after which mergesort is used. In the code comments it is mentioned that performance tests showed this value to be optimal.