# Algorithms from the Book.

Paul Erdos talked about the "Book" where God keeps the most elegant proof of each mathematical theorem. This even inspired a book (which I believe is now in its 4th edition): Proofs from the Book.

If God had a similar book for algorithms, what algorithm(s) do you think would be a candidate(s)?

If possible, please also supply a clickable reference and the key insight(s) which make it work.

-
Great question! [Edit:} One question. Where do we draw the line between algorithms and datastructures? What if the key insight to an algorithm is intimately related to a datastructure (for example UNION FIND in the inverse Ackermann function)? – Ross Snider Aug 17 '10 at 18:25
a great source and maybe a candidate for such a book is "Encyclopedia of Algorithms" springer.com/computer/theoretical+computer+science/book/… – Marcos Villagra Aug 17 '10 at 23:15
I'm a little surprised that algorithms which I consider quite tricky (KMP, linear suffix arrays) are considered by others as being "from the Book." To me, "from the Book" means simple and obvious, but only with hindsight. I'm curious how others interpret "elegant". – Radu GRIGore Aug 19 '10 at 7:33
@supercooldave You don't have to believe in God, but you should believe in his book. ;-) – Ross Snider Aug 24 '10 at 22:27
During a lecture in 1985, Erdős said, "You don't have to believe in God, but you should believe in The Book." – Robert Massaioli Dec 15 '10 at 3:17

Union-find is a beautiful problem whose best algorithm/datastructure (Disjoint Set Forest) is based on a spaghetti stack. While very simple and intuitive enough to explain to an intelligent child, it took several years to get a tight bound on its runtime. Ultimately, its behavior was discovered to be related to the inverse Ackermann Function, a function whose discovery marked a shift in perspective about computation (and was in fact included in Hilbert's On the Infinite).

Wikipedia provides a good introduction to Disjoint Set Forests.

-

Knuth-Morris-Pratt string matching. The slickest eight lines of code you'll ever see.

-
It's kinda mind boggling to realize that that was something that wasn't obvious at some time and is only obvious now because they came up with it and we learnt it... I think we should apply Carr's theory of history to Mathematics and Computer Science. – mechko Sep 2 '10 at 12:50
By the description, I'd say this is related to the Boyer-Moore fast substring search. – bart Sep 22 '10 at 7:20

The algorithm of Blum, Floyd, Pratt, Rivest, and Tarjan to find the kth element of an unsorted list in linear time is a beautiful algorithm, and only works because the numbers are just right to fit in the Master Theorem. It goes as follows:

1. Sort each sequence of five elements.
2. Pick out the median in each.
3. Recur to find the median of this list.
4. Pivot on the median of medians (as in Quicksort)
5. Select the proper side of the list and position in that list, and recur.
-
This is one of my favorite algorithms. I like an intuition for it that I learnt from Chazelle's discrepancy book: the set of medians of groups of $1/\epsilon$ elements is like an $\epsilon$-net for intervals in the ordered list of the input numbers. So the algorithm follows a general paradigm: compute an $\epsilon$-net fast, solve the problem on the net, recurse on some part of the input to refine the solution, until you have the exact solution. it's very useful technique – Sasho Nikolov Mar 30 '12 at 22:34
BTW once you parametrize the size of the groups, the constants are not so magical. they are of course optimized to give the right thing in the Master theorem – Sasho Nikolov Mar 30 '12 at 22:36

Binary Search is the most simple, beautiful, and useful algorithm I have ever run into.

-
And insanely hard to write correctly– see "Are you one of the 10% of programmers who can write a binary search?" – jon Jun 16 '13 at 19:32

I'm surprised not to see the Floyd-Warshall algorithm for all-pairs shortest paths here:

d[]: 2D array. d[i,j] is the cost of edge ij, or inf if there is no such edge.

for k from 1 to n:
for i from 1 to n:
for j from 1 to n:
d[i,j] = min(d[i,j], d[i,k] + d[k,j])


One of the shortest, clearest non-trivial algorithms going, and $O(n^3)$ performance is very snappy when you consider that there could be $O(n^2)$ edges. That would be my poster child for dynamic programming!

-
This algorithm can be also generalised in a really neat way. See e.g. r6.ca/blog/20110808T035622Z.html and cl.cam.ac.uk/~sd601/papers/semirings.pdf – Mikhail Glushenkov Nov 25 '13 at 0:31
-

Might seem somewhat trivial (especially in comparison with the other answers), but I think that Quicksort is really elegant. I remember that when I first saw it I thought it was really complicated, but now it seems all too simple.

-
Quicksort also raises interesting questions about what exactly the essence of an algorithm is. E.g. the standard elegant Haskell implementation looks exactly like the standard pseudo-code definition, but it has different asymptotic complexity. So, is Quicksort just about divide-and-conquer or is the clever in-situ pointer-fiddling an essential part of Quicksort? Can Quicksort even be implemented in a purely functional setting or does it require mutability? – Jörg W Mittag Sep 3 '10 at 22:28
The idea of the "essence" or the "moral" of an algorithm comes of course from the beautiful paper The Genuine Sieve of Eratosthenes by Melissa E. O'Neill (cs.hmc.edu/~oneill/papers/Sieve-JFP.pdf) and the quicksort discussion comes from the LtU discussion of that paper (lambda-the-ultimate.org/node/3127), specifically starting at this comment: lambda-the-ultimate.org/node/3127/#comment-45549 – Jörg W Mittag Sep 5 '10 at 18:32
@Jörg: Implementing quicksort on linked lists is completely sensible, and has the same asymptotic running time as its in-place implementation on arrays (heck, even the naive out-of-place implementation on arrays has the same running time) – both on average and in the worst case. As for space usage, this is indeed different but it must be said that even the “in-place” version requires non-constant extra space (call stack!), a fact readily overlooked. – Konrad Rudolph Sep 8 '10 at 11:31

Huffman coding for data compression.

-

Polynomial identity testing with the Schwartz-Zippel lemma:

If someone woke you up in the middle of the night and asked you to test two univariate polynomial expressions for identity, you'd probably reduce them to sum-of-products normal form and compare for structural identity. Unfortunately, the reduction can take exponential time; it's analogous to reducing Boolean expressions to disjunctive normal form.

Assuming you are the sort who likes randomized algorithms, your next attempt would probably be to evaluate the polynomials at randomly chosen points in search of counterexamples, declaring the polynomials very likely to be identical if they pass enough tests. The Schwartz-Zippel lemma shows that as the number of points grows, the chance of a false positive diminishes very rapidly.

No deterministic algorithm for the problem is known that runs in polynomial time.

-

The Miller-Rabin primality test (and similar tests) should be in The Book. The idea is to take advantage of properties of primes (ie using Fermat's little theorem) to probabilistically look for a witness to the number not being prime. If no witness is found after enough random tests, the number is classified as prime.

On that note, the AKS primality test that showed PRIMES is in P should certainly be in The Book!

-

Depth First Search. It is the basis of many other algorithms. It is also deceivingly simple: For example, if you replace the queue in a BFS implementation by a stack, do you get DFS?

-
It is also the basis of Prolog execution! – muad Aug 19 '10 at 7:14
What's the point about BFS with a stack I'm missing? I would have thought that the answer is "yes, you get DFS". – Omar Antolín-Camarena Jun 16 '13 at 13:28

The Sieve of Eratosthenes, simple & intuitive.

I also like the beauty of Horner's Algorithm.

-
-
Yes, especially for its butterfly diagrams (en.wikipedia.org/wiki/Butterfly_diagram). – psihodelia Aug 3 '11 at 11:20

Dijkstra's algorithm: the single-source shortest path problem for a graph with nonnegative edge path costs. It's used everywhere, and is one of the most beautiful algorithms out there. The internet couldn't be routed without it - it is a core part of routing protocols IS-IS and OSPF (Open Shortest Path First).

1. Assign to every node a distance value. Set it to zero for our initial node and to infinity for all other nodes.
2. Mark all nodes as unvisited. Set initial node as current.
3. For current node, consider all its unvisited neighbors and calculate their tentative distance (from the initial node). For example, if current node (A) has distance of 6, and an edge connecting it with another node (B) is 2, the distance to B through A will be 6+2=8. If this distance is less than the previously recorded distance (infinity in the beginning, zero for the initial node), overwrite the distance.
4. When we are done considering all neighbors of the current node, mark it as visited. A visited node will not be checked ever again; its distance recorded now is final and minimal.
5. If all nodes have been visited, finish. Otherwise, set the unvisited node with the smallest distance (from the initial node) as the next "current node" and continue from step 3.
-

Gentry's Fully Homomorphic Encryption Scheme (either over ideal lattices or over the integers) is terribly beautiful. It allows a third party to perform arbitrary computations on encrypted data without access to a private key.

The encryption scheme is due to several keen observations.

• To get a fully homomorphic encryption scheme, one needs only to have a scheme that is homomorphic over addition and multiplication. This is because addition and multiplication (mod 2) are enough to get AND, OR and NOT gates (and therefore Turing Completeness).
• That if such a scheme were to be had, but due to some limitations could only be executed for circuits of some finite depth, then one can homomorphically evaluate the decryption and reencyption procedure to reset the circuit depth limitation without sacrificing key privacy.
• That by "squashing" the depth of the circuit version of the decryption function for the scheme, one might enable a scheme originally limited to finite, shallow circuits an arbitrary number of computations.

In his thesis, Craig Gentry solved a long standing (and gorgeous) open problem in cryptography. The fact that a fully homomorphic scheme does exist demands that we recognize that there is some inherent structure to computability that we may have otherwise ignored.

http://crypto.stanford.edu/craig/craig-thesis.pdf

http://eprint.iacr.org/2009/616.pdf

http://portal.acm.org/citation.cfm?id=1666420.1666445

-

Strassen's algorithm for matrix multiplication.

-

The linear time algorithm for constructing suffix arrays is truly beautiful, although it didn't really receive the recognition it deserved http://www.cs.helsinki.fi/u/tpkarkka/publications/icalp03.pdf

-

An example as fundamental and "trivial" as Euclid's proof of infinitely many primes:

2-approximation for MAX-CUT -- Independently for each vertex, assign it to one of the two partitions with equal probability.

-
Yes, a very nice algorithm. Less trivially, at the cost of another factor of 2, this algorithm also works for maximizing any submodular function, not just the graph cut function. This is a result of Feige, Mirrokni, and Vondrak from FOCS 07 – Aaron Roth Aug 17 '10 at 20:23

I was impressed when I first saw the algorithm for reservoir sampling and its proof. It is the typical "brain teaser" type puzzle with an extremely simple solution. I think it definitely belongs to the book, both the one for algorithms as well as for mathematical theorems.

As for the book, the story goes that when Erdös died and went to heaven, he requested to meet with God. The request was granted and for the meeting Erdös had only one question. "May I look in the book?" God said yes and led Erdös to it. Naturally very excited, Erdös opens the book only to see the following.

Theorem 1: ...
Proof: Obvious.

Theorem 2: ...
Proof: Obvious.

Theorem 3: ...
Proof: Obvious.

-
Theorem 4: … Proof: exercise to the reader. – jon Jun 16 '13 at 19:35

The Gale-Shapley stable marriage algorithm. This algorithm is greedy and very simple, it isn't obvious at first why it would work, but then the proof of correctness is again easy to understand.

-

Gaussian elimination. It completes the generalization sequence from the Euclidean GCD algorithm to Knuth-Bendix.

-
the sequence is: Euclidean GCD -> Gaussian Elimination -> Buchberger -> Knuth-Bendix. One can also put in (instead of Gaussian Elimination) univariate polynomial division and modulo (in the generalization order it is 'apart' from Gaussian Elimination, GE is multivariate degree 1, polynomial ring is univariate unlimited degree, Buchberger's is multivariate unlimited degree. The generalization jump is largest from EGCD to GE or polynomial ring because of the addition of variables, and then also large from Buchberger to KB because of the unlimited signature. – Mitch Aug 30 '10 at 15:59

I've always been partial to Christofides' Algorithm that gives a (3/2)-approximation for metric TSP. In fact, call me easy to please, but I even liked the 2-approximation algorithm that came before it. Christofides' trick of making a minimum weight spanning tree Eulerian by adding a matching of its odd-degree vertices (instead of duplicating all edges) is simple and elegant, and it takes little to convince one that this matching has no more than half the weight of an optimum tour.

-

How about Grover's algorithm? It's one of the simplest quantum algorithms, and allows you to search an unsorted database in $O(\sqrt N)$. It is provably optimal, and also provably outperforms any classical algorithm. For a bonus it is very easy to understand and intuitive.

-

The Tortoise and hare Algorithm. I like it because I'm sure that even if I wasted my entire life trying to find it, there is no way I would come up with such an Idea.

-
Do you know the dumb algorithm that solves the problem with the same asymptotics and follows an algorithmic design pattern? I'm talking about iterative deepening. In the nth iteration you start at the 2^n-th successor of the root and look 2^n successors forward in search of a recurrence. Even though you're retracing some of your steps with every iteration, the geometric growth rate of the search radius means that it doesn't affect the asymptotics. – Per Vognsen Sep 23 '10 at 15:22

Algorithms for linear programming: Simplex, Ellipsoid, and interior point methods.

http://en.wikipedia.org/wiki/Linear_programming#Algorithms

-

Robin Moser algorithm for solving a certain class of SAT instances. Such instances are solvable by Lovasz Local Lemma. Moser algorithm is indeed a de-randomization of the statement of the lemma.

I think that is some years his algorithm (and the technique for its correctness proof) will be well digested and refined to the point of being a viable candidate for an Algorithm from the Book.

This version is an extension of his original paper, written with Gábor Tardos.

-

Knuth's Algorithm X finds all solutions to the exact cover problem. What is so magical about it is the technique he proposed to efficiently implement it: Dancing Links.

-

Merge Sort. Simple, elegant, efficient.

-

Marcus Hutter's The Fastest and Shortest Algorithm for All Well-Defined Problems.

This kind of goes against the spirit of the other offerings in this list, since it is only of theoretical and not practical interest, but then again the title kind of says it all. Perhaps it should be included as a cautionary tale for those who would look only at the asymptotic behavior of an algorithm.

-

I think we must include Schieber-Vishkin's, which answers lowest common ancestor queries in constant time, preprocessing the forest in linear time.

I like Knuth's exposition in Volume 4 Fascicle 1, and his musing. He said it took him two entire days to fully understand it, and I remember his words:

I think it's quite beautiful, but amazingly it's got a bad press in the literature (..) It's based on mathematics that excites me.

-
Wait, it may be beautiful, but if it took Knuth two entire days to fully understand it, is it really "from the book"? – ShreevatsaR Aug 31 '10 at 23:05

## protected by KavehMay 10 '13 at 6:52

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site.