4
$\begingroup$

This question was previously posted at https://math.stackexchange.com/questions/1220292/example-of-pairwise-independent-random-process-with-expected-max-load-sqrtn where it has no answers. I now realise this is relevant to hashing so I am reposting it here.

Throw $n$ balls into $n$ bins. Each bin is selected uniformly at random but the process is only pairwise independent. Call the maximum number of balls in any bin the max load.

Lemma 2 in these notes tells us that the max load is at most $\sqrt{2n}$ with probability at least $1/2$.

Can we give an explicit uniform but only pairwise independent random process for selecting the next bin to put a ball into which gives the expected max load to be $\Theta(\sqrt{n})$ asymptotically?

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ mathoverflow.net/a/202583 $\endgroup$
    – Emil Jeřábek
    Apr 10, 2015 at 19:07
  • 1
    $\begingroup$ As @EmilJeřábek implies, it seems that you can do it more generally for $k$-independence too, using theorem 1 of fr.arxiv.org/abs/1502.05729. The paper points out that for pairwise independence you can use theorem 8 of brics.dk/RS/97/16/BRICS-RS-97-16.pdf, but it seems false (i think that the universal hash family used there is not 2-independent). $\endgroup$
    – didest
    Apr 11, 2015 at 17:27
  • $\begingroup$ @DiegodeEstrada Thank you. It is also certainly very interesting if there indeed such an error in that paper. $\endgroup$
    – Simd
    Apr 11, 2015 at 20:03

1 Answer 1

Reset to default
8
$\begingroup$

Here's how to do it.

First, choose a random $k$ between 1 and $n$ to be the "crowded bin". Next, choose a random permutation $\pi$ of $1,2,\ldots, n-1$.

Now, for $1 \leq i \leq n-1$,
$$ \mbox{put ball } i \mbox{ into bin } \begin{cases} k \ \ \ \ \ \ \ \ \ \ \ \ \mbox{with probability }\ \frac{1}{\sqrt{n}}, \\ k + \pi(i) \mbox{ with probability }1 - \frac{1}{\sqrt{n}}, \end{cases}$$ where the sum is taken mod $n$.

Put ball $n$ into a random bin.

Consider balls $i,j$ where $1 \leq i < j \leq n-1$. The bins these two balls fall into are independent. They both go into the crowded bin with probability $\frac{1}{n}$, and the crowded bin is equally likely to be any of the $n$ bins. And if they don't both go into the crowded bin, by symmetry they are equally likely to be in any two different bins. Thus, the bins these two balls are placed into are independent.

Since ball $n$ is placed into a random bin, it is independent of the other balls.

The expected max load here is essentially the expected number of balls in the crowded bin, which is $$\frac{n-1}{\sqrt{n}}+\frac{1}{n} = \sqrt{n} + o(1).$$

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ Thank you very much. This is a really nice and clear answer. $\endgroup$
    – Simd
    Apr 11, 2015 at 8:51
  • $\begingroup$ I suppose you don't literally mean "the sum is taken mod $n$" as that would mean you sometimes place in bin $0$ which doesn't exist. $\endgroup$
    – Simd
    Apr 11, 2015 at 15:55
  • 1
    $\begingroup$ @felix: not quite literally: $0 \equiv n$ (mod $n$). $\endgroup$
    – Peter Shor
    Apr 11, 2015 at 16:50
  • 2
    $\begingroup$ In my case I would like to throw $2n$ balls into $n$ bins. I could just run your process twice with the second run completely independent of the first but that doesn't seem right. $\endgroup$
    – Simd
    Apr 11, 2015 at 19:03
  • 3
    $\begingroup$ Instead of a random permutation $\pi$, use a random reordering of the sequence $1,1,2,2, 3,3,\ldots,n-1,n-1$. And instead of probability $1/\sqrt{n}$, figure out which probability makes the first $2n-2$ balls pairwise independent. And just toss the last two balls into random bins. (There's undoubtedly a better way of dealing with the last two balls, but it won't make a substantial difference in the expected max load and it will make everything more complicated.) $\endgroup$
    – Peter Shor
    Apr 11, 2015 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.