# Maximize the mutual information between 2 discrete random variables

I have two random variables $$X$$ and $$Y$$. $$X$$ follows Poisson-Binomial distribution with parameters $$\{q_1, \ldots, q_k\}$$. Thus, $$X$$ can take values in the set $$\{0,1,\ldots,k\}$$.

$$Y$$ is a binary random variable. $$Y=0$$ with probability $$\frac{0.1}{1+i}$$ when $$X=i$$.

I am interested in the mutual information between $$X$$ and $$Y$$ denoted by $$I(X;Y)$$.

Specifically, I have a conjecture that the mutual information $$I(X;Y)$$ is maximum when all $$q_i$$'s are equal which leads to a Binomial distribution for $$X$$.

Can someone help me prove this? Or provide a counter example?

• Do you have an intuition or justification for this conjecture? Feb 9 at 5:48
• @ClementC. Yes, First of all, Binomial distribution has the maximum entropy among Poisson- binomial distributions. Also, for a given mean, the variance is maximum when all probability parameters are equal and we have a binomial distribution. So, I am hoping somehow the symmetry can be used to claim the conjecture. ie, From a Poisson-Binomial, you can always create a BInomial with a better mutual information by mixing the sampling probabilities. I don't know how to establish this though. Feb 9 at 16:50
• There is a significant probability I messed up the "coding" in Mathematica, but, if not, it seems that a version of the conjecture is false (for a given value of $q_1$, it's maximised when all equal) in the case $k=2$: wolframcloud.com/obj/ccanonne/Published/conjecture-mi-pbd.nb (see the two plots at the end) Feb 11 at 4:12
• yes, this is what I laboriously tried to say above. What is not true is that, fixing an arbitrary $q_1$, the maximum is for $q_2=q_1$. What might be true (supported by this) is that, optimizing over both $q_1,q_2$, the max is realized for some pair where they are equal. Feb 11 at 5:09
• @ClementC, Yes, I am interested in joint optimization over all $q_i$'s. This also is in accordance with the intuitions I stated in my first comment. They are all focused on for a given mean $(\sum q_i)$. So, if someone can prove that for a given mean, my conjecture holds. The extension is trivial. Feb 11 at 5:18