2
$\begingroup$

Please forgive me if this is not the right Stack Exchange (I also posted it at Cross Validated). Please also forgive me for inventing terms.

For discrete random variables X and Y, the mutual information of X and Y can be defined as follows: $I(X;Y) = \sum_{y \in Y} \sum_{x \in X} p(x,y) \log{ \left( \frac{p(x,y)}{p_1(x)\,p_2(y)} \right) }, \,\!$

I will define the mutual information of a "cell" $x_0$ to be: $CI(x_0,Y) = \sum_{y \in Y} p(x_0,y) \log{ \left( \frac{p(x_0,y)}{p_1(x_0)\,p_2(y)} \right) }, \,\!$

I'm not sure if this quantity goes by another name. Essentially I'm restricting focus to a single state of variable X (and then the full MI can be calculated by summing all the cell MIs).

My question: is it guaranteed that $CI(x_0,Y) \ge 0$? We know $I(X;Y)\ge0$ and we know that the pointwise mutual information can be negative. I feel like CI should be nonnegative and that I might be missing some obvious proof.

$\endgroup$
15
$\begingroup$

By the simple transformation $p(x_0, y) = p(x_0) p(y | x_0)$ your expression merely becomes $p(x_0) KL(p (y|x), p(y))$, where $KL(p,q)$ is the Kullback-Leibler divergence, and is always positive. therefore, your expression is always positive.

$\endgroup$
  • $\begingroup$ Terrific, thanks! I was fooling around with similar transformations but somehow missed the obvious. $\endgroup$ – Michael McGowan Dec 22 '10 at 17:13
  • $\begingroup$ yes. this particular transformation turns out to be very handy. $\endgroup$ – Suresh Venkat Dec 22 '10 at 17:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.