I get a bit confused about different definitions of entropy and/or self-information.


$$ H(X) = - \sum_{x \in X} P_X(x) \cdot \log{\left(P_X(x)\right)} $$


$$ I(x) = - \log{P_X(x)} $$ $$ I(X) = - \sum_{x \in X} \log{\left(P_X(x)\right)} $$

When do I use which formulate to calculate what? Generally, I am interested in the average information content of a text. Here I would calculate the frequency of each word, the probability, sum up the negative logarithm of this probability and average it by the number of words.

What am I missing here?


The entropy refers to a set of symbols (a text in your case, or the set of words in a language). The self-information refers to a symbol in a set (a word in your case).

The information content of a text depends on how common the words in the text are wrt the global usage of those words.

E.g. "Next saturday morning" should have less self-information (as a named entity) than "Large Hadron Collider", each of the words should have less self information (except for large, maybe). However the entropy may be higher in the first case, there are not many Large Hadron "things".

Maybe Markov chains or some model that is better suited for probability in natural language would give you better (more significant and useful) results.

  • 2
    $\begingroup$ One minor point: the information content of a text also depends on the order of the words. $\endgroup$ – Peter Shor Oct 3 '13 at 14:31
  • $\begingroup$ @PeterShor yes, definitively, I was going to point that out with Markov chains, due to the conditional probability of a word given the previous one(s), but then I forgot. $\endgroup$ – Trylks Oct 3 '13 at 14:35

Self-information applies to an individual outcome, $x$. It measures how surprising that specific outcome is.

The entropy of process $X$ is the average amount of Shannon self-information something coming out of $X$ will have, assuming that things come out according to the $P_X$ distribution. It's the average amount of surprise.

Let's say we have a random number generator that gives you an integer between 0 and 4. Half the time it gives you a 0, and the rest of the time it's evenly distributed among [1,2,3,4]. Thus, there is 1 bit of self-information when a 0 comes out, and 3 bits when a 1 comes out (or a 2, a 3, or a 4). We take the average, and $-(1/2 \times 1 +1/8 \times 3 + 1/8 \times 3 + 1/8 \times 3 + 1/8 \times 3) = 2$ bits of entropy.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.