3
$\begingroup$

I get a bit confused about different definitions of entropy and/or self-information.

Entropy?

$$ H(X) = - \sum_{x \in X} P_X(x) \cdot \log{\left(P_X(x)\right)} $$

Self-information?

$$ I(x) = - \log{P_X(x)} $$ $$ I(X) = - \sum_{x \in X} \log{\left(P_X(x)\right)} $$

When do I use which formulate to calculate what? Generally, I am interested in the average information content of a text. Here I would calculate the frequency of each word, the probability, sum up the negative logarithm of this probability and average it by the number of words.

What am I missing here?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
4
$\begingroup$

The entropy refers to a set of symbols (a text in your case, or the set of words in a language). The self-information refers to a symbol in a set (a word in your case).

The information content of a text depends on how common the words in the text are wrt the global usage of those words.

E.g. "Next saturday morning" should have less self-information (as a named entity) than "Large Hadron Collider", each of the words should have less self information (except for large, maybe). However the entropy may be higher in the first case, there are not many Large Hadron "things".

Maybe Markov chains or some model that is better suited for probability in natural language would give you better (more significant and useful) results.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ One minor point: the information content of a text also depends on the order of the words. $\endgroup$
    – Peter Shor
    Oct 3, 2013 at 14:31
  • $\begingroup$ @PeterShor yes, definitively, I was going to point that out with Markov chains, due to the conditional probability of a word given the previous one(s), but then I forgot. $\endgroup$
    – Trylks
    Oct 3, 2013 at 14:35
4
$\begingroup$

Self-information applies to an individual outcome, $x$. It measures how surprising that specific outcome is.

The entropy of process $X$ is the average amount of Shannon self-information something coming out of $X$ will have, assuming that things come out according to the $P_X$ distribution. It's the average amount of surprise.

Let's say we have a random number generator that gives you an integer between 0 and 4. Half the time it gives you a 0, and the rest of the time it's evenly distributed among [1,2,3,4]. Thus, there is 1 bit of self-information when a 0 comes out, and 3 bits when a 1 comes out (or a 2, a 3, or a 4). We take the average, and $(1/2 \times 1 +1/8 \times 3 + 1/8 \times 3 + 1/8 \times 3 + 1/8 \times 3) = 2$ bits of entropy.

Improve this answer
$\endgroup$
-2
$\begingroup$

Two mistakes from Post169:

  1. Post169 wrote : "....Let's say we have a random number generator that gives you an integer between 0 and 4. Half the time it gives you a 0 ....."

Random means equal probabilities. There is no random number generator, that gives you half the times 0. This is a bias generator.

  1. The equation that Post169 wrote is wrong. The result with this wrong equation is -2 (and not 2). The write equation is −[1/2×(-1) + 1/8×(-3) + 1/8×(-3) + 1/8×(-3) + 1/8×(-3)] =2 bits.

log(base2)0.5= -1, and log(base2)0.125= -3

Improve this answer
$\endgroup$
4
  • $\begingroup$ "Random means equal probabilities. " $\leadsto$ no, it doesn't. That would be uniformly random; there exist probability distributions (such as the one in Post169's answer) which are not uniform over their domain. $\endgroup$
    – Clement C.
    Feb 5, 2021 at 4:01
  • $\begingroup$ Hi @kraka, thank you for pointing things out. I've corrected the equation. While it is certainly the most random when all possibilities have equal probability, it is standard to refer to various random probability distribution functions. BTW, next time you see an error in someone's answer and want to address it, it's best to add a comment below the answer itself, rather than creating a new answer to the question. $\endgroup$
    – Post169
    Feb 5, 2021 at 14:20
  • $\begingroup$ To Clement C. : Of course there exist probability distributions which are not uniform over their domain, but in these distributions, all outcomes are NOT equally likely. "Post169" clearly talk about "random number generator". There is not random number generator, that gives outcomes (numbers) with not equally probabilities. Do not confused by words. Here is Science, not Poetry. $\endgroup$
    – kraka
    Feb 25, 2021 at 1:02
  • $\begingroup$ To Post169 : Thanks for your note, about add a comment than create a new answer. Really i did not know how to do this. I apologize. $\endgroup$
    – kraka
    Feb 25, 2021 at 1:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.