I was wondering about the self-information, the information content . If I have data and I measure different words in it, their probability and take the average mean of that, what is the lowest and what is the biggest outcome?

$$I(data) = \frac{\sum_{i=1}^{n} - log(P(word_i))}{n}$$

The summed up probability is one, but how does that change with the logarithm? It seems, if one word is very unique and another word takes up the rest of the data in a big quantity, than the information content is higher:

$$\frac{(-\log 0.9999) + (-\log 0.0001)}{2} > \frac{(-\log 0.9) + (-\log 0.1)}{2}$$

Does that mean, if one of the words limits to zero, that the information content becomes infinite? Does that make sense, seems strange.

  • $\begingroup$ If the frequency is actually zero, then that word should simply not be counted. But as the frequency of a word becomes very small, its information content does become unboundedly large. This makes sense if you view the information from word_i as the amount of "surprise" you would have if you saw word_i after drawing randomly from your data. It also makes sense if you view the information from word_i as the number of bits needed to specify its frequency. If its frequency is very small, say $2^{-30}$, then you need very many (in this case, 30) bits to specify it. $\endgroup$ Oct 1 '13 at 20:11
  • 4
    $\begingroup$ I don't think this is the standard definition of the information content, which is the weighted average of the self-information of the events, that is, the entropy: $I(\mathrm{data}) = \frac{1}{n} \sum -P(\mathrm{word}_i) \log P(\mathrm{word}_i).$ The quantity you are asking about can certainly be infinite. $\endgroup$ Oct 2 '13 at 19:06
  • $\begingroup$ I didn't even notice - my previous comment applies to the standard definition of information content as in Peter Shor's comment. $\endgroup$ Oct 2 '13 at 19:10
  • $\begingroup$ @JoshuaGrochow Mh, so I only meant self-information here and for self-information I find the definition $- \log{P(word_i)}$. The definition you have seems to be the averaged entropy? $\endgroup$ Oct 3 '13 at 11:40
  • 1
    $\begingroup$ The self-information for a single word is indeed $-\log P(\mathrm{word}_i)$. The entropy is the weighted average of the self-information over all words. What you have defined is the average self-information over all words. I don't know what this quantity if used for, but it is certainly not the information content, as you seem to think it is. If you take some samples from a distribution, and take the average self-information of this, it does indeed correspond to an information content, but it also converges to the entropy as you take more and more samples. $\endgroup$ Oct 3 '13 at 14:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.