0
$\begingroup$

I was wondering about the self-information, the information content . If I have data and I measure different words in it, their probability and take the average mean of that, what is the lowest and what is the biggest outcome?

$$I(data) = \frac{\sum_{i=1}^{n} - log(P(word_i))}{n}$$

The summed up probability is one, but how does that change with the logarithm? It seems, if one word is very unique and another word takes up the rest of the data in a big quantity, than the information content is higher:

$$\frac{(-\log 0.9999) + (-\log 0.0001)}{2} > \frac{(-\log 0.9) + (-\log 0.1)}{2}$$

Does that mean, if one of the words limits to zero, that the information content becomes infinite? Does that make sense, seems strange.

Improve this question
$\endgroup$
6
  • $\begingroup$ If the frequency is actually zero, then that word should simply not be counted. But as the frequency of a word becomes very small, its information content does become unboundedly large. This makes sense if you view the information from word_i as the amount of "surprise" you would have if you saw word_i after drawing randomly from your data. It also makes sense if you view the information from word_i as the number of bits needed to specify its frequency. If its frequency is very small, say $2^{-30}$, then you need very many (in this case, 30) bits to specify it. $\endgroup$
    – Joshua Grochow
    Commented Oct 1, 2013 at 20:11
  • 4
    $\begingroup$ I don't think this is the standard definition of the information content, which is the weighted average of the self-information of the events, that is, the entropy: $I(\mathrm{data}) = \frac{1}{n} \sum -P(\mathrm{word}_i) \log P(\mathrm{word}_i).$ The quantity you are asking about can certainly be infinite. $\endgroup$
    – Peter Shor
    Commented Oct 2, 2013 at 19:06
  • $\begingroup$ I didn't even notice - my previous comment applies to the standard definition of information content as in Peter Shor's comment. $\endgroup$
    – Joshua Grochow
    Commented Oct 2, 2013 at 19:10
  • $\begingroup$ @JoshuaGrochow Mh, so I only meant self-information here and for self-information I find the definition $- \log{P(word_i)}$. The definition you have seems to be the averaged entropy? $\endgroup$
    – Guarana Joe
    Commented Oct 3, 2013 at 11:40
  • 1
    $\begingroup$ The self-information for a single word is indeed $-\log P(\mathrm{word}_i)$. The entropy is the weighted average of the self-information over all words. What you have defined is the average self-information over all words. I don't know what this quantity if used for, but it is certainly not the information content, as you seem to think it is. If you take some samples from a distribution, and take the average self-information of this, it does indeed correspond to an information content, but it also converges to the entropy as you take more and more samples. $\endgroup$
    – Peter Shor
    Commented Oct 3, 2013 at 14:27

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.