0
$\begingroup$

I am completely new to information theory. I was learning about information content but couldn’t make sense of why the relationship between information content and probability isn’t linear? And why it is sub-linear? As the formula goes,

$$ I(E) = -\log p(E). $$

$\endgroup$
1
  • 4
    $\begingroup$ Why do you think the relationship between information content and probability should be linear? The formula for the information in a random variable is $\sum_x p_x I(x)$, where $p_x$ is the probability of outcome $x$. If a random variable takes on $2^k$ values with equal probability, the information in this random variable is $k$ bits, and from this we can deduce that $I(x) = - \log_2(p_x)$. $\endgroup$ – Peter Shor Dec 29 '20 at 15:41
0
$\begingroup$

So definition of self-information is chosen to be as follows (shamelessly copied from wiki page of "Information content" ) :

  1. An event with probability 100% is perfectly unsurprising and yields no information.

  2. The less probable an event is, the more surprising it is and the more information it yields.

  3. If two independent events are measured separately, the total amount of information is the sum of the self-informations of the individual events.

Point 3 of the definition answers your question.

If $A$ and $B$ independent events and C is an event where both $A$ and $B$ take place then $$P(C) = P(A \cap B) = P(A)\cdot P(B)$$ On the other hand information from $C$ is defined as follows: $$I(C) = I(A)+I(B)$$ Let $f(\cdot)$ be the functional form of the information expression, formally $$I(E) = f(P(E))$$ Therefore, $$f(P(C)) = f(P(A)) + f(P(B))$$ $$ = f(P(A).P(B))$$

Can you find an $f(x)$ which is of the form $k\cdot x$, such that the above properties are satisfied ? I think the answer is no.

But I would like to add that I don't recall very neat reasons for why it has to be only $k\log(x)$ as well. i.e why only $f(x) = k\log x $ satisfies $f(x\cdot y) = f(x)+f(y)$

$\endgroup$
1

Not the answer you're looking for? Browse other questions tagged or ask your own question.