I am completely new to information theory. I was learning about information content but couldn’t make sense of why the relationship between information content and probability isn’t linear? And why it is sub-linear? As the formula goes,
$$ I(E) = -\log p(E). $$
So definition of self-information is chosen to be as follows (shamelessly copied from wiki page of "Information content" ) :
An event with probability 100% is perfectly unsurprising and yields no information.
The less probable an event is, the more surprising it is and the more information it yields.
If two independent events are measured separately, the total amount of information is the sum of the self-informations of the individual events.
Point 3 of the definition answers your question.
If $A$ and $B$ independent events and C is an event where both $A$ and $B$ take place then $$P(C) = P(A \cap B) = P(A)\cdot P(B)$$ On the other hand information from $C$ is defined as follows: $$I(C) = I(A)+I(B)$$ Let $f(\cdot)$ be the functional form of the information expression, formally $$I(E) = f(P(E))$$ Therefore, $$f(P(C)) = f(P(A)) + f(P(B))$$ $$ = f(P(A).P(B))$$
Can you find an $f(x)$ which is of the form $k\cdot x$, such that the above properties are satisfied ? I think the answer is no.
But I would like to add that I don't recall very neat reasons for why it has to be only $k\log(x)$ as well. i.e why only $f(x) = k\log x $ satisfies $f(x\cdot y) = f(x)+f(y)$
