I was following the textbook by David Mackay: Information theory inference and learning algorithms.
I have question on Shannon's source coding theorem (p81):
$N$ i.i.d. random variables each with entropy $H(x)$ can be compressed into more than $N \cdot H(x)$ bits with negligible risk of information loss, as $N \to\infty$; conversely if they are compressed into fewer than $N \cdot H(x)$ bits it is virtually certain that information will be lost.
My question is the bold part of the above: what happens when you go below $N \cdot H(x)$ bits?
There is an example in the textbook (p77), where he shows that, as $N$ increases, the average entropy per symbol approaches $H(X)$, for a risk tolerance $0 \lt \delta \lt 1$. (So it becomes flatter as indicated on the diagram.) Therefore, $N \cdot H(x)$ is the number of bits you can compress no matter how much risk tolerance you are willing to take.
In the definition above, however, it is saying that information will be lost if you go below $N \cdot H(X)$. My question is, how much information will be lost?
I found something on the web regarding this question: http://www.kim-bostroem.de/Library/Notes/Shannon.pdf
It is saying that for each symbol, if you compress below $H(x)$ bits, then for $N \to\infty$, we will lose all the information. Can someone please confirm this?