I am reading this classic paper by Claude E. Shannon and I think there may be a couple of errors in his description of the properties of Entropy/Uncertainty. The screenshot shown at the bottom of this post is from page 9. Firstly, I believe there is a typo at the first red circle which should read $10^{50}$. Secondly, I think the equation shown for the value of the entropy $H$ of this problem is also incorrect. I believe the correct equation for $H$ is derived as follows:

events $=\{p_1, p_2, ... , p_{10^{30}}, q_1, q_2, ..., q_{10^{50}}\}$

where events $p_i$ each have a probability $\frac{1}{2}(10^{-30})$ of occurring, and events $q_j$ each have a probability $\frac{1}{2}(10^{-50})$ of occuring

\begin{align} H &= -\sum_{i=1}^{10^{30}} p_i\log{p_i} - \sum_{j=1}^{10^{50}} q_j\log{q_j} \\ &= -\sum_{i=1}^{10^{30}} \frac{1}{2}(10^{-30})\log{(\frac{1}{2}(10^{-30}))} -\sum_{j=1}^{10^{50}} \frac{1}{2}(10^{-50})\log{(\frac{1}{2}(10^{-50}))} \\ &= -\frac{1}{2}\log{(\frac{1}{2}(10^{-30}))} -\frac{1}{2}\log{(\frac{1}{2}(10^{-50}))} \\ &\approx 40 \end{align}

Hence, I think the equation for $H$ should read:

$H = -\frac{1}{2}\log{\frac{1}{2}10^{-30}} -\frac{1}{2}\log{\frac{1}{2}10^{-50}}$

I would really appreciate if someone could let me know if I'm correct or where I'm wrong. I am not doubting Claude, just wondering if the document itself has typos. Thanks!

page 9 part 8 of paper

  • $\begingroup$ Yes, you are right, these are typos. Btw, you can simplify $-\frac12\log(\frac1210^{-30})-\frac12\log(\frac1210^{-50})$ to $\frac12\log(2\cdot10^{30})+\frac12\log(2\cdot10^{50})=40+\log2$ (I assume from the context these are base-10 logarithms). $\endgroup$ – Emil Jeřábek Mar 22 at 9:57
  • $\begingroup$ Thanks, yes they're base-10. That's a cool simplification. $\endgroup$ – nuggimane Mar 22 at 11:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.