The shannon-entropy tag has no wiki summary.
1
vote
1answer
168 views
How to choose a correct prior
Consider a Bernoulli experiment, such as flipping a not necessarily fair coin, which results in a positive outcome (heads) with probability $p$ and with a negative outcome (tails) with probability ...
6
votes
3answers
1k views
Are Shannon entropy and Boltzmann entropy mutually convertible?
Are Shannon entropy and Boltzmann entropy mutually convertible, much like mass and energy according to Einstein's formula?
8
votes
3answers
463 views
On the entropy of a sum
I am looking for a bound on the entropy $H(X+Y)$ of the sum of two independent discrete random variables $X$ and $Y$. Naturally, $$H(X+Y) \leq H(X) + H(Y) ~~~~~~(*)$$ However, applied to the sum of ...
1
vote
0answers
167 views
Information channel with symmetric channel matrix
It took me a while to figure out that a "symmetric channel" does not mean a channel with a symmetric channel matrix. (Rather, "symmetric channel" means that the rows of the matrix are all permutations ...
1
vote
2answers
495 views
Arithmetic coding, the termination symbol, and the empty string
Suppose the source alphabet is $a, b, c$ with $a$ as the termination symbol and so the unit interval is correspondingly divided as
$[0, P(a), P(a)+P(b), 1]$.
Strings consisting of a bunch of $b$'s ...
1
vote
1answer
112 views
Landauer's principle internals - how it works
I attached a picture, where the energy dissipation (entropy increase) on information erasure is explained. Is the explanation correct?
"RESTORE TO ONE" - is it correct to identify the operation as ...
2
votes
2answers
320 views
Can reversible computations alone be used to create a computer?
We are able to perform universal computations with the reversible model. Basically, during the computations, no information should be erased, so that no involved entropy increase would occur.
Are ...
5
votes
0answers
118 views
Channel coding results using Kolmogorov complexity
Usually Shannon entropy is used to prove channel coding results. Even for source-channel separation results shannon entropy is used. Given the equivalence between Shannon (global) vs Kolmogorov ...
