Tell me more ×
Theoretical Computer Science Stack Exchange is a question and answer site for theoretical computer scientists and researchers in related fields. It's 100% free, no registration required.
up vote 6 down vote favorite
4

Are Shannon entropy and Boltzmann entropy mutually convertible, much like mass and energy according to Einstein's formula?

share|improve this question
Nothing theoretical but here is the Boltzmann formula on his grave in Vienna. – user1614113 Jan 30 at 11:57

3 Answers

up vote 15 down vote

In practice the only difference is that Boltzmann entropy deals with a thermodynamical constant $K_B$:

$ H = -K_B\sum_{i=1}^{N} P_i log_e\ P_i $

i assume you already know that if $K_B=1$ you have Shannon entropy in a different base. However the conceptual backgrounds are important; probability of events (Shannon) and probability of a particle being in one of $N$ states (Boltzmann) under a thermodynamical context.

There is theoretical work in making the formal link between the two notions of entropy. Take a look at the work of Edwin Jaynes.

http://en.wikipedia.org/wiki/Edwin_Jaynes

share|improve this answer
I would be interested to know whether there could be not only a formal link but sort of a practical link, much like when mass is transformed into energy. As layman I have a little bit "speculative" thought: If in a certain empirical study I have collected data, there is yet no information for me. I get information after I apply, say, a statistical software to the data. But that software must be run on a machine, where thermodynamical processes are certainly involved. So I "guess" that the informational entropy comes somehow from the thermodynamical entropy involved there. – Mok-Kong Shen Sep 27 '12 at 19:31
I cannot prove the following but i think in the case you mention, one should consider entropy on the first event when the data has been collected and contained. Even when at that point there is no practical information for a specific field, that is already information in terms of Shannon entropy. From here, there could be a natural representation of the data as particles for Boltzmann entropy. – labotsirc Sep 28 '12 at 3:40
3  
Changing the constant $K_B$ is the same thing as changing the base of the logarithm, because $\log_a x = \log_b x / \log_b a$. There is no difference between Shannon entropy and Boltzmann entropy except for a change of units. – Peter Shor Sep 29 '12 at 23:27
+1 for that observation – labotsirc Sep 30 '12 at 13:44
up vote 12 down vote

As has already been answered, Shannon entropy and Boltzman entropy are the same thing, although they are measured in different units. You also asked whether there is a practical link. It may not be practical yet, but the idea of algorithmic cooling uses the link between these two concepts, and has indeed been experimentally demonstrated.

share|improve this answer
up vote 2 down vote

I just found the following paragraphs from J. R. Pierce, An Introduction to Information Theory, p.206, which I surmise would also support the thinking that one could practically use a computer and a deterministic program to turn a string A into a string B of higher Shanon entropy:

"We can regard any process which specifies something concerning which state a system in in as a message source. This source generates a message which reduces our uncertainty as to what state the system is in. Such a source has a certain communication-theory entropy per message. This entropy is equal to the number of binary digits necessary to transmit a message generated by the source. It takes a particular energy per binary digit to transmit the message against a noise corresponding to the temperature T of the system."

"The message reduces our uncertainty as to what state the system is in, thus reducing the entropy (of statistical mechanics) of the system. The reduction of entropy increases the free energy of the system. But the increase in free energy is just equal to the minimum energy necessary to transmit the message which led to the increase of free energy, an energy proportional to the entropy of communication theory."

"This, I believe, is the relation between the entropy of communication theory and that of statistical mechanics. One pays a price for information which leads to a reduction of the statistical-mechanical entropy of a system. This price is proportional to the communication-theory entropy of the message source which procudes the information. It is always just high enough so that a perpetual motion machine of the second kind is impossible."

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.