All Questions
Tagged with shannon-entropy it.information-theory
31 questions
1
vote
1
answer
142
views
Information Bottleneck - Calculating the Mutual information between the Labels and the Features [closed]
I am trying to understand the Nonlinear Information Bottlecneck paper along with their implementation, but I am confused as to what is actually being calculated in the Mutual information $(I(Y, M))$ ...
- 41
0
votes
0
answers
64
views
Interesting statistical experiment concerning data compression
I want to present the following statistical experiment concerning data compression, on which I will ask you to predict the result obviously justifying the choice made.
The statistical experiment is ...
- 1
5
votes
0
answers
280
views
Maximize the mutual information between 2 discrete random variables
I have two random variables $X$ and $Y$. $X$ follows Poisson-Binomial distribution with parameters $\{q_1, \ldots, q_k\}$. Thus, $X$ can take values in the set $\{0,1,\ldots,k\}$.
$Y$ is a binary ...
- 109
4
votes
1
answer
209
views
Does this notion of entropy have a name?
Recently I stumbled upon the following notion of entropy which seems quite natural to me. I am looking for its "real" name and/or any references where it might come up. I tried searching ...
- 300
0
votes
0
answers
178
views
Error in entropy properties in Mathematical Theory of Cryptography by Claude E. Shannon
I am reading this classic paper by Claude E. Shannon and I think there may be a couple of errors in his description of the properties of Entropy/Uncertainty. The screenshot shown at the bottom of this ...
0
votes
1
answer
123
views
Why isn’t information-probability relationship linear? [closed]
I am completely new to information theory.
I was learning about information content but couldn’t make sense of why the relationship between information content and probability isn’t linear? And why it ...
- 133
0
votes
0
answers
16
views
Capacity of spike-based neuronal code
Assume that a neuronal population $A$ is connected to a neuronal population $B$ by a bunch of synapses - one-directional channels that propagate spikes. For simplicity assume that the current ...
- 101
4
votes
1
answer
202
views
Generating $k$ random bits from a pdf with entropy $H(p) = k$
All the sources online say that, intuitively, a distribution with entropy $k$ has $k$ bits of pure randomness in it.
So can we formalize this as follows? Suppose I can only sample from my distribution,...
- 649
0
votes
1
answer
158
views
Volume of elements mapped to the same codeword is $2^{H(X|\hat{X})}$
In this paper by Tishby, Pereira and Bialek they mention on page 4 in the Relevant quantization chapter the setting is the following; Given some signal space $X \sim p(x)$ and a quantized codebook $\...
- 133
2
votes
0
answers
111
views
Representing data with Shannon entropy predicted bits
Let us assume a file based on a character set where each character has equal probability of occurance. This will result in the maximum entropy for that character set. On calculating the entropy, let ...
- 121
1
vote
0
answers
473
views
Minimum number of hours of speech needed to train a neural net to recognize speech [closed]
From a theoretical computer science point of view, is there a lower limit on the number of hours of speech needed to train a neural net to translate speech to text? An estimate from CMU is 3000-5000 ...
- 107
4
votes
1
answer
147
views
Can entropicly secure encryption algorithms be used on low-entropy messages by adding noise
There exist information-theoretic notions of security like Shannon's "perfect security" that one-time pads exhibit. All methods which achieve perfect security will require long keys, however. If we ...
- 1,213
4
votes
1
answer
1k
views
Relation between variance and mutual information
Given two discrete random variables $X,Y$ such that $X,Y \in \mathbb{R}$ and $0 \leq X,Y \leq 1$, is it true that $$|\text{Cov}[X,Y] \leq \sqrt{\frac{1}{2} \text{I}[X,Y]}|. $$
This bound may be ...
- 43
4
votes
1
answer
500
views
Word length using entropy : Maximum entropy criteria
The question is based on research paper titled, Markovian language model of the DNA and its information content
In the supplementary document, the Authors show how they determine the word length of ...
- 141
-2
votes
1
answer
173
views
Lower bound on the number of objects in the universe [closed]
From Cover & Thomas' Elements of Information Theory:
Player A chooses some object in the universe,
and player B attempts to identify the object with a series of yes–no
questions. Suppose ...
4
votes
2
answers
270
views
Conditional entropy: $H(X | Y)$ large implies $H(X | Y, X \neq Y)$ large?
Suppose that $X$ and $Y$ are two random variables that are defined on the same support. Furthermore, suppose that $H(X | Y) = \log n$ for some $n$. I am now interested in how much the term $H(X | Y, X ...
- 79
2
votes
1
answer
91
views
Sufficient Statistics of $X$ from $Y$
I am reading the paper New Monotone and Lower Bounds in Unconditional Two Party Computation by Wolf and Wullschleger.
In Definition 2 on the third page, they define $f(x):=P_{Y|X}(\cdot|x)$ and they ...
- 435
8
votes
4
answers
817
views
Expected empirical entropy
I'm thinking about some properties of the empirical entropy for binary strings of length $n$ when the following question crosses my way:
$\underbrace{\large\frac{1}{2^{n}}\normalsize\sum\limits_{w\in\...
- 500
4
votes
0
answers
123
views
Inf-entropy rate and min-entropy
I am reading the paper "Generating random bits from an arbitrary source: fundamental limits" by Vembu and Verdu. This paper is written in the language of information theory, however, I need to ...
- 435
4
votes
3
answers
5k
views
Difference between self-information and entropy
I get a bit confused about different definitions of entropy and/or self-information.
Entropy?
$$ H(X) = - \sum_{x \in X} P_X(x) \cdot \log{\left(P_X(x)\right)} $$
Self-information?
$$ I(x) = - \...
- 153
0
votes
0
answers
197
views
Can the self-information be infinite?
I was wondering about the self-information, the information content . If I have data and I measure different words in it, their probability and take the average mean of that, what is the lowest and ...
- 153
8
votes
2
answers
167
views
Smoothly leaking information over time
Say I have a one bit random variable $X \in \{0,1\}$, and let $n$ be a natural number. I want a sequence of random variables $0 = X_0, X_1, \ldots, X_n = X$ s.t.
$$H\left(X~|~\{X_0,\ldots,X_k\}\...
- 3,313
7
votes
0
answers
200
views
Geometric Intuition behind Locally testable codes
Conventional coding theory provides a good geometric picture behind linear error correction codes in terms of Hamming distance. What additional geometric requirement one should add to make a code ...
- 13.3k
5
votes
1
answer
606
views
Given discrete rvs X,Y, find Z s.t. I(Z;X) is high and I(Z;Y) is low. -- known problem?
Consider the following problem. Let $X$ and $Y$ be discrete random variables. The goal is to find a random variable $Z$ such that, informally, $I(Z;X)$ is high and $I(Z;Y)$ is low.
More precisely, ...
- 113
1
vote
1
answer
205
views
How to choose a correct prior
Consider a Bernoulli experiment, such as flipping a not necessarily fair coin, which results in a positive outcome (heads) with probability $p$ and with a negative outcome (tails) with probability $(1-...
- 121
7
votes
3
answers
4k
views
Are Shannon entropy and Boltzmann entropy mutually convertible?
Are Shannon entropy and Boltzmann entropy mutually convertible, much like mass and energy according to Einstein's formula?
- 207
13
votes
3
answers
3k
views
On the entropy of a sum
I am looking for a bound on the entropy $H(X+Y)$ of the sum of two independent discrete random variables $X$ and $Y$. Naturally, $$H(X+Y) \leq H(X) + H(Y) ~~~~~~(*)$$ However, applied to the sum of $n$...
- 775
1
vote
0
answers
618
views
Information channel with symmetric channel matrix
It took me a while to figure out that a "symmetric channel" does not mean a channel with a symmetric channel matrix. (Rather, "symmetric channel" means that the rows of the matrix are all permutations ...
- 111
1
vote
2
answers
870
views
Arithmetic coding, the termination symbol, and the empty string
Suppose the source alphabet is $a, b, c$ with $a$ as the termination symbol and so the unit interval is correspondingly divided as
$[0, P(a), P(a)+P(b), 1]$.
Strings consisting of a bunch of $b$'s ...
- 83
2
votes
1
answer
305
views
Landauer's principle internals - how it works
I attached a picture, where the energy dissipation (entropy increase) on information erasure is explained. Is the explanation correct?
"RESTORE TO ONE" - is it correct to identify the operation as "...
- 431
12
votes
2
answers
458
views
Channel coding results using Kolmogorov complexity
Usually Shannon entropy is used to prove channel coding results. Even for source-channel separation results shannon entropy is used. Given the equivalence between Shannon (global) vs Kolmogorov (local)...
- 2,228