I am writing a program using high-dimensional volume (HDV) estimator to estimate entropy and mutual information for variable selection. Let $ D = (x^i_1, x^i_2, ..., x^i_M)$, N is the number of data points in the $M$ dimensional joint space.

The HDV estimators based on the equation as below:

\begin{aligned} H(X) ≈ −ψ(k) + ψ(N) + \frac{1}{N}{\sum_{i=1}^{N}{logB_m(r^{i}_{x^m})}^m} \end{aligned}

$X$ is the random variable, $H(X)$ is the entropy of $X$, $ψ(.)$ is the digamma function, $k=1$ for HDV estimator, $N$ is the number of observations, $B_m$ is the volume of m-dimensional unit ball $r^{i}_{x^m}$ is the minimum distance in each dimension:

\begin{aligned} r^{i}_{x^m} = min(||x^i_m-x^{'i}_m||) \end{aligned}

where $x^i_m \neq x^{'i}_m$ and $x^i_m , x^{'i}_m \in D$

My question is that when it is applied to estimate the entropy of Gaussian distribution with mean = 0 and standard deviation = 1, the theoretical result of entropy is around 1.4189. I cannot get the result because $−ψ(1)≈ 0.5772156$ (Euler-Mascheroni constant), $ψ(N=1000)≈ 6.9072$ (Say 1,000 of data points) and I think that $\frac{1}{N}{\sum_{i=1}^{N}{logB_m(r^{i}_{x^m})}^m}$ cannot be negative as it is about distance and volume. What is wrong of my understanding?

  • 1
    $\begingroup$ might be more suitable for the stats.SE. $\endgroup$ – Kaveh Aug 31 '11 at 16:31
  • 1
    $\begingroup$ I agree. you might try posting it there. $\endgroup$ – Suresh Venkat Aug 31 '11 at 16:57
  • 1
    $\begingroup$ I cannot comment on the estimator definition, but note that the volume of an $m$-dimensional unit ball is actually exponentially decreasing in $m$. I think $\log B_m$ is on the order of $-O(m\log m)$. $\endgroup$ – Sasho Nikolov Aug 31 '11 at 20:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.