Disclaimer: I know very (very) little about deep nets, besides what an introductory course on machine learning would teach on neural networks, and skimming some paper abstracts and introductions.
If I understood correctly the concept, an autoencoder is a specific case of neural networks with as many input nodes than output nodes (say $d$), whose goal is to approximate the identity function $$ \operatorname{id}\colon \mathbb{R}^d \to \mathbb{R}^d\,. $$ The trick is that this net can be broken into two parts, "encoder" and "decoder" of the form $\Phi\colon \mathbb{R}^d \to \mathbb{R}^k$ and $\Psi\colon \mathbb{R}^k \to \mathbb{R}^d$, such that $k\ll d$. Therefore, there has to be something non-trivial going on for $\Psi\circ \Phi$ to approximate the identity, given the restricted width of the middle layers.
This struck me as begging the question:
Can one (and has one already?) formalize and analyze this rigorously in terms of information theory, and specifically compression?
I realize the problem itself would need a more thorough and precise formulation to be tackled, as in the above we deal with full real numbers (so "hiding" information in a single real number, say the weight of a node in the middle layer, would give a very easy way out). But with this dealt with appropriately, hopefully there are non-trivial statements to be made with regard to what compression can be achieved, with respect to some distribution over the inputs?
Has this type of question been looked at from a theoretical viewpoint, in our community or another?
