Universal Approximation Theorem for non-sigmoidal activation functions

The most cited Universal Approximation Theories for multi-layer feedforward neural networks by Cybenko (1989) and Hornik (1991) assume the activation functions of the network to be sigmoidal. However, currently one mainly uses ReLU activation functions (or variants) which are not sigmoidal (not even bounded).

I found a proof of the Approximation Theory summarized in Pinkus: Approximation theory of the MLP model in neural networks (1999), where he shows the approximation result for continuous activation functions $\sigma \in C(\mathbb{R})$ if $\sigma$ is not polynomial. Unfortunately, he only does this for the case of one hidden layer. I am unable to find a proof that also covers multi-hidden-layer neural networks with continuous activation functions that are not sigmoidal.

Does anyone know of such a result? Or is it easy to extend the result by Pinkus mentioned above to a network with more hidden layers?

• Doesn't the single layer proof imply the multi-layer one? Just set layers 1 up to n-1 to compute the identity function and use the single-layer proof to approximate the function using the final hidden layer. – user4242 Jun 29 '17 at 16:11
• I thought that would not be possible if the activation functions in every layer have to be the same. But in the case of the ReLU this might be relatively easy if we want to prove uniformly dense on compacta (in $C(\mathbb{R}^n)$). For each compact $K \subset \mathbb{R}^n$ we can choose the biases such that the weighted input of the neurons in layers $1,\ldots, n-1$ are in the linear part of the ReLU activation function (>0). This settles it for the ReLU, but does not work for general continuous activation functions. Am I correct? – Robo Jun 29 '17 at 18:37
• @Robo Doesn't Theorem 2.4 in our paper, eccc.weizmann.ac.il/report/2017/098 give what you want? – Anirbit Jun 29 '17 at 19:11
• @Anirbit It does cover it for the case of the ReLU deep neural networks I think. In proving Corollary 2.2, do you use that you can represent any affine linear function (you mean an affine function of the form $A\cdot x + b$?) $l_n$ with a ReLU DNN before you apply the lemma's in the appendix? Thanks! Anyway, this only proves the result for ReLU activation functions and not for general continuous activation functions. – Robo Jun 29 '17 at 20:19
• I am a bit confused your question. Corollary 2.2 is the proof that every R^n -> R continuous piecewise linear function is representable by a log depth ReLU DNN. Theorem A.2, A.3 and Theorem 2.1 all go into its proof. – Anirbit Jul 3 '17 at 2:21