5
$\begingroup$

In Cybenko's elegant proof of the Universal Approximation Theorem (UAT) he proves that single hidden layer neural networks (with linear output layer) are universal approximators whenever their activation functions are (what he calls) discriminatory. He then shows that bounded measureable sigmoidal functions are discriminatory, completing the proof. In particular, continuous sigmoidal functions work as well.

Later Leshno et al. proves that locally bounded piecewise continuous (l.b.p.c.) activation functions yield the UAT if and only if they are nonpolynomial.

My Question(s):

What kinds of functions are discriminatory? Polynomials certainly aren't (Leshno et al.) and continuous sigmoids are (Cybenko). Are ReLUs discriminatory?

Together, the above results imply that for l.b.p.c. functions, that discriminatory implies non-polynomial. Is the converse true?

$\endgroup$
-1
$\begingroup$

I think ReLUs does not satisfy Cybenko theory requirements, i.e. are unbounded. There is another recent paper showing UAT type theorem for ReLU feed forward deep networks using wavelet approximation property https://arxiv.org/abs/1509.07385

$\endgroup$
  • $\begingroup$ As mentioned, Cybenko proves that it is enough for the function to have his "discriminatory" property, and then proves continuous sigmoidal functions (which are bounded) satisfy that property. He is silent on whether unbounded functions like ReLUs can be discriminatory. $\endgroup$ – Christian Bueno Apr 11 '18 at 18:24
  • $\begingroup$ Looking through the paper I see no mention of the notion of discriminatory which is what I am specifically asking about. $\endgroup$ – Christian Bueno Apr 11 '18 at 18:44

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.