3
$\begingroup$

Does there exist any theory (other than Cybenko's proof of the Universal Approximation Theorem with sigmoids) advocating the use of indicator functions as transfer functions for machine learning with neural networks?

After having read matus's beautiful answer in this thread explaining (among other things) Cybenko's proof, I wonder: if it weakens the approximation to use sigmoid transfer functions instead of indicator functions, what are the theoretical reasons for not using indicator functions?

As suggested here, perhaps it's because indicator functions have negative implications for generalization.

However, indicator functions are computationally far cheaper to implement than sigmoid functions, and also more closely resemble biological neural networks (ie the brain). Therefore, does there exist any other theory advocating the use of indicator functions as transfer functions for machine learning with neural networks?

|cite|improve this question
$\endgroup$
  • $\begingroup$ I think that a related paper (that talks about similar issues) can be found on faculty.georgetown.edu/kainen/Best.pdf Hope you find it useful. $\endgroup$ – user32135 Feb 18 '15 at 15:26
  • 1
    $\begingroup$ While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. $\endgroup$ – Jan Johannsen Feb 19 '15 at 8:55
2
$\begingroup$

Sigmoid functions have clear probabilistic interpretation, so one can derive optimal learning algorithms for them within the bayesian approach. Moreover, they are differentiable, so the gradient descent can be directly applicable. However, in general, neural networks are "universal approximators" in almost the same sense as, for example, polynomials. They can capture a limited set of regularities. Different activation functions (like different basis functions in functional approximation) will capture different regularities and will have different biases in approximating other regularities. Thus, what activation functions are better depends on what underlying regularities are presented in data. Indicator functions can surely be better for some tasks. However, no generally applicable (task-independent) theory advocating the use of indicator functions is possible.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ thanks! 1) would you know of any teaching material / papers giving the different regularities that various activation functions are good at capturing? 2) by regularities, do you mean properties of the target function, or of the underlying probability distribution, or of the sampling (ie training set)? Eg for sampling, Jarrett et al notice that $\mid$tanh(x)$\mid$ prevents overfitting in their image recognition task, would be nice to know why $\endgroup$ – Alexandre Holden Daly Jan 21 '14 at 13:30
  • 2
    $\begingroup$ 1) Unfortunately, I have not encountered a detailed analysis of this problem. However, we have one paper on a related subject, but it might not be what you want, since it considers somewhat exotic networks. Nevertheless, here is the link I hope, it will not be absolutely useless. $\endgroup$ – Necro0x0Der Jan 21 '14 at 14:06
  • $\begingroup$ 2) Both. Learning methods can express some regularities (computable function), and can approximate others. E.g. polinomial basis can represent only polynomial regularities, but you can approximate sine function with it. Data can be produced by a source, which "true" model is arbitrary (stochastic) algorithm. The question is how precisely regularities expressible by a method match regularities in data. $\endgroup$ – Necro0x0Der Jan 21 '14 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.