# Using Indicator Functions as Transfer Functions for Neural Networks

Does there exist any theory (other than Cybenko's proof of the Universal Approximation Theorem with sigmoids) advocating the use of indicator functions as transfer functions for machine learning with neural networks?

After having read matus's beautiful answer in this thread explaining (among other things) Cybenko's proof, I wonder: if it weakens the approximation to use sigmoid transfer functions instead of indicator functions, what are the theoretical reasons for not using indicator functions?

As suggested here, perhaps it's because indicator functions have negative implications for generalization.

However, indicator functions are computationally far cheaper to implement than sigmoid functions, and also more closely resemble biological neural networks (ie the brain). Therefore, does there exist any other theory advocating the use of indicator functions as transfer functions for machine learning with neural networks?

• I think that a related paper (that talks about similar issues) can be found on faculty.georgetown.edu/kainen/Best.pdf Hope you find it useful. – user32135 Feb 18 '15 at 15:26
• While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. – Jan Johannsen Feb 19 '15 at 8:55

• thanks! 1) would you know of any teaching material / papers giving the different regularities that various activation functions are good at capturing? 2) by regularities, do you mean properties of the target function, or of the underlying probability distribution, or of the sampling (ie training set)? Eg for sampling, Jarrett et al notice that $\mid$tanh(x)$\mid$ prevents overfitting in their image recognition task, would be nice to know why – Alexandre Holden Daly Jan 21 '14 at 13:30