Does there exist any theory (other than Cybenko's proof of the Universal Approximation Theorem with sigmoids) advocating the use of indicator functions as transfer functions for machine learning with neural networks?
After having read matus's beautiful answer in this thread explaining (among other things) Cybenko's proof, I wonder: if it weakens the approximation to use sigmoid transfer functions instead of indicator functions, what are the theoretical reasons for not using indicator functions?
As suggested here, perhaps it's because indicator functions have negative implications for generalization.
However, indicator functions are computationally far cheaper to implement than sigmoid functions, and also more closely resemble biological neural networks (ie the brain). Therefore, does there exist any other theory advocating the use of indicator functions as transfer functions for machine learning with neural networks?