The universal approximation theorem (https://en.wikipedia.org/wiki/Universal_approximation_theorem) informally states that up to several conditions, any function can be approximated by a shallow neural network up to any degree.

I was wondering if there is a version of this theorem that states the same for the gradients of the functions, i.e, for any function there is a (shallow or not shallow) neural network that approximates it and its gradient approximates the gradient of the original function (up to any degree)?

  • $\begingroup$ A possible approach: given a continuous, differentiable function $f:[0,1]^n \to \mathbb{R}$ and its gradient $\nabla f$, first construct a piecewise linear function $g:[0,1]^n \to \mathbb{R}$ such that $g(x) \approx f(x)$ and $\nabla g(x) \approx \nabla f(x)$ everywhere, then implement or approximate $g(x)$ with a neural network. At least if $n=1$ I think that with a ReLU activation function a neural network can directly and exactly compute any desired continuous piecewise linear function (not just approximate it). I don't know if that's helpful. $\endgroup$ – D.W. Dec 30 '17 at 21:33
  • $\begingroup$ Actually, that's a good answer (arxiv.org/abs/1611.01491). I forgot to mention that I'm looking for smooth activation functions. So, ReLU is off the table. I actually solved it using zmjones.com/static/statistical-learning/hornik-nn-1991.pdf $\endgroup$ – tomerg Jan 1 '18 at 13:13

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.