The universal approximation theorem (https://en.wikipedia.org/wiki/Universal_approximation_theorem) informally states that up to several conditions, any function can be approximated by a shallow neural network up to any degree.

I was wondering if there is a version of this theorem that states the same for the gradients of the functions, i.e, for any function there is a (shallow or not shallow) neural network that approximates it and its gradient approximates the gradient of the original function (up to any degree)?

  • $\begingroup$ A possible approach: given a continuous, differentiable function $f:[0,1]^n \to \mathbb{R}$ and its gradient $\nabla f$, first construct a piecewise linear function $g:[0,1]^n \to \mathbb{R}$ such that $g(x) \approx f(x)$ and $\nabla g(x) \approx \nabla f(x)$ everywhere, then implement or approximate $g(x)$ with a neural network. At least if $n=1$ I think that with a ReLU activation function a neural network can directly and exactly compute any desired continuous piecewise linear function (not just approximate it). I don't know if that's helpful. $\endgroup$ – D.W. Dec 30 '17 at 21:33
  • $\begingroup$ Actually, that's a good answer (arxiv.org/abs/1611.01491). I forgot to mention that I'm looking for smooth activation functions. So, ReLU is off the table. I actually solved it using zmjones.com/static/statistical-learning/hornik-nn-1991.pdf $\endgroup$ – tomerg Jan 1 '18 at 13:13

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.