I have trained a neural network using MATLAB and am ready to deploy it into my software. Right now, I include it in my software by programming the structure of the neural network and the connection weights and doing some matrix multiplication.

But there has to be a better way to do this: neural networks are basically linear separators. This means that it should be possible to model the outputs as some linear function of the inputs.

So here's the question: given the structure of a neural network and the connection weights after training, how would I come up with the correct linear function of the inputs that models the functionality of the network?


1 Answer 1


It would seem to depend on the complexity of your neural network. A single perceptron is a linear separator. On the other hand, it is known that multilayer feed-forward neural networks are universal approximators (see here: http://dl.acm.org/citation.cfm?id=70408 ): i.e. they can approximate any function to arbitrary accuracy. Obviously for such a network, you should not expect to be able to express its outputs as a linear function of its inputs.

  • $\begingroup$ Sure, they can approximate functions to arbitrary accuracy. But the approximation that they learn is ultimately also a function, yes? So wouldn't that mean that I could write a function `f(x1, x2, ..., xn) -> (y1, y2, ..., yn) that perfectly models this approximation? $\endgroup$ Mar 8, 2012 at 16:19
  • 1
    $\begingroup$ Of course -- a neural network represents a function. But not necessarily a linear function. $\endgroup$
    – Aaron Roth
    Mar 8, 2012 at 16:31
  • $\begingroup$ Please correct me if I'm wrong, but do you mean "not a linear function IN 2 DIMENSIONS"? It's my understanding that it's always a linear function, possibly, though in higher dimensional spaces, which would make it super-linear in 2-D space $\endgroup$ Mar 8, 2012 at 16:42
  • $\begingroup$ Incidentally, if you don't apply any sigmoid functions (ie. all neurons are linear) then any multilayer perceptron is also a linear function (every layer is a linear function, so you're just chaining linear functions). The point is that if your NN is a linear function, then you may as well be training a linear function (a single layer NN without sigmoid), or better yet, just calculate the optimal solution directly with least squares. What you can do is compute each layer as a linear function and apply the sigmoid to the resulting vector. $\endgroup$
    – Peter
    Mar 8, 2012 at 16:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.