4
$\begingroup$

If I have a multi hidden-layered neural network that is getting a better approximation for a function than a single one, does that mean that there is something "fishy" about my multi layered one because of the Universal Approximation Theorem?

I am assuming here that all weights are pre-initialized to 1 (I have my reasons) and they are trained on the same exact dataset.

$\endgroup$
  • $\begingroup$ Shouldn't multi-layered NN work better if you have enough training data? What are your concerns? $\endgroup$ – R B Mar 12 '15 at 12:26
  • $\begingroup$ I may have misread the UAT, but as far as I know, more layers should NOT improve the network performance which is why I was concerned in the first place. Unfortunately I have misplaced the source of this insight. $\endgroup$ – edgaralienfoe Mar 12 '15 at 13:17
  • $\begingroup$ I have found the source: "To summarize this concept, start without a hidden layer and examine the results. Then add one hidden layer and determine which network provides the best results, Adding additional layers should not improve network capabilities but may if the data contain complex patterns." This is taken from An Evalutation Of Artificial Neural Network Modelling For Manpower Analysis by Brian James Byrne. For the dataset, I am merely approximating a number of cosine functions and couldn't think how complex such a dataset could be as it only contains a range of real values from -1 to 1 $\endgroup$ – edgaralienfoe Mar 12 '15 at 13:25
  • $\begingroup$ How many internal nodes are you allowing in your single hidden layer network? With enough hidden nodes, it can compute (essentially) anything that's computable. $\endgroup$ – Aryeh Mar 12 '15 at 20:06
  • 1
    $\begingroup$ I feel this might be more suitable for Cross Validated. $\endgroup$ – Kaveh Mar 14 '15 at 19:03
2
$\begingroup$

Short answer: Not necessarily. Likely nothing fishy is going on.

Longer answer: The Universal Approximation Theorem (UAT) says nothing about an individual network's capacity to approximate a function. Moreover it says nothing about trainability or generalization.

The UAT roughly says that for any $\epsilon>0$ and any continuous real-valued function $F$ on compact domain $\Omega$ there exists a feed-forward network $f$ with a single hidden layer (with appropriate weights) such that $| F(x)-f(x)|<\epsilon$ for all $x\in\Omega$. Inductively using the shallow UAT, one can prove a deep UAT. So in theory depth offers no more expressiveness.

Note that this merely guarantees the existence of some appropriately tuned neural network, and not that any particular network can be trained up to that desired accuracy.

On the other hand, deep neural networks have been seen to be often much better in many applications. This is often attributed to their ability to learn hierarchical representations of the data. The UAT says nothing about generalization, which is the bar by which success is usually measured in practice. Here it is argued that deep networks can approximate compositional functions with less parameters than shallow counterparts.

But on the other-other hand, it was noticed that sometimes adding more depth can hurt when training a network. But why should it? One could in principle just set the new layer to be the identity. Apparently the later layers were actually trying to approximate the identity but doing it very poorly. So why not learn how different your layer is from the identity? This observation led to Residual Learning which in turn led to some really, really deep networks.

$\endgroup$
1
$\begingroup$

In case you're still looking for more information on this, I'll chip in my two cents.

It may help to think of neural networks as just fitting some function based on the training data. Each hidden layer increases the ability of the neural network to fit more complex functions. A network without hidden layers, often called a perceptron, is only able to fit linear (technically linearly separable) functions. As a rule of thumb, each hidden layer adds the ability to fit one higher degree polynomial function. So for example a network with one hidden layer should be able to fit a quadratic function, two hidden layers can fit a cubic function and so on.

Perhaps contrary to your intuition, the cosine you're trying to train is actually a complex dataset for a neural network, precisely because it fits polynomial functions. Implicitly the network will relax to weights that are in principle approximating your cosine with a polynomial. If you look at the tailor expansion for a sine for example, you'll see that a close approximation would require a 7th degree polynomial, which together with the rather small $\frac{1}{7 !}$ weight should prove quite a challenge for your network to train.

It might also be interesting to note that the approximation only holds well for $−1 < x < 1$, since a typical polynomial will quickly diverge from the sine/cosine beyond this interval. You might want to take that into account for your training set.

To wrap up it up, the primary reason you should be cautious with just adding hidden layers is the problem of overfitting. If you add too many layers (ie more layers than would be required for the -possibly unknown- polynomial underlying your training data), the network will typically fit too tightly to your training dataset, making it less useful for interpolation or extrapolation, or just evaluating new data outside the set.

A good example would actually be that tailor expansion again. If you read on in that section you'll come across the note that higher degree polynomials only marginally improve the accuracy in $−1 < x < 1$, while they perform (much) worse outside this interval. I should note that this may not exactly be the same problem, but it does illustrate what could go wrong.

$\endgroup$
  • 4
    $\begingroup$ I don't understand how adding a hidden layer gives the ability to fit one higher degree of a polynomial function, given a sufficient number of hidden nodes shouldn't just one hidden layer be enough to fit any continuous function? Multiple hidden layers would probably be able to perform better since you are adding an extra composition of the activation functions (I'm not absolutely certain of this) but I don't see how they're directly linked with the approximation of higher degree polynomials. $\endgroup$ – edgaralienfoe Apr 13 '15 at 10:53
  • $\begingroup$ I didn't mean to imply there actually is a direct mathematical link, in fact I'd be hard-pressed to rigorously argue why this should be so myself. I wouldn't even presume to do any hand-waving argument to linear algebraic curve fitting. As I said though, it's a rule of thumb which in my experience has proven to be sound. $\endgroup$ – Fasermaler Apr 17 '15 at 13:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.