Theoretical Computer Science Stack Exchange is a question and answer site for theoretical computer scientists and researchers in related fields. It only takes a minute to sign up.
Sign up to join this community
If I have a multi hidden-layered neural network that is getting a better approximation for a function than a single one, does that mean that there is something "fishy" about my multi layered one because of the Universal Approximation Theorem?
I am assuming here that all weights are pre-initialized to 1 (I have my reasons) and they are trained on the same exact dataset.
Short answer: Not necessarily. Likely nothing fishy is going on.
Longer answer: The Universal Approximation Theorem (UAT) says nothing about an individual network's capacity to approximate a function. Moreover it says nothing about trainability or generalization.
The UAT roughly says that for any $\epsilon>0$ and any continuous real-valued function $F$ on compact domain $\Omega$ there exists a feed-forward network $f$ with a single hidden layer (with appropriate weights) such that $| F(x)-f(x)|<\epsilon$ for all $x\in\Omega$. Inductively using the shallow UAT, one can prove a deep UAT. So in theory depth offers no more expressiveness.
Note that this merely guarantees the existence of some appropriately tuned neural network, and not that any particular network can be trained up to that desired accuracy.
On the other hand, deep neural networks have been seen to be often much better in many applications. This is often attributed to their ability to learn hierarchical representations of the data. The UAT says nothing about generalization, which is the bar by which success is usually measured in practice. Here it is argued that deep networks can approximate compositional functions with less parameters than shallow counterparts.
But on the other-other hand, it was noticed that sometimes adding more depth can hurt when training a network. But why should it? One could in principle just set the new layer to be the identity. Apparently the later layers were actually trying to approximate the identity but doing it very poorly. So why not learn how different your layer is from the identity? This observation led to Residual Learning which in turn led to some really, really deep networks.
In case you're still looking for more information on this, I'll chip in my two cents.
It may help to think of neural networks as just fitting some function based on the training data. Each hidden layer increases the ability of the neural network to fit more complex functions. A network without hidden layers, often called a perceptron, is only able to fit linear (technically linearly separable) functions. As a rule of thumb, each hidden layer adds the ability to fit one higher degree polynomial function. So for example a network with one hidden layer should be able to fit a quadratic function, two hidden layers can fit a cubic function and so on.
Perhaps contrary to your intuition, the cosine you're trying to train is actually a complex dataset for a neural network, precisely because it fits polynomial functions. Implicitly the network will relax to weights that are in principle approximating your cosine with a polynomial. If you look at the tailor expansion for a sine for example, you'll see that a close approximation would require a 7th degree polynomial, which together with the rather small $\frac{1}{7 !}$ weight should prove quite a challenge for your network to train.
It might also be interesting to note that the approximation only holds well for $−1 < x < 1$, since a typical polynomial will quickly diverge from the sine/cosine beyond this interval. You might want to take that into account for your training set.
To wrap up it up, the primary reason you should be cautious with just adding hidden layers is the problem of overfitting. If you add too many layers (ie more layers than would be required for the -possibly unknown- polynomial underlying your training data), the network will typically fit too tightly to your training dataset, making it less useful for interpolation or extrapolation, or just evaluating new data outside the set.
A good example would actually be that tailor expansion again. If you read on in that section you'll come across the note that higher degree polynomials only marginally improve the accuracy in $−1 < x < 1$, while they perform (much) worse outside this interval. I should note that this may not exactly be the same problem, but it does illustrate what could go wrong.
