# Neural Networks: what's the point of learning features that don't linearly separate?

Unless I'm mistaken, deep neural networks are good for learning functions that are nonlinear in the input.

In such cases, the input set is linearly inseparable, so the optimisation problem that results from the approximation problem is not convex, so it cannot be globally optimised with local optimization. Support Vector Machines (try to) get around this by choosing features such that the projection of the input space into the feature space is linearly separable, and we have a convex optimisation problem once again.

I don't suppose that a deep neural network always learns features that make the input set linearly separable in feature space. So what's the point in it learning features?

• can you rephrase that last question "whats the point in it learning features?" it seems not a question or to not be what you are intending to ask. plausibly the whole point/raison d'etre of ML is to learn features. also, ANNs have been learning nonlinear functions ever since early days (1980s), that in particular is not a twist of deep learning. maybe what this question is getting at is the following: deep learning also seems to learn corresponding deep features that cannot be extracted from more "shallow" networks. that is demonstrated in improved performance over the shallow networks. – vzn Feb 7 '14 at 15:56

Indeed, one would like to construct features that make classes linearly separable. However, what patterns exactly should be linearly separable?

It is easy to make patterns in training set linearly separable by greatly increasing feature space dimensionality. However, this will not necessarily imply good generalization (correct classification of new patterns). Quite opposite, it can lead to overfitting. Thus, it is problematic to use the criterion of linear separability directly to select good nonlinear features for classification.

Deep learning networks adopt different approach. They try to select features with good generalization of patterns irrelative to the classification task. Thus, “useful” nonlinear features are selected without danger of overfitting. Indeed, it doesn’t necessarily ensure linear separability of classes, but it makes sense from induction point of view. If one extract statistically independent factors from initial features, in which these factors are mixed, it will be easier to solve any further classification task.

• I like your viewpoint about getting statistically independent factors. I'm curious if you have any intuition as to how the threshold gates in neural networks try to select features with good generalization, or statistically independent features, or both. That is, why is it that learning nested linear-threshold gates is an a priori good idea to achieve these goals? – Joshua Grochow Feb 10 '14 at 19:58
• Well, I don't think that this idea is really good. Indeed, only linear features are learned on each level. Then, some fixed nonlinearity is added, and linear features are learned again. It cannot be guaranteed that this fixed nonlinearity will be appropriate to derive independent hidden factors. It is clear from the theory of universal induction based on algorithmic probability. Simpler motivation, why deep learning networks are not so good, is mentioned here videolectures.net/nips2010_tenenbaum_hgm – Necro0x0Der Feb 11 '14 at 9:00

Since each node in a neural network is a weighted linear threshold function, you can in fact view neural networks as mapping into a feature space where it then applies a linear separator, as follows. Consider the functions $g_1,\dotsc,g_k$ computed by the nodes just before the final output node. Here I'm assuming it's a binary classification task, so there's just one output node, but the general case is similar. Then the function $(x_1,\dotsc,x_n) \to (g_1(\vec{x}), \dotsc, g_k(\vec{x}))$ is a mapping into a feature space where the data is then linearly separable (by the final threshold gate).

Why is this useful, compared to the ways kernels are used in SVMs? Well, first of all, I think the features computed by neural networks are more general than the kernels people often use in practice, which come from a limited set of forms.

Second and more importantly, my guess would be that the reason it's useful is that it gives a principled, automatic way to generate the features. In the case of SVMs (if I understand correctly), you more or less have to guess a kernel based on your intuition, prior experience, trial and error, and luck. In the case of neural networks viewed as above, training the neural network builds the features for you.

As you said, the other nodes in the deep neural network - those that come before the penultimate layer of nodes $g_1,\dotsc,g_k$ - are not learning features that make the data linearly separable. I think their primary utility is in giving an automated way to represent and build the features $g_i$ that will make the data linearly separable.