6
$\begingroup$

Unless I'm mistaken, deep neural networks are good for learning functions that are nonlinear in the input.

In such cases, the input set is linearly inseparable, so the optimisation problem that results from the approximation problem is not convex, so it cannot be globally optimised with local optimization. Support Vector Machines (try to) get around this by choosing features such that the projection of the input space into the feature space is linearly separable, and we have a convex optimisation problem once again.

I don't suppose that a deep neural network always learns features that make the input set linearly separable in feature space. So what's the point in it learning features?

$\endgroup$
  • 1
    $\begingroup$ can you rephrase that last question "whats the point in it learning features?" it seems not a question or to not be what you are intending to ask. plausibly the whole point/raison d'etre of ML is to learn features. also, ANNs have been learning nonlinear functions ever since early days (1980s), that in particular is not a twist of deep learning. maybe what this question is getting at is the following: deep learning also seems to learn corresponding deep features that cannot be extracted from more "shallow" networks. that is demonstrated in improved performance over the shallow networks. $\endgroup$ – vzn Feb 7 '14 at 15:56
1
$\begingroup$

Indeed, one would like to construct features that make classes linearly separable. However, what patterns exactly should be linearly separable?

It is easy to make patterns in training set linearly separable by greatly increasing feature space dimensionality. However, this will not necessarily imply good generalization (correct classification of new patterns). Quite opposite, it can lead to overfitting. Thus, it is problematic to use the criterion of linear separability directly to select good nonlinear features for classification.

Deep learning networks adopt different approach. They try to select features with good generalization of patterns irrelative to the classification task. Thus, “useful” nonlinear features are selected without danger of overfitting. Indeed, it doesn’t necessarily ensure linear separability of classes, but it makes sense from induction point of view. If one extract statistically independent factors from initial features, in which these factors are mixed, it will be easier to solve any further classification task.

$\endgroup$
  • $\begingroup$ I like your viewpoint about getting statistically independent factors. I'm curious if you have any intuition as to how the threshold gates in neural networks try to select features with good generalization, or statistically independent features, or both. That is, why is it that learning nested linear-threshold gates is an a priori good idea to achieve these goals? $\endgroup$ – Joshua Grochow Feb 10 '14 at 19:58
  • $\begingroup$ Well, I don't think that this idea is really good. Indeed, only linear features are learned on each level. Then, some fixed nonlinearity is added, and linear features are learned again. It cannot be guaranteed that this fixed nonlinearity will be appropriate to derive independent hidden factors. It is clear from the theory of universal induction based on algorithmic probability. Simpler motivation, why deep learning networks are not so good, is mentioned here videolectures.net/nips2010_tenenbaum_hgm $\endgroup$ – Necro0x0Der Feb 11 '14 at 9:00
1
$\begingroup$

Since each node in a neural network is a weighted linear threshold function, you can in fact view neural networks as mapping into a feature space where it then applies a linear separator, as follows. Consider the functions $g_1,\dotsc,g_k$ computed by the nodes just before the final output node. Here I'm assuming it's a binary classification task, so there's just one output node, but the general case is similar. Then the function $(x_1,\dotsc,x_n) \to (g_1(\vec{x}), \dotsc, g_k(\vec{x}))$ is a mapping into a feature space where the data is then linearly separable (by the final threshold gate).

Why is this useful, compared to the ways kernels are used in SVMs? Well, first of all, I think the features computed by neural networks are more general than the kernels people often use in practice, which come from a limited set of forms.

Second and more importantly, my guess would be that the reason it's useful is that it gives a principled, automatic way to generate the features. In the case of SVMs (if I understand correctly), you more or less have to guess a kernel based on your intuition, prior experience, trial and error, and luck. In the case of neural networks viewed as above, training the neural network builds the features for you.

As you said, the other nodes in the deep neural network - those that come before the penultimate layer of nodes $g_1,\dotsc,g_k$ - are not learning features that make the data linearly separable. I think their primary utility is in giving an automated way to represent and build the features $g_i$ that will make the data linearly separable.

$\endgroup$
0
$\begingroup$

I thought about this question once, and I convinced myself with an argument similar to this:

Unlike the SVM (kernel), the features are not projected to a different space, instead they are weighted! The weight of the features make them acts like like they would under "gravity" and balances the "space", that way the features that have more weights associated to them actually has more "dominance" over the output (of the activation function).

I like to think of it this way- consider a non linear data set with just 2 attributes, now, after the training phase, each attribute has some weight assigned to them. these weights inherently change their "role in the activation function" thus might actually make one of them to shrink in in size relative to the other, now as this happens if you then again look at the points, you might see that the new data does not look as non linear but instead you can actually make a lot of sense of it..

This was a very high level "reasoning" with a lot of "handwaving". But intuitively on a high level this is what I think is happening under the hood.

$\endgroup$
  • 2
    $\begingroup$ I am not an expert in the area, but I don't think weights alone explain the real meat (so to speak) of deep neural networks. Changing weights still gives you a linear separator. I think the key has to be in the fact that by combining the outputs of (linear) threshold gates into another level of gates, you can get nonlinear separators, because a threshold function isn't linear as a function (even though it does correspond to a linear separator). $\endgroup$ – Joshua Grochow Feb 8 '14 at 17:13
  • 2
    $\begingroup$ @JoshuaGrochow but this doesn't explain how it's different from an SVM, which also gives a nonlinear separator $\endgroup$ – Suresh Venkat Feb 9 '14 at 7:10
  • $\begingroup$ @SureshVenkat: Very true. $\endgroup$ – Joshua Grochow Feb 9 '14 at 15:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.