0
$\begingroup$

I'm reading the paper (http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf) from Glorot and Bengio. There is something that I don't understand at the abstract section on page 1.

"Training may be more difficult when the singular values of the Jacobian associated with each layer are far from 1"

Why is the singular value of the Jacobian important for training?

Why is the singular value being far from 1 difficult for training?

$\endgroup$

1 Answer 1

1
$\begingroup$

If the singular values of each layer are smaller than 1, then you can have vanishing gradients. If the singular values are larger than 1, you can have exploding gradients. Both cause problems for training.

Why? As a heuristic, imagine that all the singular values for each layer were 2, and you have $n$ layers. Then the singular values for the entire network would be $2^n$. So, if the singular values of the layers are larger than 1, the singular values for the entire network can grow exponentially in the number of layers. Similarly if they're less than 1, they can vanish exponentially fast.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.