Theoretical Computer Science Stack Exchange is a question and answer site for theoretical computer scientists and researchers in related fields. It only takes a minute to sign up.
Sign up to join this community
I'm reading the paper (http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf) from Glorot and Bengio. There is something that I don't understand at the abstract section on page 1.
"Training may be more difficult when the singular values of the Jacobian associated with each layer are far from 1"
Why is the singular value of the Jacobian important for training?
Why is the singular value being far from 1 difficult for training?
If the singular values of each layer are smaller than 1, then you can have vanishing gradients. If the singular values are larger than 1, you can have exploding gradients. Both cause problems for training.
Why? As a heuristic, imagine that all the singular values for each layer were 2, and you have $n$ layers. Then the singular values for the entire network would be $2^n$. So, if the singular values of the layers are larger than 1, the singular values for the entire network can grow exponentially in the number of layers. Similarly if they're less than 1, they can vanish exponentially fast.
