I was trying to understand if there was a way to express the back-propagation equations from neural networks in a better way as to understand them better. I believe the equations can be written in a recursive manner as follows (using the notation from these notes from Andrew Ng):

$$ \delta^{(n_l)}_{i} = \left( \sum^{s_{l+1}}_{j=1} W^{(l)}_{ji} \delta^{(l+1)}_{j} \right) f'(z^{(l)}_{i})$$

I know that some recursions/recurrences have closed form "equations"/solutions. A typical example would be the Fibonacci sequence (or things that use Akra-Bazzi to solve them). I was wondering, because of the similarity to recursion of this equation, does something like that exist for back-propagation equations in Neural Networks? If there is not, does that mean its impossible to get such an equations? I was wondering if there were formal mathematical proofs arguments that might explain the impossibility of such an attempt or any work that has been tried to that goal.

In reality, it seems that the main use of this equation is algorithmically. But I was hoping that having some sort of equation (or even approximation to this recurrence) could help understand back-propagation and neural networks better. Might this require new mathematics to achieve something like this? Or maybe trying to do something like this makes no sense? If so why so?

  • $\begingroup$ I think you will have a better luck asking this on Cross Validated. You can flag the question for moderator attention if you want the question to be migrated there. $\endgroup$ – Kaveh Aug 14 '15 at 0:30
  • $\begingroup$ Well, in the case that f is the identity (or just linear), and assuming that each weight matrix is the same $W$, then we have $\delta^{(n)} = (W^{m - n})^T$ where $T$ denotes transpose, and $m$ denotes the layer number of the output layer. If $W$ is diagonalizable with $W = R^T D R$ with diagonal matrix $D$ and orthogonal matrices $R$, then you have a closed form given by $\delta^{(n)} = R^T D^{m-n} R$. Hopefully this helps with your intuition a bit, but note that in practice it's much much more efficient to just to the multiplications, since finding eigenvalues is in generally much harder. $\endgroup$ – SorcererofDM Aug 14 '15 at 2:08
  • $\begingroup$ I think nobody's trying to solve for a "close form" here because this is about as simple as it gets, and any abstraction on top is entirely unnecessary from a practical point of view. With the field moving as it is, there are plenty of unformalized mathematics. $\endgroup$ – SorcererofDM Aug 14 '15 at 2:12
  • $\begingroup$ The other tricky part is that solving non-linear equations is really hard. This is a big reason why neural networks are so general, and why training them requires so many herustics - it's very difficult to prove things about such complex systems, without solving many open problems in mathematics first. There are proofs you can make about regularization and probability distributions and such, but to actually "solve" neural networks and immediately jump to the optimal form is most likely impossible. $\endgroup$ – Phylliida Aug 14 '15 at 3:57
  • $\begingroup$ @DanielleEnsign do you know of any specific open problems that one would have to be solve before solving this one? $\endgroup$ – Charlie Parker Aug 15 '15 at 20:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.