Which algorithms have been proposed to learn the architecture of a deep neural network?

Question

Yoshua Benhgio's Learning Deep Architectures for AI book mentions that

we should [...] strive to develop learning algorithms that use the data to determine the depth of the final architecture.

Would anyone know of any algorithms proposed thus far to achieve this?

---

So far I have come across:

The tiling algorithm for building a feed-forward network to learn a Boolean function. It adds layers as well as units, but Boolean functions aren't too relevant for applied problems.

Answer 1

This predates the start of the "deep learning craze" in the mid-2000s, but for me the cascade correlation algorithm (Fahlman & Lebiere, 1989; pdf) is the build-your-own-topology NN algorithm. I am not sure how popular the algorithm is in ML now-a-days, but it is still popular in cognitive science despite it's unrealistic grounding in biology. If you want intersections with genetic algorithms (like the one @vzn points out with more recent examples) then that has been done with CC as well (Potter, 1992; pdf).

You might also find the following questions on other SEs to be of interest:

• Why do neural networks seem to perform better with restrictions placed on their topology?
• Dynamically adjusting NN architecture: inventing the unnecessary? (also on MetaOptimize)

Answer 2

this is generally an open question under active research; however there is quite a bit of research using GAs to determine NN architectures see eg

• Finding Optimal Neural Network Architecture Using Genetic Algorithms Britos et al

• Using genetic algorithms to select architecture of a feedforward artificial neural network Arifovic/Gencay

however caveat this type of approach is very expensive computationally and possibly hasnt yet been used with deep learning.

Answer 3

here are two other more recent papers that touch on this. as AK cites, cascade correlation algorithms were designed around ~1990 for this purpose of dynamic ANN growth but they seem not to be applied in deep learning experiments so far. the 2nd paper below cites cascade correlation as an inspiration for an incremental/online algorithm that adds feature detectors over time built out of lower-level feature detectors.

• Learning Android Control using Growing Neural Networks 2006 Amor et al

  Fixed sized neural networks are a popular technique for learning the adaptive control of non-linear plants. When applied to the complex control of android robots, however, they suffer from serious limitations, such as the moving target problem i.e. the interference between old and newly learned knowledge. To overcome these problems, we propose the use of growing neural networks in a new learning framework based on the process of consolidation.

• this paper comes from Deep learning and unsupervised feature learning NIPS workshop 2012. Online Representation Search and Its Interactions with Unsupervised Learning by Mahmood/Sutton. see fig 4 p4. the network performance vs incremental growth in feature detectors is graphed.

  We consider the problem of finding good hidden units, or features, for use in multilayer neural networks. Solution methods that generate candidate features, evaluate them, and retain the most useful ones (such as cascade correlation and NEAT), we call representation search methods. In this paper, we explore novel representation search methods in an online setting, compare them with two simple unsupervised learning algorithms that also scale online.