Yoshua Benhgio's Learning Deep Architectures for AI book mentions that

we should [...] strive to develop learning algorithms that use the data to determine the depth of the final architecture.

Would anyone know of any algorithms proposed thus far to achieve this?

So far I have come across:

The tiling algorithm for building a feed-forward network to learn a Boolean function. It adds layers as well as units, but Boolean functions aren't too relevant for applied problems.

  • $\begingroup$ an emerging idea is to somehow measure how the response of how "random" neurons are in response to inputs, and one builds new neural connections on top of neurons that are "nonrandom" ie "orderly" and that find nonrandom patterns... $\endgroup$ – vzn Feb 6 '14 at 16:10
  • 1
    $\begingroup$ @vzn what do you mean by "measure how the response of how "random" neurons are in response to inputs", and what do you mean by "neural connections on top of neurons"? would you have a paper to link us to? sounds very interesting! $\endgroup$ – Alexandre Holden Daly Feb 7 '14 at 0:34
  • $\begingroup$ the basic idea is that neurons act as feature detectors. neurons succeed or fail as feature detectors. if the output of a neuron is random, the neuron has failed to find a feature. features are nonrandom/extracted order. new features are hierarchically built out of other features. havent seen this in the literature so far. the closest concept comes from neural darwinism originated by edelman but apparently not yet incorporated into ("artificial"/algorithmic) connectionist models. may look for more papers on the subj wrt application in ML. $\endgroup$ – vzn Feb 7 '14 at 0:37

This predates the start of the "deep learning craze" in the mid-2000s, but for me the cascade correlation algorithm (Fahlman & Lebiere, 1989; pdf) is the build-your-own-topology NN algorithm. I am not sure how popular the algorithm is in ML now-a-days, but it is still popular in cognitive science despite it's unrealistic grounding in biology. If you want intersections with genetic algorithms (like the one @vzn points out with more recent examples) then that has been done with CC as well (Potter, 1992; pdf).

You might also find the following questions on other SEs to be of interest:


this is generally an open question under active research; however there is quite a bit of research using GAs to determine NN architectures see eg

however caveat this type of approach is very expensive computationally and possibly hasnt yet been used with deep learning.


here are two other more recent papers that touch on this. as AK cites, cascade correlation algorithms were designed around ~1990 for this purpose of dynamic ANN growth but they seem not to be applied in deep learning experiments so far. the 2nd paper below cites cascade correlation as an inspiration for an incremental/online algorithm that adds feature detectors over time built out of lower-level feature detectors.

  • Learning Android Control using Growing Neural Networks 2006 Amor et al

    Fixed sized neural networks are a popular technique for learning the adaptive control of non-linear plants. When applied to the complex control of android robots, however, they suffer from serious limitations, such as the moving target problem i.e. the interference between old and newly learned knowledge. To overcome these problems, we propose the use of growing neural networks in a new learning framework based on the process of consolidation.

  • this paper comes from Deep learning and unsupervised feature learning NIPS workshop 2012. Online Representation Search and Its Interactions with Unsupervised Learning by Mahmood/Sutton. see fig 4 p4. the network performance vs incremental growth in feature detectors is graphed.

    We consider the problem of finding good hidden units, or features, for use in multilayer neural networks. Solution methods that generate candidate features, evaluate them, and retain the most useful ones (such as cascade correlation and NEAT), we call representation search methods. In this paper, we explore novel representation search methods in an online setting, compare them with two simple unsupervised learning algorithms that also scale online.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.