27

Cybenko's result is fairly intuitive, as I hope to convey below; what makes things more tricky is he was aiming both for generality, as well as a minimal number of hidden layers. Kolmogorov's result (mentioned by vzn) in fact achieves a stronger guarantee, but is somewhat less relevant to machine learning (in particular, it does not build a standard neural ...


10

There is no way to choose the parameters A, B, C, D properly; as it is the case for most heuristics, the parameters are chosen ''by experience''. Worse, there is no guarantee that the solution (the array of the outputs) of this heuristic is indeed a feasible TSP tour! In this context, perhaps interesting to read: Wilson and Pawley, On the stability of ...


7

Most of the patterns in my collection at http://www.ics.uci.edu/~eppstein/ca/replicators/ were found with a somewhat different heuristic search process, that was intended to find puffers but also turns out to work for replicators: Pick a small random seed pattern Simulate a moderate number of steps of a modified version of the cellular automaton rule, ...


4

No, you don't need a bias. You can have a "dummy" input (input(n+1) in your formualtion) which is always set to 1. Then the bias term is absorbed into the weights.


3

According to this blog By Reza Zadeh, training a neural network to produce correct output even for just two-thirds of the training examples is computationally hard: Indeed, in 1988 J. Stephen Judd shows the following problem to be NP-hard: Given a general neural network and a set of training examples, does there exist a set of edge weights for the network ...


3

There is an advanced result, key to machine learning, known as Kolmogorov's theorem [1]; I have never seen an intuitive sketch of why it works. This may have to do with the different cultures that approach it. The applied learning crowd regards Kolmogorov's theorem as an existence theorem that merely indicates that NNs may exist, so at least the structure ...


3

This predates the start of the "deep learning craze" in the mid-2000s, but for me the cascade correlation algorithm (Fahlman & Lebiere, 1989; pdf) is the build-your-own-topology NN algorithm. I am not sure how popular the algorithm is in ML now-a-days, but it is still popular in cognitive science despite it's unrealistic grounding in biology. If you want ...


2

Sigmoid functions have clear probabilistic interpretation, so one can derive optimal learning algorithms for them within the bayesian approach. Moreover, they are differentiable, so the gradient descent can be directly applicable. However, in general, neural networks are "universal approximators" in almost the same sense as, for example, polynomials. They ...


2

Neural networks can arbitrarily approximate any computable function; see http://en.wikipedia.org/wiki/Artificial_neural_network or http://en.wikipedia.org/wiki/Cybenko_theorem Constructing a NN for a given function -- or worse yet, learning it from examples -- is a hard problem.


2

This 2013 paper integrates deep learning models with structured hierarchical Bayesian models, for the net to learn novel concepts from very few training examples. It shows encouraging results. Another approach is to import a general-purpose top-performing neural net, and replace the top fully connected layers with initialised ones (including an output layer ...


2

It seems the answer must be positive, because what might a reasonable negative answer to "Can neural networks be used to devise algorithms?" look like? If you accept the premise that human brains are neural networks -- or at least can be modeled by such to a very high degree of accuracy -- doesn't that provide an existence proof? We can devise ...


2

Short answer: Not necessarily. Likely nothing fishy is going on. Longer answer: The Universal Approximation Theorem (UAT) says nothing about an individual network's capacity to approximate a function. Moreover it says nothing about trainability or generalization. The UAT roughly says that for any $\epsilon>0$ and any continuous real-valued function $F$ ...


1

In case you're still looking for more information on this, I'll chip in my two cents. It may help to think of neural networks as just fitting some function based on the training data. Each hidden layer increases the ability of the neural network to fit more complex functions. A network without hidden layers, often called a perceptron, is only able to fit ...


1

Since each node in a neural network is a weighted linear threshold function, you can in fact view neural networks as mapping into a feature space where it then applies a linear separator, as follows. Consider the functions $g_1,\dotsc,g_k$ computed by the nodes just before the final output node. Here I'm assuming it's a binary classification task, so there's ...


1

Indeed, one would like to construct features that make classes linearly separable. However, what patterns exactly should be linearly separable? It is easy to make patterns in training set linearly separable by greatly increasing feature space dimensionality. However, this will not necessarily imply good generalization (correct classification of new patterns)...


1

The annual GECCO conference (pretty much the premier venue for evolutionary computation research) has a `Real World Applications' track. See also this recent presentation:


Only top voted, non community-wiki answers of a minimum length are eligible