0
$\begingroup$

A well known result (Cybenko 1989) holds that a single-layer feedforward "multilayer perceptron" style network:

$f(x) = \sum_{i=1}^{K}\alpha_{i}\sigma(w_{i}^{T}x + \theta_{i})$

Can approximate any continuous function on $R^{N}$ to an arbitrary degree of precision. Does this result still hold if one or more weight in each $w_{i}$ in the expression above is constrained to zero? Assume that at least one weight in each hidden unit is always non-zero. I.e. is a single layer partially connected network still able to well approximate any continuous function provided there exist enough hidden units?

This seems like a question that should have been answered already but I've been having a hard time finding it. If anyone can answer or point me in the direction of some related work, I would appreciate it.

$\endgroup$
  • $\begingroup$ I don't understand what you mean with one weight in each hidden unit is non-zero. Take $\phi$ a smooth approximation of $1_{x \in [-1,1]^N}$ then it suffices to check that $f$ is able to approximate $\phi$ on $[-10,10]^N$. Also do you know how to use convolution followed by sampling to approximate locally a smooth or continuous function ? If the convolution kernel is analytic then the approximation is analytic so you can take the first few Taylor coefficients to obtain a polynomial approximation. $\endgroup$ – user1952009 Nov 1 '17 at 23:49
  • $\begingroup$ I mean the following: In the formulation of an MLP used by Cybenko and others in proofs of the UAP, there are no constraints on the weights inside the sigmoids. I want to know what implications there are, if any, to requiring one or more weight inside each sigmoid be zero. Here's a two feature, two unit example: $\sigma(w_{1}^{1}x_{1} + w_{2}^{1}x_{2}) + \sigma(w_{1}^{2}x_{1} + w_{1}^{2}x_{2})$ versus $\sigma(w_{1}^{1}x_{1} + 0*x_{2}) + \sigma(0*x_{1} + w_{2}^{2}x_{2})$ $\endgroup$ – aht Nov 2 '17 at 17:33
  • $\begingroup$ Did you try approximating $1_{x \in [-1,1]^N}$ ? $\endgroup$ – user1952009 Nov 3 '17 at 3:55
  • $\begingroup$ I'm confused by what you are asking. If it's a one-layer network, such as the equation you showed, there are no hidden layers, so I'm confused by your mention of a "hidden unit". Also, what attempts at solving this have you made yourself? This site works best for questions where you have been thinking hard about the problem on your own first and can show us what attempts you've made. See our help center. $\endgroup$ – D.W. Nov 4 '17 at 0:49
  • $\begingroup$ Thanks for the feedback @D.W - I just noticed I omitted the activation function in the equation in the body of the question which may have been the source of some confusion. As far as my own attempts to answer the question, I've just gone through Cybenko's proof and tried to see what part would be violated by a constraint on the weights. Nothing jumped out immediately, but my analysis skills aren't very strong so I'm not confident in this. As far as the suggestion above to try approximating the indicator function - I haven't had time to think about this yet. $\endgroup$ – aht Nov 4 '17 at 2:15

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.