I'm new in this community and I don't know whether my question is proper for this community. I will delete this post if it is not proper.

I'm interested in deep learning network models and have a question about this.

Suppose we have a feedforward neural network models with different hyperparameters (for example, the number of layers, neurons and so on), say $H_1,..., H_n$. Also assume all the samples are selected from a particular distribution.

First we got $100$ samples. If we find that a typical hyperparameter set, $H_i$, performs best. Now I want to increase the number of samples to $1000, 10000,$ even millions. Is $H_i$ still the best hyperparameter set to perform among $H_i$'s? I want to know whether there are references about such topics. The best is a mathematical proof about this fact.

Again, please tell me if this question is not proper to the community. Thanks a lot!


No. You're asking about model selection. Larger sample sizes will allow you to choose more complex models. Read up on the keywords overfit/underfit, model selection, Structural Risk Minimization, and in particular this article: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=705570

  • $\begingroup$ My question is for fixed parameters set, do not consider about more complex models. Anyway, I will read this paper. Thank you very much $\endgroup$
    – CSH
    Feb 26 '19 at 11:33
  • $\begingroup$ If you use "number of layers" as a hyperparameter (as in your OP), this is NOT a fixed-parameter model, since more layers => more parameters. $\endgroup$
    – Aryeh
    Feb 26 '19 at 12:09
  • $\begingroup$ Sorry I didn't mention in details. Since we have hyperparameter sets $H_1$ to $H_n$, numbers of layers, for example, can be chosen only from those $n$ choices, but I found it is not a common way because as you said, we can construct more complex models to enhance the results. $\endgroup$
    – CSH
    Feb 26 '19 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.