# Does fixed hyperparameters perform well regardless the number of training examples?

I'm new in this community and I don't know whether my question is proper for this community. I will delete this post if it is not proper.

Suppose we have a feedforward neural network models with different hyperparameters (for example, the number of layers, neurons and so on), say $$H_1,..., H_n$$. Also assume all the samples are selected from a particular distribution.

First we got $$100$$ samples. If we find that a typical hyperparameter set, $$H_i$$, performs best. Now I want to increase the number of samples to $$1000, 10000,$$ even millions. Is $$H_i$$ still the best hyperparameter set to perform among $$H_i$$'s? I want to know whether there are references about such topics. The best is a mathematical proof about this fact.

Again, please tell me if this question is not proper to the community. Thanks a lot!

No. You're asking about model selection. Larger sample sizes will allow you to choose more complex models. Read up on the keywords overfit/underfit, model selection, Structural Risk Minimization, and in particular this article: https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=705570

• My question is for fixed parameters set, do not consider about more complex models. Anyway, I will read this paper. Thank you very much
– CSH
Feb 26 '19 at 11:33
• If you use "number of layers" as a hyperparameter (as in your OP), this is NOT a fixed-parameter model, since more layers => more parameters. Feb 26 '19 at 12:09
• Sorry I didn't mention in details. Since we have hyperparameter sets $H_1$ to $H_n$, numbers of layers, for example, can be chosen only from those $n$ choices, but I found it is not a common way because as you said, we can construct more complex models to enhance the results.
– CSH
Feb 26 '19 at 12:17