# Does fixed hyperparameters perform well regardless the number of training examples?

I'm new in this community and I don't know whether my question is proper for this community. I will delete this post if it is not proper.

Suppose we have a feedforward neural network models with different hyperparameters (for example, the number of layers, neurons and so on), say $$H_1,..., H_n$$. Also assume all the samples are selected from a particular distribution.

First we got $$100$$ samples. If we find that a typical hyperparameter set, $$H_i$$, performs best. Now I want to increase the number of samples to $$1000, 10000,$$ even millions. Is $$H_i$$ still the best hyperparameter set to perform among $$H_i$$'s? I want to know whether there are references about such topics. The best is a mathematical proof about this fact.

Again, please tell me if this question is not proper to the community. Thanks a lot!

• Sorry I didn't mention in details. Since we have hyperparameter sets $H_1$ to $H_n$, numbers of layers, for example, can be chosen only from those $n$ choices, but I found it is not a common way because as you said, we can construct more complex models to enhance the results. – CSH Feb 26 at 12:17