I might not be in the good stack-exchange site, so please forgive me.
Let consider a neural network that we want to train to a certain task. For the purpose of my research, I am wiling to reduce the number of operations during the training process within my encrypted neural network (homomorphically encrypted). However, I'm definitely not an expert in this domain and I would need some insights.
The weights inside my neural network are all encrypted in a way that allows normal computations (and hence proper training phase). However, encrypted operations are much more costly than operations on clear data.
Practically, if we want to reach a certain value for the error function (let's say $E < 0.01$) for instance, does there exist a tradeoff between number of operations and depth of the neural network.
As far as I know, people tend to find the accurate size of a neural network by trials and intuition but maybe some people made some theoretical results on this kind of tradeoff (even for only one specific kind of NN) ? If you have any paper to recommend I would gladly take them!