2
$\begingroup$

Let $\rho(\sigma): \mathbb{R} \rightarrow \mathbb{R}$ be a probability density that is parametrized by a parameter vector $\sigma \in \mathbb{R}^s$ (for example the normal distribution where $s = 1$ and $\sigma$ is the variance), $x \in \mathbb{R}^n$ a vector with $x_{i+1} - x_i = c\;\; \forall \, 1 \leq i < n$ for a constant $c \in \mathbb{R}$ and $v \in \mathbb{R}^n$ a vector with

$$ v_i \equiv \int_{x_i-c/2}^{x_i+c/2}dx\;\; \rho(x | \sigma) $$

$f(\sigma, \theta): \mathbb{R}^2 \rightarrow \mathbb{R}$ is a function which is parametrized by the same $\sigma$ and an additional parameter vector $\theta \in \mathbb{R}^t$. $f$ is not known analytically but can be computed numerically.
$w \in \mathbb{R}^n$ is another vector with

$$ w_i \equiv \int_{x_1-c/2}^{x_n+c/2}dx \int_{x_i-c/2}^{x_i+c/2}dy\;\;f(x, y | \sigma, \theta) \cdot \rho(x | \sigma) $$

I want to train a neural network (specifically a multilayer perceptron) to compute $v$ (or equally $\sigma$) from a given $\{w, \theta\}$ ($x$ is fixed and known). For this I create training data $\{v, w\}$ by sampling the complete parameter space $\mathbb{R}^s \times \mathbb{R}^t$. The integrals are computed by Monte Carlo sampling. And this is where my question sets in:

I train the network to go from $\{w, \theta\} \rightarrow v$ and $v$ only depends on $\sigma$ so for traversing directions of the parameter space for which $\sigma$ is constant I feed the very same $v$ to the neural network. Especially for cases $t > s$ (but also for others) the network will see many different inputs that all map to the very same output. This holds if I use the same seed for the random number generator for every case. Instead I could also use a different seed each time which will lead to slightly different $v$ (and $w$) while their overall features are preserved. Those statistical fluctuations should behave like white noise (for a RNG with long autocorrelations). Doing so the neural network would see a different output each time while still learning the correct features.

My question is, what effect does the addition of white noise to the output channels have on the training of the neural network (MLP)? Does it improve the learning because it gets to see a greater variety with respect to the training data? Or does it make it worse because it suppresses the feature that the output actually should be identical?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.