Converting a Bernoulli to a Gaussian

It is not hard to see that, given one sample from a univariate unit-variance Gaussian $$X\sim \mathcal{N}(\mu,1)$$ with unknown $$\mu$$ s.t. $$0<|\mu|\leq 1$$, one can simulate one draw from a "Bernoulli" random variable $$Y\in\{-1,1\}$$ with $$|\mathbb{E}[Y]| = \Theta(|\mu|)$$. (Basically, just take $$Y=\operatorname{sign}(X)$$.)

Is it possible to do the converse (even allowing extra randomness, independent of the sample)? That is:

Given one realization of some random variable $$Y\in\{-1,1\}$$ with (unknown) $$\mathbb{E}[Y] = \mu$$, output $$X\sim \mathcal{N}(\nu,1)$$ s.t. $$|\nu| = \Theta(|\mu|)$$.

I conjecture that to be impossible (I have some mild contrived evidence), but I don't really see how one would go about proving that.

Suppose you had such a randomized procedure that takes a value in $$\{-1,1\}$$ and outputs a real number. Let $$P$$ and $$Q$$ be the output distribution on input $$+1$$ and $$-1$$ respectively.
Consider the extreme case of $$\mu = +1$$. In this case $$Y = +1$$ for sure, and you are outputting a sample from $$P$$, which means that $$P$$ should be an $$\mathcal{N}(\nu, 1)$$ distribution for some $$\nu$$ with $$|\nu|$$ being $$\Omega(1)$$. Similarly, the other extreme of $$\mu=-1$$ tells us that $$Q$$ is a $$\mathcal{N}(\nu', 1)$$ distribution for some $$\nu'$$ with $$|\nu'|$$ being $$\Omega(1)$$.
But now, for $$\mu=0$$, you are outputting the uniform mixture of two normals, which cannot be a normal distribution unless $$\nu=\nu'$$. In the latter case, you get $$\mathcal{N}(\nu, 1)$$ for some $$\nu$$ bounded away from zero.
Thus you cannot satisfy this requirement simultaneously for $$\mu\in \{-1, 0, 1\}$$.