# equivalence between Bayesian prior distribution and regularization metric?

Ridge and LASSO can be interpreted as OLS with priors over the coefficients (respectively, Gaussian and Laplacian). How much does this generalize? Given a prior, does it imply a regularization term corresponding to a norm? Given a symmetric prior, is there such a regularization term?

I believe that the answer to the first question is `no'. To see why, consider a linear model with Gaussian likelihood function $$p(y | X, \beta) = \frac{1}{\sigma \sqrt{2 \pi}} \exp \left( - \frac{1}{2 \sigma^2} (y - \beta X)^2 \right)$$ and lognormal prior $$p (\beta) = \frac{1}{\beta \tau \sqrt{2 \pi}} \exp \left( - \frac{1}{2 \tau^2} (\ln \beta)^2 \right).$$ Its posterior is then $$p(\beta | y, X) \propto \exp \left( - \frac{1}{2 \sigma^2} (y - \beta X)^2 - \frac{1}{2 \tau^2} (\ln \beta)^2 - \ln \beta \right),$$ with regularization term $$\frac{\sigma^2}{\tau^2} (\ln \beta)^2 + 2 \sigma^2 \ln \beta$$.

This does not correspond to a norm as it does not satisfy non-negativity for $$\beta \in \left( \exp(-2 \tau^2) , 1 \right)$$, a non-degenerate interval for all $$\tau > 0$$.

The generalization is to add a regularization term of the form $$-\log p(\theta)$$ where $$p(\theta)$$ is the probability of coefficients $$\theta$$ according to your prior. Note how Ridge and LASSO can be viewed as special cases of this (try calculating what $$\log p(\theta)$$ is for the Gaussian and Laplacian distributions).
• @ColinRowat, my answer works whether the prior is symmetric or not, so I believe it answers both questions. If that doesn't sound right to you, I suggest asking a new questions and providing more context to explain what you are asking. (Incidentally, I don't know why you want a distance metric or what it would even mean here; $\theta$ is a single variable, and distance is only meaningful if you have two points, not just one. There is no requirement that regularization terms be associated with a distance metric.) You might be interested in stats.stackexchange.com