-1
$\begingroup$

Ridge and LASSO can be interpreted as OLS with priors over the coefficients (respectively, Gaussian and Laplacian). How much does this generalize? Given a prior, does it imply a regularization term corresponding to a norm? Given a symmetric prior, is there such a regularization term?

I believe that the answer to the first question is `no'. To see why, consider a linear model with Gaussian likelihood function $ p(y | X, \beta) = \frac{1}{\sigma \sqrt{2 \pi}} \exp \left( - \frac{1}{2 \sigma^2} (y - \beta X)^2 \right) $ and lognormal prior $ p (\beta) = \frac{1}{\beta \tau \sqrt{2 \pi}} \exp \left( - \frac{1}{2 \tau^2} (\ln \beta)^2 \right). $ Its posterior is then $ p(\beta | y, X) \propto \exp \left( - \frac{1}{2 \sigma^2} (y - \beta X)^2 - \frac{1}{2 \tau^2} (\ln \beta)^2 - \ln \beta \right), $ with regularization term $\frac{\sigma^2}{\tau^2} (\ln \beta)^2 + 2 \sigma^2 \ln \beta$.

This does not correspond to a norm as it does not satisfy non-negativity for $\beta \in \left( \exp(-2 \tau^2) , 1 \right)$, a non-degenerate interval for all $\tau > 0$.

$\endgroup$

1 Answer 1

1
$\begingroup$

The generalization is to add a regularization term of the form $-\log p(\theta)$ where $p(\theta)$ is the probability of coefficients $\theta$ according to your prior. Note how Ridge and LASSO can be viewed as special cases of this (try calculating what $\log p(\theta)$ is for the Gaussian and Laplacian distributions).

$\endgroup$
5
  • $\begingroup$ Thank you @D.W. Thus, applied to my original questions: (1) given a prior, there is a corresponding regularized regression; the regularization term may not be a metric. (2) remains an open question for me, but a symmetric prior seems a necessary condition? $\endgroup$ Commented Dec 23, 2021 at 0:39
  • 1
    $\begingroup$ @ColinRowat, my answer works whether the prior is symmetric or not, so I believe it answers both questions. If that doesn't sound right to you, I suggest asking a new questions and providing more context to explain what you are asking. (Incidentally, I don't know why you want a distance metric or what it would even mean here; $\theta$ is a single variable, and distance is only meaningful if you have two points, not just one. There is no requirement that regularization terms be associated with a distance metric.) You might be interested in stats.stackexchange.com $\endgroup$
    – D.W.
    Commented Dec 23, 2021 at 1:00
  • $\begingroup$ Thanks @D.W. I've clarified my original question in response to your comments, and provided what I think is an example demonstrating that a prior need not be associated with a regularization term interpretable as a norm. $\endgroup$ Commented Dec 28, 2021 at 23:22
  • 1
    $\begingroup$ @ColinRowat, that's fine. To tell whether it matches a regularization term that can be expressed as a norm, compute the regularization term as explained in my answer, then check whether that term is a norm. In my experience it doesn't matter whether the regularization term is a norm or not. $\endgroup$
    – D.W.
    Commented Dec 28, 2021 at 23:25
  • $\begingroup$ thanks @D.W. - my interest in whether the regularization term is a norm arises purely from curiosity: I don't have any practical application in mind at this point. $\endgroup$ Commented Dec 29, 2021 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.