3
$\begingroup$

I am just hoping to confirm my hypothesis, that a single MLP (untrained and randomly initialized) can be used for random projection for dimensionality reduction.

If a random MLP layer with no activation is:

$$Y = X \ W $$

is there really any difference to the random projection method, especially if its weights are sampled from a distribution with a variance following the Johnson–Lindenstrauss lemma ?

$\endgroup$

1 Answer 1

1
$\begingroup$

A single MLP layer with no activation function is a linear function, as you already seem to know, and hence trivially equivalent to a projection. It is equivalent to a random projection if you choose the entries in the matrix randomly from the appropriate distribution.

Normally a single-layer MLP is understood to include an activation. That is not equivalent to a random projection.

The value of this equivalence is unclear. If you want a random projection, it seems clearer to simply use a random projection, without needing to introduce a MLP.

$\endgroup$
1
  • $\begingroup$ Thanks for the response. I'm asking because I want to use a random projection in a Tensorflow project, and as far as I can tell, TF doesn't have it yet. As for an MLP without an activation, I've seen some projects use it for linear operations (like here for example). $\endgroup$
    – Liam F-A
    Commented Jul 11, 2023 at 2:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.