I want to compute Euclidian distance between similar vectors in a database (SQLite). So each column in the database is a value from my vector.
The first problem appears, I have a large number of values that I first need to reduce to fit a reasonable number of rows (<50) since the original data have thousands of columns. I have also thought of storing a blob, load it in memory and do my computations ?
Since I am trying to find the best matches (similarity mesure), I am currently filtering the database by applying a threshold for each value. Applying a filter for a small number of column is feasible, doing this for many columns is too time consuming.
Sqlite has a RTree module limited to 5 dimensions (11 columns). But it is not the filtering that I want.
What is the best compromise or technique/algorithm to sort and compare vectors using a database ?