4
$\begingroup$

An easy question perhaps?

Taking a (fictional) concrete example, let's say I have two ranking methods for HTML documents: PageRank and HITS. I derive an ordered list over the same set of documents according to their rank values (there can be ties) for both ranking methods.

Now, I want to quantify how much the two orderings correlate. I'm not interested in the absolute values of the ranking measures, but I'm looking for measures of correlation between the orderings.

I can think of some intuitive correlation measures one could apply like "average distance" or such:

${\displaystyle\frac{\sum_{doc_i\in Docs}\mathrm{abs}(pos_{rank1}(doc_i) - pos_{rank2}(doc_i))}{|Docs|^2}}$

but I'm wondering if anyone knows of any standard techniques I could check out?

In particular, I would prefer a symmetric measure which is easy to intuit and whose results are tangible/"portable". I've a number of different orderings which I'll be comparing pair-wise, but the set of $Docs$ remains the same.

Many thanks in advance!

(having difficulty tagging... please feel free to edit tags)

$\endgroup$
8
$\begingroup$

The standard approach to computing the correlation between two total orders is Kendall's $\tau$, defined as the difference between the number of agreeing pairs and disagreeing pairs (in rank) divided by $\binom{n}{2}$.

If you only have partial orders, then things get a little trickier.

$\endgroup$
1
  • $\begingroup$ Many thanks for the answer and pointers! "Rank correlation" and Google was what I needed... :) $\endgroup$
    – badroit
    Feb 23 '11 at 3:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.