Suppose we have a dataframe with ~10M rows with ~9M duplicate records. What is the most time efficient way of selecting the unique records from this dataframe?
Some sort of sampling algorithm?
Theoretical Computer Science Stack Exchange is a question and answer site for theoretical computer scientists and researchers in related fields. It only takes a minute to sign up.
Sign up to join this communitySuppose we have a dataframe with ~10M rows with ~9M duplicate records. What is the most time efficient way of selecting the unique records from this dataframe?
Some sort of sampling algorithm?
In the context of Theoretical Computer Science, there are various strategies to (quickly) select the unique elements of a list, mainly comparison based and value based.
See any algorithmic textbook (e.g. CLRS comes to mind) for an introduction to Hash tables. For sorting multisets while taking advantage of repetitions (and even of the entropy of the distribution of the frequencies of the elements), see Munro and Spira's 1976 seminal article, or Barbay et al. synergistic sorting algoritm (shameless self plug!) which takes optimally advantage of both (some measures of) input order and structure (i.e. the repetitions, down to the entropy of the distribution of frequencies).
I hope it helps! Take care!