Mp-rw-lsh: An efficient multi-probe lsh solution to anns-l1

Huayi Wang, Jingfan Meng, Long Gong, Jun Xu, Mitsunori Ogihara

Research output: Contribution to journalConference articlepeer-review

Abstract

Approximate Nearest Neighbor Search (ANNS) is a fundamental algorithmic problem, with numerous applications in many areas of computer science. Locality-Sensitive Hashing (LSH) is one of the most popular solution approaches for ANNS. A common shortcoming of many LSH schemes is that since they probe only a single bucket in a hash table, they need to use a large number of hash tables to achieve a high query accuracy. For ANNS-L2, a multi-probe scheme was proposed to overcome this drawback by strategically probing multiple buckets in a hash table. In this work, we propose MP-RW-LSH, the first and so far only multi-probe LSH solution to ANNS in L1 distance, and show that it achieves a better tradeoff between scalability and query efficiency than all existing LSH-based solutions. We also explain why a state-of-the-art ANNS-L1 solution called Cauchy projection LSH (CP-LSH) is fundamentally not suitable for multi-probe extension. Finally, as a use case, we construct, using MP-RW-LSH as the underlying “ANNS-L1 engine”, a new ANNS-E (E for edit distance) solution that beats the state of the art.

Original languageEnglish (US)
Pages (from-to)3267-3280
Number of pages14
JournalProceedings of the VLDB Endowment
Volume14
Issue number13
DOIs
StatePublished - 2021
Event47th International Conference on Very Large Data Bases, VLDB 2021 - Virtual, Online
Duration: Aug 16 2021Aug 20 2021

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Computer Science(all)

Fingerprint

Dive into the research topics of 'Mp-rw-lsh: An efficient multi-probe lsh solution to anns-l<sub>1</sub>'. Together they form a unique fingerprint.

Cite this