Space-and computationally-efficient set reconciliation via parity bitmap sketch (Pbs)

Long Gong, Ziheng Liu, Liang Liu, Jun Xu, Mitsunori Ogihara, Tong Yang

Research output: Contribution to journalArticlepeer-review

Abstract

Set reconciliation is a fundamental algorithmic problem that arises in many networking, system, and database applications. In this problem, two large sets A and k of objects (bitcoins, files, records, etc.) are stored respectively at two different network-connected hosts, which we name Alice and Bob respectively. Alice and Bob communicate with each other to learn A△k, the difference between A and k, and as a result the reconciled set A k. Current set reconciliation schemes are based on either invertible Bloom filters (IBF) or error-correction codes (ECC). The former has a low computational complexity of A (k), where k is the cardinality of A△k, but has a high communication overhead that is several times larger than the theoretical minimum. The latter has a low communication overhead close to the theoretical minimum, but has a much higher computational complexity of A (k2 ). In this work, we propose Parity Bitmap Sketch (PBS), an ECC-based set reconciliation scheme that gets the better of both worlds: PBS has both a low computational complexity of A (k) just like IBF-based solutions and a low communication overhead of roughly twice the theoretical minimum. A separate contribution of this work is a novel rigorous analytical framework that can be used for the precise calculation of various performance metrics and for the near-optimal parameter tuning of PBS.

Original languageEnglish (US)
Pages (from-to)458-470
Number of pages13
JournalProceedings of the VLDB Endowment
Volume14
Issue number4
DOIs
StatePublished - 2020

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Computer Science(all)

Fingerprint Dive into the research topics of 'Space-and computationally-efficient set reconciliation via parity bitmap sketch (Pbs)'. Together they form a unique fingerprint.

Cite this