TY - JOUR

T1 - Space-and computationally-efficient set reconciliation via parity bitmap sketch (Pbs)

AU - Gong, Long

AU - Liu, Ziheng

AU - Liu, Liang

AU - Xu, Jun

AU - Ogihara, Mitsunori

AU - Yang, Tong

N1 - Funding Information:
This material is based upon work supported by the National Science Foundation under Grant No. CNS-1909048 and CNS-2007006. We thank the reviewers and the shepherd for their critiques, comments and suggestions that have greatly improved the quality and the readability of this paper.

PY - 2020

Y1 - 2020

N2 - Set reconciliation is a fundamental algorithmic problem that arises in many networking, system, and database applications. In this problem, two large sets A and k of objects (bitcoins, files, records, etc.) are stored respectively at two different network-connected hosts, which we name Alice and Bob respectively. Alice and Bob communicate with each other to learn A△k, the difference between A and k, and as a result the reconciled set A⋃ k. Current set reconciliation schemes are based on either invertible Bloom filters (IBF) or error-correction codes (ECC). The former has a low computational complexity of A (k), where k is the cardinality of A△k, but has a high communication overhead that is several times larger than the theoretical minimum. The latter has a low communication overhead close to the theoretical minimum, but has a much higher computational complexity of A (k2 ). In this work, we propose Parity Bitmap Sketch (PBS), an ECC-based set reconciliation scheme that gets the better of both worlds: PBS has both a low computational complexity of A (k) just like IBF-based solutions and a low communication overhead of roughly twice the theoretical minimum. A separate contribution of this work is a novel rigorous analytical framework that can be used for the precise calculation of various performance metrics and for the near-optimal parameter tuning of PBS.

AB - Set reconciliation is a fundamental algorithmic problem that arises in many networking, system, and database applications. In this problem, two large sets A and k of objects (bitcoins, files, records, etc.) are stored respectively at two different network-connected hosts, which we name Alice and Bob respectively. Alice and Bob communicate with each other to learn A△k, the difference between A and k, and as a result the reconciled set A⋃ k. Current set reconciliation schemes are based on either invertible Bloom filters (IBF) or error-correction codes (ECC). The former has a low computational complexity of A (k), where k is the cardinality of A△k, but has a high communication overhead that is several times larger than the theoretical minimum. The latter has a low communication overhead close to the theoretical minimum, but has a much higher computational complexity of A (k2 ). In this work, we propose Parity Bitmap Sketch (PBS), an ECC-based set reconciliation scheme that gets the better of both worlds: PBS has both a low computational complexity of A (k) just like IBF-based solutions and a low communication overhead of roughly twice the theoretical minimum. A separate contribution of this work is a novel rigorous analytical framework that can be used for the precise calculation of various performance metrics and for the near-optimal parameter tuning of PBS.

UR - http://www.scopus.com/inward/record.url?scp=85099158125&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85099158125&partnerID=8YFLogxK

U2 - 10.14778/3436905.3436906

DO - 10.14778/3436905.3436906

M3 - Article

AN - SCOPUS:85099158125

VL - 14

SP - 458

EP - 470

JO - Proceedings of the VLDB Endowment

JF - Proceedings of the VLDB Endowment

SN - 2150-8097

IS - 4

ER -