Set reconciliation, where two parties hold fixed-length bit strings and run a protocol to learn the strings they are missing from each other, is a fundamental task in many distributed systems. We present Rateless Invertible Bloom Lookup Tables (Rateless IBLT), the first set reconciliation protocol, to the best of our knowledge, that achieves low computation cost and near-optimal communication cost across a wide range of scenarios: set differences of one to millions, bit strings of a few bytes to megabytes, and workloads injected by potential adversaries. Rateless IBLT is based on a novel encoder that incrementally encodes the set difference into an infinite stream of coded symbols, resembling rateless error-correcting codes. We compare Rateless IBLT with state-of-the-art set reconciliation schemes and demonstrate significant improvements. Rateless IBLT achieves 3--4x lower communication cost than non-rateless schemes with similar computation cost, and 2--2000x lower computation cost than schemes with similar communication cost. We show the real-world benefits of Rateless IBLT by applying it to synchronize the state of the Ethereum blockchain, and demonstrate 5.6x lower end-to-end completion time and 4.4x lower communication cost compared to the system used in production.

翻译：集合协调是指两方各自持有固定长度的比特串，并通过运行协议来获知对方持有而己方缺失的串，这是许多分布式系统中的一项基本任务。我们提出了无速率可逆布鲁姆查找表，据我们所知，这是首个在广泛场景下实现低计算成本和接近最优通信成本的集合协调协议：这些场景包括集合差异从一到数百万、比特串从几个字节到数兆字节，以及潜在对手注入的工作负载。无速率可逆布鲁姆查找表基于一种新颖的编码器，该编码器将集合差异增量编码成一个无限的编码符号流，类似于无速率纠错码。我们将无速率可逆布鲁姆查找表与最先进的集合协调方案进行比较，并展示了显著的改进。与计算成本相似的非无速率方案相比，无速率可逆布鲁姆查找表实现了3-4倍更低的通信成本；与通信成本相似的方案相比，其计算成本降低了2-2000倍。我们通过将无速率可逆布鲁姆查找表应用于同步以太坊区块链的状态，展示了其实际效益：与生产环境中使用的系统相比，端到端完成时间降低了5.6倍，通信成本降低了4.4倍。