Modern databases use dynamic search structures that store an enormous amount of data, and often serve them using multi-threaded algorithms to support the ever-increasing throughput needs. When this throughput need exceeds the capacity of the machine hosting the structure, one either needs to replace the underlying hardware (an option that is typically not viable and introduces a long down time) or make the data structure distributed. Static partitioning of the data structure for distribution is not desirable, as it is prone to uneven load distribution over time, and having to change the partitioning scheme later will require downtime. The goal of this paper is to extend a concurrent data structure to distributed data structures that provide dynamic load balancing while preserving important properties such as lock freedom. With this intuition, first, we introduce the notion of conditional lock-freedom that extends the notion of lock-free computation with reasonable assumptions about communication between processes. Then, we present DiLi, a conditional lock-free, linearizable, and distributable linked list that can be asynchronously and dynamically (1) partitioned into multiple sublists and (2) load balanced by distributing sublists across multiple machines. DiLi contains primitives for these that also maintain the lock-free property of the underlying search structure that supports find, remove, and insert of a key as the client operations. We show that DiLi bridges the gap between concurrent data structures and distributed data structures. Specifically, DiLi provides comparable (and better in write-intensive workloads) performance to skip lists (which are typically the fastest data structures for search in a concurrent environment). Plus, it provides horizontal scaling by permitting dynamic load balancing on a distributed environment.

翻译：现代数据库采用动态搜索结构存储海量数据，通常通过多线程算法提供服务以满足日益增长的吞吐量需求。当吞吐需求超出承载该结构的机器容量时，要么需要更换底层硬件（这一方案通常不可行且会导致长时间停机），要么必须将数据结构分布式化。对数据结构进行静态分区以实现分布并不可取，因为这种方式容易随时间推移导致负载分布不均，且后续若需更改分区方案将再次引发停机。本文的目标是将并发数据结构扩展为分布式数据结构，使其在保持无锁等重要特性的同时提供动态负载均衡。基于这一思路，我们首先提出条件无锁的概念，该概念在合理假设进程间通信的基础上扩展了无锁计算的定义。随后，我们提出DiLi——一种具备条件无锁性、可线性化且可分布化的链表结构，能够异步动态地（1）被分割为多个子链表，并（2）通过将子链表分布到多台机器实现负载均衡。DiLi包含实现这些功能的原语，同时保持了底层搜索结构的无锁特性，支持查找、删除和插入键值等客户端操作。我们证明DiLi填补了并发数据结构与分布式数据结构之间的空白。具体而言，DiLi在性能上可与跳表（并发环境下通常最快的搜索数据结构）相媲美（在写密集型负载中表现更优），同时通过允许在分布式环境中进行动态负载均衡，实现了横向扩展能力。