Ensuring resilience in distributed systems has become an acute concern. In today's environment, it is crucial to develop light-weight mechanisms that recover a distributed system from faults quickly and with only a small impact on the live-system throughput. To address this need, this paper proposes a new low-overhead, general recovery scheme for modern non-transactional leaderless distributed systems. We call our scheme HEAL. On a node failure, HEAL performs an optimized online incremental recovery. This paper presents HEAL's algorithms for settings with Linearizable consistency and different memory persistency models. We implement HEAL on a 6-node Intel cluster. Our experiments running TAOBench workloads show that HEAL is very effective. HEAL recovers the cluster in 120 milliseconds on average, while reducing the throughput of the running workload by an average of 8.7%. In contrast, a conventional recovery scheme for leaderless systems needs 360 seconds to recover, reducing the throughput of the system by 16.2%. Finally, compared to an incremental recovery scheme for a state-of-the-art leader-based system, HEAL reduces the average recovery latency by 20.7x and the throughput degradation by 62.4%.

翻译：确保分布式系统的弹性已成为一个紧迫问题。在当今环境中，开发轻量级机制以快速从故障中恢复分布式系统，且对在线系统吞吐量影响微小，至关重要。为满足这一需求，本文提出了一种适用于现代非事务性无领导者分布式系统的新型低开销通用恢复方案。我们称该方案为HEAL。在节点发生故障时，HEAL执行优化的在线增量恢复。本文阐述了HEAL在可线性化一致性及不同内存持久性模型设置下的算法。我们在一个6节点Intel集群上实现了HEAL。运行TAOBench工作负载的实验表明，HEAL非常有效。HEAL平均在120毫秒内恢复集群，同时使运行工作负载的吞吐量平均仅降低8.7%。相比之下，传统的无领导者系统恢复方案需要360秒进行恢复，并使系统吞吐量降低16.2%。最后，与最先进的基于领导者系统的增量恢复方案相比，HEAL将平均恢复延迟降低了20.7倍，并将吞吐量下降减少了62.4%。