Distributed Stream Processing systems are becoming an increasingly essential part of Big Data processing platforms as users grow ever more reliant on their ability to provide fast access to new results. As such, making timely decisions based on these results is dependent on a system's ability to tolerate failure. Typically, these systems achieve fault tolerance and the ability to recover automatically from partial failures by implementing checkpoint and rollback recovery. However, owing to the statistical probability of partial failures occurring in these distributed environments and the variability of workloads upon which jobs are expected to operate, static configurations will often not meet Quality of Service constraints with low overhead. In this paper we present Khaos, a new approach which utilizes the parallel processing capabilities of virtual cloud automation technologies for the automatic runtime optimization of fault tolerance configurations in Distributed Stream Processing jobs. Our approach employs three subsequent phases which borrows from the principles of Chaos Engineering: establish the steady-state processing conditions, conduct experiments to better understand how the system performs under failure, and use this knowledge to continuously minimize Quality of Service violations. We implemented Khaos prototypically together with Apache Flink and demonstrate its usefulness experimentally.

翻译：分布式流处理系统正日益成为大数据处理平台的关键组成部分，用户越来越依赖其快速提供新结果的能力。基于这些结果做出及时决策，取决于系统对故障的容错能力。通常，这些系统通过实现检查点与回滚恢复机制来达到容错性，并具备从部分故障中自动恢复的能力。然而，由于分布式环境中部分故障发生的统计概率以及作业所处理工作负载的可变性，静态配置往往无法在低开销下满足服务质量约束。本文提出了一种名为Khaos的新方法，该方法利用虚拟云自动化技术的并行处理能力，对分布式流处理作业中的容错配置进行自动运行时优化。我们的方法借鉴混沌工程原理，包含三个后续阶段：建立稳态处理条件、通过实验更深入地了解系统在故障下的表现，并利用这些知识持续减少服务质量违规。我们将Khaos与Apache Flink进行了原型实现，并通过实验证明了其有效性。