Nowadays, several software systems rely on stream processing architectures to deliver scalable performance and handle large volumes of data in near real-time. Stream processing frameworks facilitate scalable computing by distributing the application's execution across multiple machines. Despite performance being extensively studied, the measurement of fault tolerance-a key feature offered by stream processing frameworks-has still not been measured properly with updated and comprehensive testbeds. Moreover, the impact that fault recovery can have on performance is mostly ignored. This paper provides a comprehensive analysis of fault recovery performance, stability, and recovery time in a cloud-native environment with modern open-source frameworks, namely Flink, Kafka Streams, and Spark Structured Streaming. Our benchmarking analysis is inspired by chaos engineering to inject failures. Generally, our results indicate that much has changed compared to previous studies on fault recovery in distributed stream processing. In particular, the results indicate that Flink is the most stable and has one of the best fault recovery. Moreover, Kafka Streams shows performance instabilities after failures, which is due to its current rebalancing strategy that can be suboptimal in terms of load balancing. Spark Structured Streaming shows suitable fault recovery performance and stability, but with higher event latency. Our study intends to (i) help industry practitioners in choosing the most suitable stream processing framework for efficient and reliable executions of data-intensive applications; (ii) support researchers in applying and extending our research method as well as our benchmark; (iii) identify, prevent, and assist in solving potential issues in production deployments.

翻译：当前，众多软件系统依赖流处理架构以实现可扩展性能并近乎实时地处理海量数据。流处理框架通过将应用程序执行分布至多台机器，促进了可扩展计算。尽管性能已得到广泛研究，但流处理框架所提供的关键特性——容错能力的评估，至今仍缺乏基于更新且全面测试平台的恰当衡量。此外，故障恢复对性能的影响大多被忽视。本文在现代开源框架（即 Flink、Kafka Streams 和 Spark Structured Streaming）构成的云原生环境中，对故障恢复性能、稳定性及恢复时间进行了综合分析。我们的基准测试方法受混沌工程启发，通过注入故障开展实验。总体而言，结果表明相较于以往关于分布式流处理故障恢复的研究，现状已发生显著变化。具体而言，结果显示 Flink 最为稳定且具备最佳的故障恢复能力之一。此外，Kafka Streams 在故障后表现出性能不稳定性，这源于其当前再平衡策略在负载均衡方面可能并非最优。Spark Structured Streaming 展现出适宜的故障恢复性能与稳定性，但事件延迟较高。本研究旨在：（i）帮助行业从业者选择最适合的流处理框架，以实现数据密集型应用的高效可靠执行；（ii）支持研究人员应用并扩展我们的研究方法及基准测试框架；（iii）识别、预防并协助解决生产部署中的潜在问题。