In this project we explore ways to dynamically load balance actors in a streaming framework. This is used to address input data skew that might lead to stragglers. We continuously monitor actors' input queue lengths for load, and redistribute inputs among reducers using consistent hashing if we detect stragglers. To ensure consistent processing post-redistribution, we adopt an approach that uses input forwarding combined with a state merge step at the end of the processing. We show that this approach can greatly alleviate stragglers for skewed data.

翻译：在本项目中，我们探索了在流式处理框架中动态平衡Actor负载的方法，用于解决可能导致落后者的输入数据倾斜问题。我们持续监测Actor的输入队列长度以获取负载信息，当检测到落后节点时，通过一致性哈希在归约器之间重新分配输入。为确保重新分配后处理的一致性，我们采用了一种结合输入转发与处理结束时状态合并的方法。实验表明，该方法能显著缓解数据倾斜导致的落后者问题。