Message brokers often mediate communication between data producers and consumers by adding variable-sized messages to ordered distributed queues. Our goal is to determine the number of consumers and consumer-partition assignments needed to ensure that the rate of data consumption keeps up with the rate of data production. We model the problem as a variable item size bin packing problem. As the rate of production varies, new consumer-partition assignments are computed, which may require rebalancing a partition from one consumer to another. While rebalancing a queue, the data being produced into the queue is not read leading to additional latency costs. As such, we focus on the multi-objective optimization cost of minimizing both the number of consumers and queue migrations. We present a variety of algorithms and compare them to established bin packing heuristics for this application. Comparing our proposed consumer group assignment strategy with Kafka's, a commonly employed strategy, our strategy presents a 90th percentile latency of 4.52s compared to Kafka's 217s with both using the same amount of consumers. Kafka's assignment strategy only improved the consumer group's performance with regards to latency with configurations that used at least 60% more resources than our approach.

翻译：消息代理通常通过向有序分布式队列中添加可变大小的消息，来协调数据生产者和消费者之间的通信。我们的目标是确定消费者数量以及消费者与分区的分配方案，以确保数据消费速率能够跟上数据生产速率。我们将该问题建模为可变物品尺寸的装箱问题。随着生产速率的变化，需要计算新的消费者-分区分配方案，这可能要求重新平衡分区从一个消费者迁移到另一个消费者。在重新平衡队列期间，正在写入队列的数据无法被读取，从而导致额外的延迟成本。因此，我们聚焦于最小化消费者数量和队列迁移次数的多目标优化成本。我们提出了一系列算法，并将其与针对该应用场景的经典装箱启发式算法进行比较。将我们提出的消费组分配策略与常用的Kafka策略对比，在两者使用相同消费者数量的情况下，我们的策略实现了第90百分位延迟4.52秒，而Kafka为217秒。Kafka的分配策略仅在配置资源比我们的方法至少多出60%时，才能改善消费组在延迟方面的性能。