StreamBed is a capacity planning system for stream processing. It predicts, ahead of any production deployment, the resources that a query will require to process an incoming data rate sustainably, and the appropriate configuration of these resources. StreamBed builds a capacity planning model by piloting a series of runs of the target query in a small-scale, controlled testbed. We implement StreamBed for the popular Flink DSP engine. Our evaluation with large-scale queries of the Nexmark benchmark demonstrates that StreamBed can effectively and accurately predict capacity requirements for jobs spanning more than 1,000 cores using a testbed of only 48 cores.
翻译:StreamBed 是一套面向流处理系统的容量规划系统。该系统能够在生产部署之前,预测查询在可持续处理指定数据速率时所需的资源量,以及这些资源的合理配置方案。StreamBed 通过在小规模受控测试平台上引导目标查询的系列运行,构建容量规划模型。我们基于 Flink 分布式流处理引擎实现了 StreamBed。针对 Nexmark 基准测试中大规模查询的评估表明,StreamBed 能够利用仅含 48 个核心的测试平台,有效且精准地预测横跨 1000 多个核心作业的容量需求。