StreamBed is a capacity planning system for stream processing.It predicts, ahead of any production deployment, the resources that a query will require to process an incoming data rate sustainably, and the appropriate configuration of these resources. StreamBed builds a capacity planning model by piloting a series of runs of the target query in a small-scale, controlled testbed. We implement StreamBed for the popular Flink DSP engine. Our evaluation with large-scale queries of the Nexmark benchmark demonstrates that StreamBed can effectively and accurately predict capacity requirements for jobs spanning more than 1,000 cores using a testbed of only 48 cores.
翻译:StreamBed是一个面向流处理的容量规划系统。它能在生产环境部署之前,预测某条查询语句在可持续处理输入数据速率时所需的资源量以及这些资源的适当配置。StreamBed通过在小规模受控测试平台上对目标查询语句进行一系列引导性运行来构建容量规划模型。我们针对流行的Flink DSP引擎实现了StreamBed。使用Nexmark基准测试中的大规模查询进行的评估表明,StreamBed能够仅凭48个核心的测试平台,有效且准确地预测需要超过1000个核心的作业的容量需求。