Various resources as the essential elements of data centers, and the completion time is vital to users. In terms of the persistence, the periodicity and the spatial-temporal dependence of stream workload, a new Storm scheduler with Advantage Actor-Critic is proposed to improve resource utilization for minimizing the completion time. A new weighted embedding with a Graph Neural Network is designed to depend on the features of a job comprehensively, which includes the dependence, the types and the positions of tasks in a job. An improved Advantage Actor-Critic integrating task chosen and executor assignment is proposed to schedule tasks to executors in order to better resource utilization. Then the status of tasks and executors are updated for the next scheduling. Compared to existing methods, experimental results show that the proposed Storm scheduler improves resource utilization. The completion time is reduced by almost 17\% on the TPC-H data set and reduced by almost 25\% on the Alibaba data set.
翻译:数据中心的核心要素是多样化的资源,而作业完成时间对用户至关重要。针对流式工作负载的持久性、周期性和时空依赖性特征,本文提出了一种基于优势行动者-评论家(Advantage Actor-Critic)的新型Storm调度器,通过提升资源利用率以最小化作业完成时间。首先设计了一种基于图神经网络(Graph Neural Network)的加权嵌入方法,该方法能全面表征作业特征,包括任务的依赖关系、类型及在作业中的位置。其次提出了一种融合任务选择与执行器分配的改进型优势行动者-评论家算法,通过将任务调度至执行器以实现更优的资源利用率。最终,系统会更新任务与执行器的状态以用于下一轮调度。实验结果表明,与现有方法相比,所提出的Storm调度器显著提升了资源利用率:在TPC-H数据集上完成时间减少约17%,在阿里巴巴数据集上减少约25%。