Workloads in data processing clusters are often represented in the form of DAG (Directed Acyclic Graph) jobs. Scheduling DAG jobs is challenging. Simple heuristic scheduling algorithms are often adopted in practice in production data centres. There is much room for scheduling performance optimisation for cost saving. Recently, reinforcement learning approaches (like decima) have been attempted to optimise DAG job scheduling and demonstrate clear performance gain in comparison to traditional algorithms. However, reinforcement learning (RL) approaches face their own problems in real-world deployment. In particular, their black-box decision making processes and generalizability in unseen workloads may add a non-trivial burden to the cluster administrators. Moreover, adapting RL models on unseen workloads often requires significant amount of training data, which leaves edge cases run in a sub-optimal mode. To fill the gap, we propose a new method to distill a simple scheduling policy based on observations of the behaviours of a complex deep learning model. The simple model not only provides interpretability of scheduling decisions, but also adaptive to edge cases easily through tuning. We show that our method achieves high fidelity to the decisions made by deep learning models and outperforms these models when additional heuristics are taken into account.

翻译：数据处理集群中的工作负载通常以有向无环图（DAG）作业的形式表示。DAG作业的调度具有挑战性。在实际生产数据中心中，通常采用简单的启发式调度算法。调度性能在成本节约方面存在很大的优化空间。最近，强化学习方法（如decima）已被尝试用于优化DAG作业调度，并展现出相较于传统算法的明显性能提升。然而，强化学习方法在现实世界部署中面临自身的问题。特别是，其黑盒决策过程以及对未见工作负载的泛化能力，可能会给集群管理员带来不小的负担。此外，在未见工作负载上调整强化学习模型通常需要大量训练数据，这导致边缘案例在次优模式下运行。为填补这一空白，我们提出了一种新方法，通过观察复杂深度学习模型的行为，蒸馏出一个简单的调度策略。该简单模型不仅提供了调度决策的可解释性，还能通过调优轻松适应边缘案例。我们证明，我们的方法对深度学习模型做出的决策具有很高的保真度，并且在考虑额外启发式规则时，其性能优于这些模型。