Scientific workflow management systems (SWMSs) and resource managers together ensure that tasks are scheduled on provisioned resources so that all dependencies are obeyed, and some optimization goal, such as makespan minimization, is achieved. In practice, however, there is no clear separation of scheduling responsibilities between an SWMS and a resource manager because there exists no agreed-upon separation of concerns between their different components. This has two consequences. First, the lack of a standardized API to exchange scheduling information between SWMSs and resource managers hinders portability. It incurs costly adaptations when a component should be replaced by a different one (e.g., an SWMS with another SWMS on the same resource manager). Second, due to overlapping functionalities, current installations often actually have two schedulers, both making partial scheduling decisions under incomplete information, leading to suboptimal workflow scheduling. In this paper, we propose a simple REST interface between SWMSs and resource managers, which allows any SWMS to pass dynamic workflow information to a resource manager, enabling maximally informed scheduling decisions. We provide an implementation of this API as an example, using Nextflow as an SWMS and Kubernetes as a resource manager. Our experiments with nine real-world workflows show that this strategy reduces makespan by up to 25.1% and 10.8% on average compared to the standard Nextflow/Kubernetes configuration. Furthermore, a more widespread implementation of this API would enable leaner code bases, a simpler exchange of components of workflow systems, and a unified place to implement new scheduling algorithms.

翻译：科学工作流管理系统（SWMS）与资源管理器共同确保任务在已配置资源上按时调度，使得所有依赖关系得到遵守，并实现如完工时间最小化等优化目标。然而在实际中，由于SWMS与资源管理器各组件之间缺乏公认的关注点分离原则，导致两者间不存在清晰的调度职责划分。这带来两个后果：首先，SWMS与资源管理器之间缺乏标准化的应用编程接口（API）来交换调度信息，阻碍了可移植性——当某个组件需被替换为不同组件时（例如在同一资源管理器上替换SWMS），将产生高昂的适配成本；其次，由于功能重叠，现有系统部署中往往存在两个调度器，它们都在信息不完整的情况下做出部分调度决策，导致工作流调度次优。本文提出一个SWMS与资源管理器之间的简易REST接口，允许任意SWMS将动态工作流信息传递给资源管理器，从而实现信息最大化的调度决策。我们以Nextflow作为SWMS、Kubernetes作为资源管理器，提供了该API的示例实现。对九个真实工作流的实验表明，与标准Nextflow/Kubernetes配置相比，该策略可使完工时间平均降低10.8%，最高降低25.1%。此外，该API的广泛推广将有助于更精简的代码库、更简单的工作流系统组件替换，以及统一实现新调度算法的平台。