Scientific workflow management systems (SWMSs) and resource managers together ensure that tasks are scheduled on provisioned resources so that all dependencies are obeyed, and some optimization goal, such as makespan minimization, is fulfilled. In practice, however, there is no clear separation of scheduling responsibilities between an SWMS and a resource manager because there exists no agreed-upon separation of concerns between their different components. This has two consequences. First, the lack of a standardized API to exchange scheduling information between SWMSs and resource managers hinders portability. It incurs costly adaptations when a component should be replaced by another one (e.g., an SWMS with another SWMS on the same resource manager). Second, due to overlapping functionalities, current installations often actually have two schedulers, both making partial scheduling decisions under incomplete information, leading to suboptimal workflow scheduling. In this paper, we propose a simple REST interface between SWMSs and resource managers, which allows any SWMS to pass dynamic workflow information to a resource manager, enabling maximally informed scheduling decisions. We provide an exemplary implementation of this API for Nextflow as an SWMS and Kubernetes as a resource manager. Our experiments with nine real-world workflows show that this strategy reduces makespan by up to 25.1% and 10.8% on average compared to the standard Nextflow/Kubernetes configuration. Furthermore, a more widespread implementation of this API would enable leaner code bases, a simpler exchange of components of workflow systems, and a unified place to implement new scheduling algorithms.

翻译：科学工作流管理系统（SWMS）与资源管理器共同确保任务在已配置资源上调度，从而满足所有依赖关系并实现诸如最小化完工时间等优化目标。然而在实践中，由于两者不同组件之间缺乏公认的关注点分离原则，SWMS与资源管理器之间并不存在清晰的调度职责划分。这导致两个后果：首先，缺乏标准化的API用于在SWMS与资源管理器之间交换调度信息，这阻碍了可移植性——当某个组件需被替换（例如在同一资源管理器上更换另一个SWMS）时，会引发高昂的适配成本；其次，由于功能重叠，当前部署环境通常实际运行着两个调度器，各自基于不完整信息做出部分调度决策，导致工作流调度结果欠优。本文提出一个介于SWMS与资源管理器之间的简单REST接口，该接口允许任意SWMS向资源管理器传递动态工作流信息，从而实现信息最充分的调度决策。我们以Nextflow为SWMS、Kubernetes为资源管理器给出了该API的示例实现。基于九个真实工作流的实验表明，与标准Nextflow/Kubernetes配置相比，该策略可将完工时间平均缩短10.8%，最高达25.1%。此外，该API的更广泛实现将有助于精简代码库、简化工作流系统组件的替换流程，并为新型调度算法的实现提供统一平台。