Scientific workflow management systems (SWMSs) and resource managers together ensure that tasks are scheduled on provisioned resources so that all dependencies are obeyed, and some optimization goal, such as makespan minimization, is achieved. In practice, however, there is no clear separation of scheduling responsibilities between an SWMS and a resource manager because there exists no agreed-upon separation of concerns between their different components. This has two consequences. First, the lack of a standardized API to exchange scheduling information between SWMSs and resource managers hinders portability. It incurs costly adaptations when a component should be replaced by a different one (e.g., an SWMS with another SWMS on the same resource manager). Second, due to overlapping functionalities, current installations often actually have two schedulers, both making partial scheduling decisions under incomplete information, leading to suboptimal workflow scheduling. In this paper, we propose a simple REST interface between SWMSs and resource managers, which allows any SWMS to pass dynamic workflow information to a resource manager, enabling maximally informed scheduling decisions. We provide an implementation of this API as an example, using Nextflow as an SWMS and Kubernetes as a resource manager. Our experiments with nine real-world workflows show that this strategy reduces makespan by up to 25.1% and 10.8% on average compared to the standard Nextflow/Kubernetes configuration. Furthermore, a more widespread implementation of this API would enable leaner code bases, a simpler exchange of components of workflow systems, and a unified place to implement new scheduling algorithms.

翻译：科学工作流管理系统（SWMS）与资源管理器共同确保任务在配置资源上被调度执行，以满足所有依赖关系并实现优化目标（如最小化完工时间）。然而在实际应用中，两者之间缺乏明确的调度职责划分，因为其不同组件之间不存在公认的关注点分离原则。这导致两个后果：首先，SWMS与资源管理器之间缺乏标准化API来交换调度信息，阻碍了系统可移植性——当某个组件需要被替换（例如在同一资源管理器上更换SWMS）时，将产生高昂的适配成本；其次，由于功能重叠，当前系统通常实际运行着两个调度器，两者均在信息不完整的情况下做出部分调度决策，导致工作流调度效果欠佳。本文提出一种SWMS与资源管理器间的简洁REST接口，允许任意SWMS向资源管理器传递动态工作流信息，从而实现信息最大化的调度决策。我们以Nextflow作为SWMS、Kubernetes作为资源管理器，提供了该API的参考实现。基于九个真实工作流的实验表明，与标准Nextflow/Kubernetes配置相比，该策略可将完工时间平均缩短10.8%，最高达25.1%。此外，该API的广泛部署将有助于精简代码库、简化工作流系统组件的更换流程，并为新型调度算法提供统一实现平台。