Extreme dynamic heterogeneity in high performance computing systems and the convergence of traditional HPC with new simulation, analysis, and data science approaches impose increasingly more complex requirements on resource and job management software (RJMS). However, there is a paucity of RJMS techniques that can solve key technical challenges associated with those new requirements, particularly when they are coupled. In this paper, we propose a novel dynamic and multi-level resource model approach to address three key well-known challenges individually and in combination: i.e., 1) RJMS dynamism to facilitate job and workflow adaptability, 2) integration of specialized external resources (e.g. user-centric cloud bursting), and 3) scheduling cloud orchestration framework tasks. The core idea is to combine a dynamic directed graph resource model with fully hierarchical scheduling to provide a unified solution to all three key challenges. Our empirical and analytical evaluations of the solution using our prototype extension to Fluxion, a production hierarchical graph-based scheduler, suggest that our unified solution can significantly improve flexibility, performance and scalability across all three problems in comparison to limited traditional approaches.

翻译：高性能计算机系统的极端动态差异性,传统高频计算系统与新的模拟、分析和数据科学方法的趋同,对资源和工作管理软件(RJMS)提出了越来越复杂的要求。然而,RJMS技术很少能够解决与这些新要求相关的关键技术挑战,特别是当这些要求同时出现时。在本文件中,我们提出了一个新的动态和多层次资源模型方法,以单独和综合地应对三大众所周知的关键挑战:1)RJMS动态,以促进工作和工作流程的适应性;2)专门外部资源(例如,以用户为中心的云爆破)的整合,以及3)云管框架任务的时间安排。核心思想是将动态定向图表资源模型与完全分级安排结合起来,为所有三大挑战提供统一的解决办法。我们用原型扩展至通量、生产分级图表仪对解决方案进行的经验和分析性评价表明,与有限的传统方法相比,我们的统一解决方案可以大大提高所有三个问题的灵活性、性能和可扩展性。