Datacenters are increasingly becoming heterogeneous, and are starting to include specialized hardware for networking, video processing, and especially deep learning. To leverage the heterogeneous compute capability of modern datacenters, we develop an approach for compiler-level partitioning of deep neural networks (DNNs) onto multiple interconnected hardware devices. We present a general framework for heterogeneous DNN compilation, offering automatic partitioning and device mapping. Our scheduler integrates both an exact solver, through a mixed integer linear programming (MILP) formulation, and a modularity-based heuristic for scalability. Furthermore, we propose a theoretical lower bound formula for the optimal solution, which enables the assessment of the heuristic solutions' quality. We evaluate our scheduler in optimizing both conventional DNNs and randomly-wired neural networks, subject to latency and throughput constraints, on a heterogeneous system comprised of a CPU and two distinct GPUs. Compared to na\"ively running DNNs on the fastest GPU, he proposed framework can achieve more than 3$\times$ times lower latency and up to 2.9$\times$ higher throughput by automatically leveraging both data and model parallelism to deploy DNNs on our sample heterogeneous server node. Moreover, our modularity-based "splitting" heuristic improves the solution runtime up to 395$\times$ without noticeably sacrificing solution quality compared to an exact MILP solution, and outperforms all other heuristics by 30-60% solution quality. Finally, our case study shows how we can extend our framework to schedule large language models across multiple heterogeneous servers by exploiting symmetry in the hardware setup. Our code can be easily plugged in to existing frameworks, and is available at https://github.com/abdelfattah-lab/diviml.

翻译：数据中心正日益走向异构化，并开始集成用于网络传输、视频处理尤其是深度学习的专用硬件。为充分利用现代数据中心的异构计算能力，我们提出了一种编译器级方法，用于将深度神经网络（DNN）分割到多个互连硬件设备上。本文呈现了一个通用的异构DNN编译框架，可实现自动划分与设备映射。我们的调度器集成了精确求解器（通过混合整数线性规划（MILP）公式）和基于模块化的可扩展启发式方法。此外，我们提出了一种最优解的理论下界公式，可用于评估启发式解的质量。我们在由单个CPU和两个不同GPU组成的异构系统上，针对延迟和吞吐量约束，对常规DNN和随机连接神经网络进行了调度优化。相较在最快GPU上直接运行DNN，所提框架通过自动利用数据并行和模型并行将DNN部署到样本异构服务器节点上，可降低延迟3倍以上，吞吐量最高提升2.9倍。此外，与精确MILP解相比，我们的模块化"分割"启发式方法在不明显牺牲解质量的前提下，将求解运行时间提升高达395倍，且解质量优于其他所有启发式方法30-60%。最后，我们的案例研究表明，如何通过利用硬件配置的对称性，将该框架扩展至跨多个异构服务器调度大型语言模型。相关代码可轻松嵌入现有框架，并已开源至https://github.com/abdelfattah-lab/diviml。