The escalating size of Deep Neural Networks (DNNs) has spurred a growing research interest in hosting and serving DNN models across multiple devices. A number of studies have been reported to partition a DNN model across devices, providing device placement solutions. The methods appeared in the literature, however, either suffer from poor placement performance due to the exponential search space or miss an optimal placement as a consequence of the reduced search space with limited heuristics. Moreover, these methods have ignored the runtime inter-operator optimization of a computation graph when coarsening the graph, which degrades the end-to-end inference performance. This paper presents Moirai that better exploits runtime inter-operator fusion in a model to render a coarsened computation graph, reducing the search space while maintaining the inter-operator optimization provided by inference backends. Moirai also generalizes the device placement algorithm from multiple perspectives by considering inference constraints and device heterogeneity.Extensive experimental evaluation with 11 large DNNs demonstrates that Moirai outperforms the state-of-the-art counterparts, i.e., Placeto, m-SCT, and GETF, up to 4.28$\times$ in reduction of the end-to-end inference latency. Moirai code is anonymously released at \url{https://github.com/moirai-placement/moirai}.

翻译：深度神经网络规模的不断增长，促使人们越来越多地研究如何跨多台设备托管和部署DNN模型。已有研究提出将DNN模型分割至多台设备，并提供设备放置方案。然而，现有方法要么因指数级搜索空间而导致放置性能不佳，要么因有限启发式方法缩小搜索空间而错过最优放置。此外，这些方法在粗化计算图时忽略了运行时算子间优化，从而降低了端到端推理性能。本文提出Moirai，该方法能更好地利用模型中的运行时算子融合生成粗化计算图，在保持推理后端提供的算子间优化的同时缩小搜索空间。Moirai还通过考虑推理约束和设备异质性，从多个角度泛化了设备放置算法。在11个大型DNN上的广泛实验评估表明，Moirai在端到端推理延迟缩减方面优于最先进的对比方法（即Placeto、m-SCT和GETF），最高可达4.28倍。Moirai代码已匿名发布至 \url{https://github.com/moirai-placement/moirai}。