Deep learning models have demonstrated impressive performance in various domains. However, the prolonged training time of these models remains a critical problem. Manually designed parallel training strategies could enhance efficiency but require considerable time and deliver little flexibility. Hence, automatic parallelism is proposed to automate the parallel strategy searching process. Even so, existing approaches suffer from sub-optimal strategy space because they treat automatic parallelism as two independent stages, namely inter- and intra-layer parallelism. To address this issue, we propose UniAP, which utilizes mixed integer quadratic programming to unify inter- and intra-layer automatic parallelism. To the best of our knowledge, UniAP is the first work to unify these two categories to search for a globally optimal strategy. The experimental results show that UniAP outperforms state-of-the-art methods by up to 1.70$\times$ in throughput and reduces strategy searching time by up to 16$\times$ across four Transformer-like models.

翻译：摘要：深度学习模型已在多个领域展现出卓越性能。然而，这些模型的长时间训练问题仍是一个关键挑战。人工设计的并行训练策略虽能提升效率，但耗时且灵活性较差。因此，自动并行方法被提出以实现并行策略搜索的自动化。尽管如此，现有方法因将自动并行视为层间与层内两个独立阶段，导致策略搜索空间次优。为解决此问题，我们提出UniAP，该方法通过混合整数二次规划统一层间与层内自动并行策略。据我们所知，UniAP是首个将这两类策略统一以搜索全局最优方案的工作。实验结果表明，在四种类Transformer模型上，UniAP在吞吐量上较现有最优方法提升高达1.70倍，策略搜索时间缩减最高达16倍。