In a multi-task learning (MTL) setting, a single model is trained to tackle a diverse set of tasks jointly. Despite rapid progress in the field, MTL remains challenging due to optimization issues such as conflicting and dominating gradients. In this work, we propose using a condition number of a linear system of gradients as a stability criterion of an MTL optimization. We theoretically demonstrate that a condition number reflects the aforementioned optimization issues. Accordingly, we present Aligned-MTL, a novel MTL optimization approach based on the proposed criterion, that eliminates instability in the training process by aligning the orthogonal components of the linear system of gradients. While many recent MTL approaches guarantee convergence to a minimum, task trade-offs cannot be specified in advance. In contrast, Aligned-MTL provably converges to an optimal point with pre-defined task-specific weights, which provides more control over the optimization result. Through experiments, we show that the proposed approach consistently improves performance on a diverse set of MTL benchmarks, including semantic and instance segmentation, depth estimation, surface normal estimation, and reinforcement learning. The source code is publicly available at https://github.com/SamsungLabs/MTL .

翻译：在多任务学习场景中，单一模型被训练以联合处理 diverse 任务集合。尽管该领域发展迅速，但由梯度冲突和梯度主导等优化难题引发的挑战依然存在。本文提出以梯度线性系统的条件数作为多任务学习优化的稳定性判据，并从理论上论证条件数能够反映上述优化问题。基于该判据，我们提出 Aligned-MTL——一种新颖的多任务学习优化方法，通过对齐梯度线性系统的正交分量消除训练过程中的不稳定性。近期诸多多任务学习方法虽能保证收敛到极小值点，但无法预先指定任务权衡关系。相比之下，Aligned-MTL 可证明收敛到具有预定义任务特定权重的优化点，从而为优化结果提供更强控制能力。实验表明，所提方法在语义分割、实例分割、深度估计、表面法向估计及强化学习等多任务学习基准测试中持续提升性能。源代码已开源至 https://github.com/SamsungLabs/MTL。