Different conflicting optimization criteria arise naturally in various Deep Learning scenarios. These can address different main tasks (i.e., in the setting of Multi-Task Learning), but also main and secondary tasks such as loss minimization versus sparsity. The usual approach is a simple weighting of the criteria, which formally only works in the convex setting. In this paper, we present a Multi-Objective Optimization algorithm using a modified Weighted Chebyshev scalarization for training Deep Neural Networks (DNNs) with respect to several tasks. By employing this scalarization technique, the algorithm can identify all optimal solutions of the original problem while reducing its complexity to a sequence of single-objective problems. The simplified problems are then solved using an Augmented Lagrangian method, enabling the use of popular optimization techniques such as Adam and Stochastic Gradient Descent, while efficaciously handling constraints. Our work aims to address the (economical and also ecological) sustainability issue of DNN models, with a particular focus on Deep Multi-Task models, which are typically designed with a very large number of weights to perform equally well on multiple tasks. Through experiments conducted on two Machine Learning datasets, we demonstrate the possibility of adaptively sparsifying the model during training without significantly impacting its performance, if we are willing to apply task-specific adaptations to the network weights. The code is available at https://github.com/salomonhotegni/MDMTN

翻译：不同的冲突优化准则自然出现在各种深度学习场景中。这些准则既可以处理不同的主要任务（即多任务学习场景），也可以处理主要任务与次要任务（如损失最小化与稀疏性）。常规方法是简单的加权求和，但这仅在凸优化场景下形式有效。本文提出一种基于修正加权切比雪夫标量化的多目标优化算法，用于训练面向多任务的深度神经网络。通过采用该标量化技术，算法能够识别原始问题的所有最优解，同时将其复杂度降低为一系列单目标问题。简化后的问题采用增广拉格朗日方法求解，使得Adam和随机梯度下降等主流优化技术既能高效处理约束，又能有效应用。本研究旨在解决深度神经网络模型的（经济与生态）可持续性问题，特别关注通常具有大量权重的深度多任务模型——这类模型需在所有任务上表现均衡。通过两个机器学习数据集的实验表明，若愿意对网络权重进行任务特定调整，可在训练过程中自适应地对模型进行稀疏化处理而不显著影响其性能。代码开源地址：https://github.com/salomonhotegni/MDMTN