Learning discriminative task-specific features simultaneously for multiple distinct tasks is a fundamental problem in multi-task learning. Recent state-of-the-art models consider directly decoding task-specific features from one shared task-generic feature (e.g., feature from a backbone layer), and utilize carefully designed decoders to produce multi-task features. However, as the input feature is fully shared and each task decoder also shares decoding parameters for different input samples, it leads to a static feature decoding process, producing less discriminative task-specific representations. To tackle this limitation, we propose TaskExpert, a novel multi-task mixture-of-experts model that enables learning multiple representative task-generic feature spaces and decoding task-specific features in a dynamic manner. Specifically, TaskExpert introduces a set of expert networks to decompose the backbone feature into several representative task-generic features. Then, the task-specific features are decoded by using dynamic task-specific gating networks operating on the decomposed task-generic features. Furthermore, to establish long-range modeling of the task-specific representations from different layers of TaskExpert, we design a multi-task feature memory that updates at each layer and acts as an additional feature expert for dynamic task-specific feature decoding. Extensive experiments demonstrate that our TaskExpert clearly outperforms previous best-performing methods on all 9 metrics of two competitive multi-task learning benchmarks for visual scene understanding (i.e., PASCAL-Context and NYUD-v2). Codes and models will be made publicly available at https://github.com/prismformore/Multi-Task-Transformer

翻译：学习同时为多个不同任务判别任务特定特征是多任务学习中的基本问题。当前最优模型通常直接从共享的通用任务特征（例如骨干网络层提取的特征）中解码任务特定特征，并通过精心设计的解码器生成多任务特征。然而，由于输入特征完全共享且每个任务解码器对不同输入样本采用相同的解码参数，导致特征解码过程静态化，难以获得具有判别性的任务特定表征。为解决这一局限，我们提出TaskExpert——一种新型多任务混合专家模型，能够学习多个具有代表性的通用任务特征空间，并以动态方式解码任务特定特征。具体而言，TaskExpert引入一组专家网络将骨干特征分解为多个代表性通用任务特征，随后通过作用于分解后通用任务特征上的动态任务特定门控网络解码任务特定特征。为建立TaskExpert不同层任务特定表征的长程建模能力，我们设计了多任务特征记忆机制，该机制逐层更新并作为额外的特征专家参与动态任务特定特征解码。大量实验表明，在视觉场景理解领域两个主流多任务学习基准（PASCAL-Context与NYUD-v2）的全部9项评估指标上，我们的TaskExpert均显著优于先前最优方法。相关代码与模型将开源至https://github.com/prismformore/Multi-Task-Transformer。