Multitask learning (MTL) has become prominent for its ability to predict multiple tasks jointly, achieving better per-task performance with fewer parameters than single-task learning. Recently, decoder-focused architectures have significantly improved multitask performance by refining task predictions using features from related tasks. However, most refinement methods struggle to efficiently capture both local and long-range dependencies between task-specific representations and cross-task patterns. In this paper, we introduce the Cross-Task Affinity Learning (CTAL) module, a lightweight framework that enhances task refinement in multitask networks. CTAL effectively captures local and long-range cross-task interactions by optimizing task affinity matrices for parameter-efficient grouped convolutions without concern for information loss. Our results demonstrate state-of-the-art MTL performance for both CNN and transformer backbones, using significantly fewer parameters than single-task learning. Our code is publicly available at https://github.com/Armanfard-Lab/EMA-Net.

翻译：多任务学习（MTL）因其能够联合预测多个任务而变得突出，与单任务学习相比，它以更少的参数实现了更好的单任务性能。最近，以解码器为中心的架构通过利用相关任务的特征细化任务预测，显著提升了多任务性能。然而，大多数细化方法难以有效捕获任务特定表示与跨任务模式之间的局部和长程依赖关系。本文提出跨任务亲和力学习（CTAL）模块，这是一个轻量级框架，用于增强多任务网络中的任务细化。CTAL通过优化任务亲和力矩阵以实现参数高效的分组卷积，有效捕获局部和长程跨任务交互，而无需担心信息丢失。我们的结果表明，无论是基于CNN还是Transformer骨干网络，CTAL均实现了最先进的多任务学习性能，且所用参数显著少于单任务学习。我们的代码公开在https://github.com/Armanfard-Lab/EMA-Net。