We aim at exploiting additional auxiliary labels from an independent (auxiliary) task to boost the primary task performance which we focus on, while preserving a single task inference cost of the primary task. While most existing auxiliary learning methods are optimization-based relying on loss weights/gradients manipulation, our method is architecture-based with a flexible asymmetric structure for the primary and auxiliary tasks, which produces different networks for training and inference. Specifically, starting from two single task networks/branches (each representing a task), we propose a novel method with evolving networks where only primary-to-auxiliary links exist as the cross-task connections after convergence. These connections can be removed during the primary task inference, resulting in a single-task inference cost. We achieve this by formulating a Neural Architecture Search (NAS) problem, where we initialize bi-directional connections in the search space and guide the NAS optimization converging to an architecture with only the single-side primary-to-auxiliary connections. Moreover, our method can be incorporated with optimization-based auxiliary learning approaches. Extensive experiments with six tasks on NYU v2, CityScapes, and Taskonomy datasets using VGG, ResNet, and ViT backbones validate the promising performance. The codes are available at https://github.com/ethanygao/Aux-NAS.

翻译：我们旨在通过独立（辅助）任务中的额外辅助标签来提升我们关注的主任务性能，同时保持主任务在单任务推理时的计算成本。现有的大多数辅助学习方法基于优化策略，依赖于损失权重或梯度操控，而我们的方法基于架构设计，为主任务和辅助任务构建了一种灵活的非对称结构，从而在训练和推理阶段产生不同的网络。具体而言，从两个单任务网络/分支（每个分支代表一个任务）出发，我们提出了一种新颖的渐进式网络演化方法，其中仅在收敛后存在主任务到辅助任务的跨任务连接。这些连接可在主任务推理时被移除，从而实现单任务推理成本。我们通过制定神经架构搜索（NAS）问题来实现这一目标，在搜索空间中初始化双向连接，并引导NAS优化收敛至仅保留单向主任务到辅助任务连接的架构。此外，我们的方法可与基于优化的辅助学习方法相结合。在NYU v2、CityScapes和Taskonomy数据集上，使用VGG、ResNet和ViT骨干网络的六项任务实验验证了其优越性能。相关代码已在https://github.com/ethanygao/Aux-NAS 开源。