Previous multi-task dense prediction studies developed complex pipelines such as multi-modal distillations in multiple stages or searching for task relational contexts for each task. The core insight beyond these methods is to maximize the mutual effects of each task. Inspired by the recent query-based Transformers, we propose a simple pipeline named Multi-Query Transformer (MQTransformer) that is equipped with multiple queries from different tasks to facilitate the reasoning among multiple tasks and simplify the cross-task interaction pipeline. Instead of modeling the dense per-pixel context among different tasks, we seek a task-specific proxy to perform cross-task reasoning via multiple queries where each query encodes the task-related context. The MQTransformer is composed of three key components: shared encoder, cross-task query attention module and shared decoder. We first model each task with a task-relevant query. Then both the task-specific feature output by the feature extractor and the task-relevant query are fed into the shared encoder, thus encoding the task-relevant query from the task-specific feature. Secondly, we design a cross-task query attention module to reason the dependencies among multiple task-relevant queries; this enables the module to only focus on the query-level interaction. Finally, we use a shared decoder to gradually refine the image features with the reasoned query features from different tasks. Extensive experiment results on two dense prediction datasets (NYUD-v2 and PASCAL-Context) show that the proposed method is an effective approach and achieves state-of-the-art results. Code and models are available at https://github.com/yangyangxu0/MQTransformer.

翻译：以往的密集预测多任务研究开发了复杂的流程，例如多阶段的跨模态蒸馏或为每个任务搜索任务间关系上下文。这些方法的核心洞察在于最大化任务间的相互影响。受近期基于查询的Transformer启发，我们提出一种名为多查询Transformer（MQTransformer）的简洁流程，该模型配备来自不同任务的多个查询，以促进多任务间的推理并简化跨任务交互流程。不同于对不同任务间的密集逐像素上下文建模，我们寻求一种任务特定的代理，通过多个查询执行跨任务推理，其中每个查询编码任务相关上下文。MQTransformer由三个关键组件构成：共享编码器、跨任务查询注意力模块和共享解码器。我们首先为每个任务建模一个任务相关查询。随后，特征提取器输出的任务特定特征与任务相关查询共同输入共享编码器，从而从任务特定特征中编码任务相关查询。其次，我们设计跨任务查询注意力模块以推理多个任务相关查询间的依赖关系；这使得该模块仅关注查询层面的交互。最后，我们使用共享解码器，利用来自不同任务的经过推理的查询特征逐步精炼图像特征。在两个密集预测数据集（NYUD-v2和PASCAL-Context）上的大量实验结果表明，所提方法是一种有效方案，并达到了最先进的性能。代码和模型已开源至https://github.com/yangyangxu0/MQTransformer。