This paper proposes a GeneraLIst encoder-Decoder (GLID) pre-training method for better handling various downstream computer vision tasks. While self-supervised pre-training approaches, e.g., Masked Autoencoder, have shown success in transfer learning, task-specific sub-architectures are still required to be appended for different downstream tasks, which cannot enjoy the benefits of large-scale pre-training. GLID overcomes this challenge by allowing the pre-trained generalist encoder-decoder to be fine-tuned on various vision tasks with minimal task-specific architecture modifications. In the GLID training scheme, pre-training pretext task and other downstream tasks are modeled as "query-to-answer" problems, including the pre-training pretext task and other downstream tasks. We pre-train a task-agnostic encoder-decoder with query-mask pairs. During fine-tuning, GLID maintains the pre-trained encoder-decoder and queries, only replacing the topmost linear transformation layer with task-specific linear heads. This minimizes the pretrain-finetune architecture inconsistency and enables the pre-trained model to better adapt to downstream tasks. GLID achieves competitive performance on various vision tasks, including object detection, image segmentation, pose estimation, and depth estimation, outperforming or matching specialist models such as Mask2Former, DETR, ViTPose, and BinsFormer.

翻译：本文提出了一种通用编码器-解码器（GLID）预训练方法，旨在更好地处理各类下游计算机视觉任务。尽管自监督预训练方法（如掩码自编码器）已在迁移学习中取得成功，但不同下游任务仍需附加特定任务的子架构，这无法享受大规模预训练的优势。GLID通过允许预训练的通用编码器-解码器在仅进行最小任务特定架构修改的情况下微调于各类视觉任务，克服了这一挑战。在GLID训练框架中，预训练代理任务及其他下游任务均被建模为“查询-答案”问题。我们使用查询-掩码对预训练一个任务无关的编码器-解码器。微调时，GLID保持预训练的编码器-解码器与查询不变，仅将顶层线性变换层替换为任务特定的线性头部，从而最小化预训练与微调之间的架构不一致性，使预训练模型能更好地适应下游任务。GLID在目标检测、图像分割、姿态估计与深度估计等多项视觉任务中取得了竞争性性能，超越或持平于Mask2Former、DETR、ViTPose和BinsFormer等专用模型。