The development of language models have moved from encoder-decoder to decoder-only designs. In addition, we observe that the two most popular multimodal tasks, the generative and contrastive tasks, are nontrivial to accommodate in one architecture, and further need adaptations for downstream tasks. We propose a novel paradigm of training with a decoder-only model for multimodal tasks, which is surprisingly effective in jointly learning of these disparate vision-language tasks. This is done with a simple model, called MaMMUT. It consists of a single vision encoder and a text decoder, and is able to accommodate contrastive and generative learning by a novel two-pass approach on the text decoder. We demonstrate that joint learning of these diverse objectives is simple, effective, and maximizes the weight-sharing of the model across these tasks. Furthermore, the same architecture enables straightforward extensions to open-vocabulary object detection and video-language tasks. The model tackles a diverse range of tasks, while being modest in capacity. Our model achieves the state of the art on image-text and text-image retrieval, video question answering and open-vocabulary detection tasks, outperforming much larger and more extensively trained foundational models. It shows very competitive results on VQA and Video Captioning, especially considering its capacity. Ablations confirm the flexibility and advantages of our approach.

翻译：语言模型的发展已从编码器-解码器结构转向纯解码器设计。此外，我们观察到最流行的两种多模态任务——生成式任务与对比式任务——难以在同一架构中兼容，且需针对下游任务进行适配。本文提出了一种新颖的训练范式，采用纯解码器模型处理多模态任务，令人惊讶地实现了对这类差异显著的视觉-语言任务的联合学习。我们通过名为MaMMUT的简易模型完成这一目标。该模型仅包含单一视觉编码器和文本解码器，并通过对文本解码器采用新颖的双通道方法，能够同时容纳对比学习与生成学习。实验证明，这些不同目标的联合学习简单而高效，并最大化了模型跨任务的权重共享。此外，同一架构还可直接拓展至开放词汇目标检测和视频语言任务。该模型在容量适中的情况下处理了多种任务，在图像-文本与文本-图像检索、视频问答及开放词汇检测任务中达到业界领先水平，性能优于更大规模、更广泛训练的基础模型。在VQA和视频字幕生成任务中，考虑到其模型容量，该方法展现出极具竞争力的结果。消融实验进一步证实了我们方法的灵活性与优势。