The development of language models have moved from encoder-decoder to decoder-only designs. In addition, the common knowledge has it that the two most popular multimodal tasks, the generative and contrastive tasks, tend to conflict with one another, are hard to accommodate in one architecture, and further need complex adaptations for downstream tasks. We propose a novel paradigm of training with a decoder-only model for multimodal tasks, which is surprisingly effective in jointly learning of these disparate vision-language tasks. This is done with a simple model, called MaMMUT. It consists of a single vision encoder and a text decoder, and is able to accommodate contrastive and generative learning by a novel two-pass approach on the text decoder. We demonstrate that joint training of these diverse-objective tasks is simple, effective, and maximizes the weight-sharing of the model. Furthermore, the same architecture enables straightforward extensions to open-vocabulary object detection and video-language tasks. The model tackles a diverse range of tasks, while being modest in capacity. Our model achieves the SOTA on image-text and text-image retrieval, video question answering and open-vocabulary detection tasks, outperforming much larger and more extensively trained foundational models. It shows competitive results on VQA and Video Captioning, especially considering its size. Ablations confirm the flexibility and advantages of our approach.

翻译：语言模型的发展已从编码器-解码器架构转向仅解码器设计。此外，普遍认知中，两种最流行的多模态任务——生成式任务与对比式任务——往往相互冲突，难以在同一架构中共存，且需要为下游任务进行复杂适配。我们提出一种新颖的训练范式，采用仅解码器模型处理多模态任务，该范式在联合学习这些截然不同的视觉-语言任务时表现出惊人的有效性。这一成果通过名为MaMMUT的简洁模型实现，该模型由单一视觉编码器和文本解码器组成，通过对文本解码器采用新颖的双通道方法，能够同时容纳对比式学习与生成式学习。我们证明，联合训练这些目标各异的任务不仅简单高效，且能最大化模型参数共享。此外，相同架构可直接扩展至开放词汇目标检测和视频-语言任务。该模型在保持适中容量的同时，可处理多种任务。在图像-文本与文本-图像检索、视频问答及开放词汇检测任务中，我们的模型达到了最先进水平，超越了参数规模更大、训练更充分的基础模型。在VQA和视频描述生成任务上，该模型也展现出具有竞争力的表现，尤其是考虑到其模型规模。消融研究进一步验证了我们方法的灵活性与优势。