Recent advances in vision-language pre-training have pushed the state-of-the-art on various vision-language tasks, making machines more capable of multi-modal writing (image-to-text generation) and painting (text-to-image generation). However, few studies investigate if these two essential capabilities can be learned together and boost each other, making a versatile and powerful multi-modal foundation model. In this work, we disclose the potential of symmetric generative vision-language pre-training in learning to write and paint concurrently, and propose a new unified modal model, named DaVinci, trained with prefix language modeling and prefix image modeling, a simple generative self-supervised objective on image-text pairs. Thanks to the proposed prefix multi-modal modeling framework, DaVinci is simple to train, scalable to huge data, adaptable to both writing and painting tasks, and also strong on other vision, text, and multi-modal understanding tasks. DaVinci achieves competitive performance on a wide range of 27 generation/understanding tasks and demonstrates the superiority of combining vision/language generative pre-training. Furthermore, we carefully benchmark the performance of different vision-language pre-training objectives on different scales of pre-training datasets on a heterogeneous and broad distribution coverage. Our results demonstrate the potential of exploiting self-supervision in both language and vision inputs, and establish new, stronger baselines for future comparisons at different data scales. The code and pre-trained models are available at https://github.com/shizhediao/DaVinci.

翻译：近年来，视觉-语言预训练的进展推动了各类视觉-语言任务达到新水平，使机器在多模态写作（图像到文本生成）和绘画（文本到图像生成）方面能力更强。然而，少有研究探讨这两种核心能力能否共同学习并相互促进，从而构建一个多功能且强大的多模态基础模型。本文揭示了对称生成式视觉-语言预训练在同时学习写作与绘画方面的潜力，并提出一种新的统一模态模型——DaVinci。该模型通过前缀语言建模与前缀图像建模（一种针对图像-文本对的简单生成式自监督目标）进行训练。得益于所提出的前缀多模态建模框架，DaVinci 训练简单、可扩展至海量数据、适应写作与绘画任务，并在其他视觉、文本及多模态理解任务上表现强劲。DaVinci 在27种生成/理解任务上均取得具有竞争力的性能，证明了视觉/语言生成式预训练相结合的优势。此外，我们系统地在不同规模的预训练数据集上，以异质且广泛覆盖的分布为标准，评测了不同视觉-语言预训练目标的性能。实验结果揭示了在语言与视觉输入中充分利用自监督的潜力，并为未来不同数据规模下的比较建立了更强的新基线。代码与预训练模型已开源至 https://github.com/shizhediao/DaVinci。