Recent advances in vision-language pre-training have pushed the state-of-the-art on various vision-language tasks, making machines more capable of multi-modal writing (image-to-text generation) and painting (text-to-image generation). However, few studies investigate if these two essential capabilities can be learned together and boost each other, making a versatile and powerful multi-modal foundation model. In this work, we disclose the potential of symmetric generative vision-language pre-training in learning to write and paint concurrently, and propose a new unified modal model, named DaVinci, trained with prefix language modeling and prefix image modeling, a simple generative self-supervised objective on image-text pairs. Thanks to the proposed prefix multi-modal modeling framework, DaVinci is simple to train, scalable to huge data, adaptable to both writing and painting tasks, and also strong on other vision, text, and multi-modal understanding tasks. DaVinci achieves competitive performance on a wide range of 27 generation/understanding tasks and demonstrates the superiority of combining vision/language generative pre-training. Furthermore, we carefully benchmark the performance of different vision-language pre-training objectives on different scales of pre-training datasets on a heterogeneous and broad distribution coverage. Our results demonstrate the potential of exploiting self-supervision in both language and vision inputs, and establish new, stronger baselines for future comparisons at different data scales. The code and pre-trained models are available at https://github.com/shizhediao/DaVinci.

翻译：近期视觉-语言预训练的进展推动了各类视觉-语言任务的技术前沿，使机器在跨模态写作（图像到文本生成）与绘画（文本到图像生成）方面更具能力。然而，鲜有研究探讨这两种核心能力能否协同学习并相互增益，从而构建通用且强大的多模态基础模型。本文揭示了对称式生成视觉-语言预训练在同时学习写作与绘画中的潜力，并提出一种新型统一模态模型——DaVinci。该模型通过前缀语言建模与前缀图像建模进行训练，这是一种基于图像-文本对的简单生成式自监督目标函数。得益于所提出的前缀多模态建模框架，DaVinci具备训练简单、可扩展至海量数据、适配写作与绘画任务，并在其他视觉、文本及多模态理解任务中表现优异等特性。在涵盖27项生成/理解任务的广泛评测中，DaVinci取得了具有竞争力的性能，充分验证了融合视觉/语言生成式预训练的优势。此外，我们基于异构且分布广泛的预训练数据集，系统评估了不同视觉-语言预训练目标在不同数据规模下的表现。实验结果表明，利用语言与视觉输入中的自监督信息具有显著潜力，并为未来不同数据规模下的性能对比建立了新的、更强的基线。代码及预训练模型已开源于https://github.com/shizhediao/DaVinci。