Despite having impressive vision-language (VL) pretraining with BERT-based encoder for VL understanding, the pretraining of a universal encoder-decoder for both VL understanding and generation remains challenging. The difficulty originates from the inherently different peculiarities of the two disciplines, e.g., VL understanding tasks capitalize on the unrestricted message passing across modalities, while generation tasks only employ visual-to-textual message passing. In this paper, we start with a two-stream decoupled design of encoder-decoder structure, in which two decoupled cross-modal encoder and decoder are involved to separately perform each type of proxy tasks, for simultaneous VL understanding and generation pretraining. Moreover, for VL pretraining, the dominant way is to replace some input visual/word tokens with mask tokens and enforce the multi-modal encoder/decoder to reconstruct the original tokens, but no mask token is involved when fine-tuning on downstream tasks. As an alternative, we propose a primary scheduled sampling strategy that elegantly mitigates such discrepancy via pretraining encoder-decoder in a two-pass manner. Extensive experiments demonstrate the compelling generalizability of our pretrained encoder-decoder by fine-tuning on four VL understanding and generation downstream tasks. Source code is available at \url{https://github.com/YehLi/TDEN}.

翻译：尽管对VL的理解和生成都具有令人印象深刻的视觉语言(VL)预培训,但对于VL的理解和生成而言,通用编码器的通用编码器的预培训仍然具有挑战性。难点来自两个学科固有的不同特性,例如VL理解任务利用跨模式传递的不受限制的信息,而生成任务只采用视觉到文字传递信息。在本文件中,我们首先从双流解码器-代碼器结构的编码器设计开始,其中涉及两个分couped的跨式编码器和编码器分别执行每一种代名任务,同时进行VL理解和生成前训练。此外,对于VLL前训练,主要的方式是用面具取代某些输入的视觉/文字符号,执行多式编码器/代码,以重建原有的标识,但在对下游任务进行微调时,没有使用遮码。作为替代办法,我们提出一个主要排定的主要取样战略,通过前期的版本,通过升级的版本的版本/版本的版本,将这种差异性地减轻。