Far beyond learning long-range interactions of natural language, transformers are becoming the de-facto standard for many vision tasks with their power and scalabilty. Especially with cross-modal tasks between image and text, vector quantized variational autoencoders (VQ-VAEs) are widely used to make a raw RGB image into a sequence of feature vectors. To better leverage the correlation between image and text, we propose L-Verse, a novel architecture consisting of feature-augmented variational autoencoder (AugVAE) and bidirectional auto-regressive transformer (BiART) for text-to-image and image-to-text generation. Our AugVAE shows the state-of-the-art reconstruction performance on ImageNet1K validation set, along with the robustness to unseen images in the wild. Unlike other models, BiART can distinguish between image (or text) as a conditional reference and a generation target. L-Verse can be directly used for image-to-text or text-to-image generation tasks without any finetuning or extra object detection frameworks. In quantitative and qualitative experiments, L-Verse shows impressive results against previous methods in both image-to-text and text-to-image generation on MS-COCO Captions. We furthermore assess the scalability of L-Verse architecture on Conceptual Captions and present the initial results of bidirectional vision-language representation learning on general domain. Codes available at: https://github.com/tgisaturday/L-Verse

翻译：除了学习自然语言的长距离互动外,变压器正在成为许多视觉任务及其功率和变压器的脱形标准。 特别是在图像和文本之间的交叉模式任务中, 矢量量化变异自动读数器( VQ- VAEs) 被广泛用于将原始 RGB 图像转化为一系列特性矢量。 为了更好地利用图像和文字之间的关联, 我们提议使用L- Verse, 一种由功能强化变异自动变压器( AugVAE) 和双向自动递增变变压器( BiART) 组成的新结构。 用于文本到图像和文本的双向自动递增变变变变变变变变变器( BiARTR ) 。 我们的AGOGVAEE在图像Net1K 验证器上展示了最先进的重建性表现, 加上对野外图像的坚固性。 与其他模型不同, BiART 可以将图像( 或文字) 作为双向生成的参考和生成目标目标。 L- Versecom 直接用于图像到文字的图像的图像的图像的图像的图像- tal- tal- imal- real- imal- im- lade- 和 ladeal- lade- im- s- ladeal- tabal- ladeal- s- la la la la 和 lab- s- s- s- la- s- s- sal- sal- sal- lab- sal- lade- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- ad- ad- sal- sal- sal- ad- ad- ad- ad- sal- sal- sal- sal- sal- sal- lad- sal- sal- sal- sal- sal- sal- sal- sal- sal- sal- seral- seral- seral- sal-