Recently, the remarkable advance of the Large Language Model (LLM) has inspired researchers to transfer its extraordinary reasoning capability to both vision and language data. However, the prevailing approaches primarily regard the visual input as a prompt and focus exclusively on optimizing the text generation process conditioned upon vision content by a frozen LLM. Such an inequitable treatment of vision and language heavily constrains the model's potential. In this paper, we break through this limitation by representing both vision and language in a unified form. Specifically, we introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language that LLM can read. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. Coped with this tokenizer, the presented foundation model called LaVIT can handle both image and text indiscriminately under the same generative learning paradigm. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously. Extensive experiments further showcase that it outperforms the existing models by a large margin on massive vision-language tasks. Our code and models will be available at https://github.com/jy0205/LaVIT.

翻译：近期，大语言模型的显著进步激励研究者将其卓越的推理能力迁移至视觉与语言数据。然而，现有主流方法主要将视觉输入视为提示，专注于优化由冻结的LLM基于视觉内容调节的文本生成过程。这种对视觉与语言的不平等处理严重限制了模型的潜力。本文通过将视觉与语言统一表征突破了这一局限。具体而言，我们引入精心设计的视觉分词器，将非语言图像转化为LLM可读的离散标记序列（类似外语）。所生成的视觉标记既包含具有词语意义的高层语义，又支持根据图像变化的动态序列长度。配合该分词器，所提出的基础模型LaVIT能够在统一生成学习范式下无差别地处理图像与文本。这种统一性使LaVIT成为能够同时理解并生成多模态内容的强大通用接口。大量实验进一步表明，该模型在多项视觉-语言任务上以显著优势超越现有模型。我们的代码与模型将在https://github.com/jy0205/LaVIT公开。