Despite Multi-modal Large Language Models (MM-LLMs) have made exciting strides recently, they are still struggling to efficiently model the interactions among multi-modal inputs and the generation in non-textual modalities. In this work, we propose TEAL (Tokenize and Embed ALl)}, an approach to treat the input from any modality as a token sequence and learn a joint embedding space for all modalities. Specifically, for the input from any modality, TEAL first discretizes it into a token sequence with the off-the-shelf tokenizer and embeds the token sequence into a joint embedding space with a learnable embedding matrix. MM-LLMs just need to predict the multi-modal tokens autoregressively as the textual LLMs do. Finally, the corresponding de-tokenizer is applied to generate the output in each modality based on the predicted token sequence. With the joint embedding space, TEAL enables the frozen LLMs to perform both understanding and generation tasks involving non-textual modalities, such as image and audio. Thus, the textual LLM can just work as an interface and maintain its high performance in textual understanding and generation. Experiments show that TEAL achieves substantial improvements in multi-modal understanding, and implements a simple scheme for multi-modal generations.

翻译：尽管多模态大语言模型（MM-LLMs）近期取得了令人瞩目的进展，但在高效建模多模态输入间的交互以及非文本模态的生成任务方面仍面临挑战。本文提出TEAL（Tokenize and Embed ALl）方法，将任意模态的输入视作符号序列，并为所有模态学习联合嵌入空间。具体而言，对于任意模态的输入，TEAL首先利用现成的分词器将其离散化为符号序列，再通过可学习的嵌入矩阵将该符号序列映射至联合嵌入空间。多模态大语言模型仅需像文本大语言模型那样自回归预测多模态符号，最后基于预测的符号序列，通过相应的解分词器生成对应模态的输出。借助联合嵌入空间，TEAL使冻结参数的大语言模型能够执行涉及图像、音频等非文本模态的理解与生成任务。由此，文本大语言模型可仅作为接口运行，同时保持其在文本理解与生成方面的卓越性能。实验表明，TEAL在多模态理解任务中取得显著提升，并实现了简洁的多模态生成方案。