Large Language Models (LLMs), primarily trained on text-based datasets, exhibit exceptional proficiencies in understanding and executing complex linguistic instructions via text outputs. However, they falter when requests to generate non-text ones. Concurrently, modality conversion models, such as text-to-image, despite generating high-quality images, suffer from a lack of extensive textual pretraining. As a result, these models are only capable of accommodating specific image descriptions rather than comprehending more complex instructions. To bridge this gap, we propose a novel approach, \methodname, from a modality conversion perspective that evolves a text-based LLM into a multi-modal one. We specifically employ a minimal dataset to instruct LLMs to recognize the intended output modality as directed by the instructions. Consequently, the adapted LLM can effectively summon various off-the-shelf modality conversion models from the model zoos to generate non-text responses. This circumvents the necessity for complicated pretraining that typically requires immense quantities of paired multi-modal data, while simultaneously inheriting the extensive knowledge of LLMs and the ability of high-quality generative models. To evaluate and compare the adapted multi-modal LLM with its traditional counterparts, we have constructed a multi-modal instruction benchmark that solicits diverse modality outputs. The experiment results reveal that, with minimal training, LLMs can be conveniently adapted to comprehend requests for non-text responses, thus achieving higher flexibility in multi-modal scenarios. Code and data will be made available at https://github.com/xinke-wang/SwitchGPT.

翻译：大语言模型（LLMs）主要基于文本数据集训练，在通过文本输出理解和执行复杂语言指令方面展现了卓越能力。然而，当需要生成非文本输出时，它们表现不佳。与此同时，文本到图像等模态转换模型虽然能生成高质量图像，但因缺乏大规模文本预训练而存在局限：这些模型仅能处理特定图像描述，无法理解更复杂的指令。为弥合这一差距，我们提出了一种名为\methodname的新方法，从模态转换角度将基于文本的LLM演化为多模态模型。具体而言，我们利用最小数据集指导LLM识别指令所指向的目标输出模态。经过适配后，LLM能有效调用模型库中各类现成的模态转换模型，生成非文本响应。这避免了通常需要海量配对多模态数据的复杂预训练，同时继承了LLM的广泛知识与高质量生成模型的能力。为评估和比较适配后的多模态LLM与传统模型，我们构建了一个要求多模态输出的指令基准。实验结果表明，通过最小化训练，LLM可便捷地适配以理解非文本响应请求，从而在多模态场景中实现更高灵活性。代码与数据将公开于https://github.com/xinke-wang/SwitchGPT。