We present a vision and language model named MultiModal-GPT to conduct multi-round dialogue with humans. MultiModal-GPT can follow various instructions from humans, such as generating a detailed caption, counting the number of interested objects, and answering general questions from users. MultiModal-GPT is parameter-efficiently fine-tuned from OpenFlamingo, with Low-rank Adapter (LoRA) added both in the cross-attention part and the self-attention part of the language model. We first construct instruction templates with vision and language data for multi-modality instruction tuning to make the model understand and follow human instructions. We find the quality of training data is vital for the dialogue performance, where few data containing short answers can lead the model to respond shortly to any instructions. To further enhance the ability to chat with humans of the MultiModal-GPT, we utilize language-only instruction-following data to train the MultiModal-GPT jointly. The joint training of language-only and visual-language instructions with the \emph{same} instruction template effectively improves dialogue performance. Various demos show the ability of continuous dialogue of MultiModal-GPT with humans. Code and demo are at https://github.com/open-mmlab/Multimodal-GPT

翻译：我们提出了一种名为MultiModal-GPT的视觉语言模型，用于与人类进行多轮对话。MultiModal-GPT能够遵循人类的多种指令，例如生成详细描述、统计感兴趣物体的数量以及回答用户的常规问题。该模型通过参数高效微调方法从OpenFlamingo基础上进行优化，在语言模型的交叉注意力和自注意力部分均引入了低秩适配器（LoRA）。我们首先构建包含视觉与语言数据的指令模板，通过多模态指令微调使模型理解并遵循人类指令。研究发现，训练数据质量对对话性能至关重要——包含简短回答的少量数据会导致模型对任何指令均给出简短回复。为进一步增强MultiModal-GPT的多轮对话能力，我们联合使用纯语言指令遵循数据进行训练。采用相同指令模板的语言指令与视觉语言指令联合训练可有效提升对话性能。多项演示展示了MultiModal-GPT与人类进行持续对话的能力。代码与演示见 https://github.com/open-mmlab/Multimodal-GPT