We present a vision and language model named MultiModal-GPT to conduct multi-round dialogue with humans. MultiModal-GPT can follow various instructions from humans, such as generating a detailed caption, counting the number of interested objects, and answering general questions from users. MultiModal-GPT is parameter-efficiently fine-tuned from OpenFlamingo, with Low-rank Adapter (LoRA) added both in the cross-attention part and the self-attention part of the language model. We first construct instruction templates with vision and language data for multi-modality instruction tuning to make the model understand and follow human instructions. We find the quality of training data is vital for the dialogue performance, where few data containing short answers can lead the model to respond shortly to any instructions. To further enhance the ability to chat with humans of the MultiModal-GPT, we utilize language-only instruction-following data to train the MultiModal-GPT jointly. The joint training of language-only and visual-language instructions with the \emph{same} instruction template effectively improves dialogue performance. Various demos show the ability of continuous dialogue of MultiModal-GPT with humans. Code, dataset, and demo are at https://github.com/open-mmlab/Multimodal-GPT

翻译：我们提出了一种名为多模态GPT（MultiModal-GPT）的视觉与语言模型，用于与人类进行多轮对话。该模型能够遵循人类的各类指令，例如生成详细描述、统计感兴趣对象的数量，以及回答用户的通用问题。MultiModal-GPT通过参数高效微调技术，从OpenFlamingo模型出发，在语言模型的交叉注意力部分和自注意力部分均添加了低秩适配器（LoRA）。我们首先使用视觉与语言数据构建指令模板，进行多模态指令微调，使模型能够理解并遵循人类指令。研究发现，训练数据的质量对对话性能至关重要：包含简短回答的少量数据可能导致模型对所有指令均给出简短回应。为进一步增强MultiModal-GPT的对话能力，我们还利用纯语言指令遵循数据对其进行联合训练。通过采用\emph{相同}指令模板的纯语言与视觉-语言指令联合训练，有效提升了对话性能。多项演示展示了MultiModal-GPT与人类进行持续对话的能力。代码、数据集及演示请访问：https://github.com/open-mmlab/Multimodal-GPT