Large language models (LLMs) have demonstrated significant universal capabilities as few/zero-shot learners in various tasks due to their pre-training on vast amounts of text data, as exemplified by GPT-3, which boosted to InstrctGPT and ChatGPT, effectively following natural language instructions to accomplish real-world tasks. In this paper, we propose to introduce instruction tuning into multi-modal models, motivated by the Flamingo model's upstream interleaved format pretraining dataset. We adopt a similar approach to construct our MultI-Modal In-Context Instruction Tuning (MIMIC-IT) dataset. We then introduce Otter, a multi-modal model based on OpenFlamingo (open-sourced version of DeepMind's Flamingo), trained on MIMIC-IT and showcasing improved instruction-following ability and in-context learning. We also optimize OpenFlamingo's implementation for researchers, democratizing the required training resources from 1$\times$ A100 GPU to 4$\times$ RTX-3090 GPUs, and integrate both OpenFlamingo and Otter into Huggingface Transformers for more researchers to incorporate the models into their customized training and inference pipelines.

翻译：大型语言模型（LLMs）因其在海量文本数据上的预训练，已在各种任务中展现出显著的通用能力，成为少样本/零样本学习器，例如GPT-3，进而推动InstructGPT和ChatGPT的发展，能够有效遵循自然语言指令完成现实世界任务。本文受Flamingo模型上游交错格式预训练数据集的启发，提出将指令调优引入多模态模型。我们采用类似方法构建了多模态上下文指令调优（MIMIC-IT）数据集。随后，我们推出Otter——一种基于OpenFlamingo（DeepMind Flamingo的开源版本）的多模态模型，在MIMIC-IT上训练并展示了更强的指令遵循能力和上下文学习能力。此外，我们为研究人员优化了OpenFlamingo的实现，将所需训练资源从1× A100 GPU降低至4× RTX-3090 GPU，并将OpenFlamingo和Otter集成到Huggingface Transformers中，以便更多研究人员将这些模型纳入其定制化的训练和推理流程。