Instruction tuning, a new learning paradigm that fine-tunes pre-trained language models on tasks specified through instructions, has shown promising zero-shot performance on various natural language processing tasks. However, it's still not explored for vision and multimodal tasks. In this work, we introduce MultiInstruct, the first multimodal instruction tuning benchmark dataset that consists of 47 diverse multimodal tasks covering 11 broad categories. Each task is designed at least with 5,000 instances (input-out pairs) from existing open-source datasets and 5 expert-written instructions. We take OFA as the base pre-trained model for multimodal instruction tuning, and to improve its performance, we explore multiple transfer learning strategies to leverage the large-scale Natural Instructions dataset. Experimental results demonstrate its strong zero-shot performance on various unseen multimodal tasks and the benefit of transfer learning from text-only instructions. We also design a new evaluation metric: Sensitivity, to evaluate how sensitive the model is to the variety of instructions. Our results indicate that the model is less sensitive to the varying instructions after finetuning on a diverse set of tasks and instructions for each task.

翻译：指令调优作为一种新型学习范式，通过在基于指令定义的任务上对预训练语言模型进行微调，已在多种自然语言处理任务中展现出良好的零样本性能。然而，该范式在视觉及多模态任务中尚未得到探索。本研究提出首个多模态指令调优基准数据集MultiInstruct，包含涵盖11大类的47项多样化多模态任务。每项任务均基于现有开源数据集设计至少5000组实例（输入-输出对）及5条专家撰写的指令。我们采用OFA作为多模态指令调优的基座预训练模型，并通过探索多种迁移学习策略以充分利用大规模自然指令数据集。实验结果表明，该模型在各类未见多模态任务上展现出强大的零样本性能，且从纯文本指令进行迁移学习具有显著优势。此外，我们设计了新的评估指标“敏感性”，用于衡量模型对不同指令的敏感程度。结果显示，经过多样化任务及每项任务的多条指令微调后，模型对指令变体的敏感性有所降低。