Foundation language models obtain the instruction-following ability through supervised fine-tuning (SFT). Diversity and complexity are considered critical factors of a successful SFT dataset, while their definitions remain obscure and lack quantitative analyses. In this work, we propose InsTag, an open-set fine-grained tagger, to tag samples within SFT datasets based on semantics and intentions and define instruction diversity and complexity regarding tags. We obtain 6.6K tags to describe comprehensive user queries. Then we analyze popular open-sourced SFT datasets and find that the model ability grows with more diverse and complex data. Based on this observation, we propose a data selector based on InsTag to select 6K diverse and complex samples from open-source datasets and fine-tune models on InsTag-selected data. The resulting models, TagLM, outperform open-source models based on considerably larger SFT data evaluated by MT-Bench, echoing the importance of query diversity and complexity. We open-source InsTag in https://github.com/OFA-Sys/InsTag.

翻译：摘要：基础语言模型通过监督微调获得指令遵循能力。多样性和复杂性被认为是成功监督微调数据集的关键因素，但其定义仍模糊且缺乏定量分析。本文提出InsTag——一种开放集细粒度标注器——基于语义和意图对监督微调数据集中的样本进行标注，并基于标签定义指令多样性和复杂性。我们获得了6.6K个标签来描述全面的用户查询。随后分析主流开源监督微调数据集发现，模型能力随数据多样性和复杂性的提升而增强。基于此发现，我们提出基于InsTag的数据选择器，从开源数据集中选取6K个多样且复杂的样本，并在InsTag选择的数据上微调模型。由此得到的模型TagLM在MT-Bench评估中优于使用更大规模监督微调数据的开源模型，印证了查询多样性与复杂性的重要性。我们在https://github.com/OFA-Sys/InsTag开源InsTag。