Instruction-tuned large language models have revolutionized natural language processing and have shown great potential in applications such as conversational agents. These models, such as GPT-4, can not only master language but also solve complex tasks in areas like mathematics, coding, medicine, and law. Despite their impressive capabilities, there is still a lack of comprehensive understanding regarding their full potential, primarily due to the black-box nature of many models and the absence of holistic evaluation studies. To address these challenges, we present INSTRUCTEVAL, a more comprehensive evaluation suite designed specifically for instruction-tuned large language models. Unlike previous works, our evaluation involves a rigorous assessment of models based on problem-solving, writing ability, and alignment to human values. We take a holistic approach to analyze various factors affecting model performance, including the pretraining foundation, instruction-tuning data, and training methods. Our findings reveal that the quality of instruction data is the most crucial factor in scaling model performance. While open-source models demonstrate impressive writing abilities, there is substantial room for improvement in problem-solving and alignment. We are encouraged by the rapid development of models by the open-source community, but we also highlight the need for rigorous evaluation to support claims made about these models. Through INSTRUCTEVAL, we aim to foster a deeper understanding of instruction-tuned models and advancements in their capabilities. INSTRUCTEVAL is publicly available at https://github.com/declare-lab/instruct-eval.

翻译：指令微调大语言模型已彻底变革自然语言处理，并在对话智能体等应用中展现出巨大潜力。诸如GPT-4等模型不仅能掌握语言，还能解决数学、编程、医学和法律等领域的复杂任务。尽管能力令人瞩目，但由于许多模型的黑箱特性及缺乏整体评估研究，对其全部潜力仍缺乏全面理解。为应对这些挑战，我们提出INSTRUCTEVAL——一个专为指令微调大语言模型设计的更全面评估套件。与以往工作不同，我们的评估基于问题解决能力、写作能力及与人类价值观一致性对模型进行严格评测。我们采取整体方法分析影响模型性能的多种因素，包括预训练基础、指令微调数据及训练方法。研究结果表明，指令数据质量是扩展模型性能的最关键因素。尽管开源模型展现出令人印象深刻的写作能力，但在问题解决与价值观对齐方面仍有显著提升空间。我们对开源社区的快速模型开发感到鼓舞，但同时也强调需通过严格评估来支撑关于这些模型的主张。通过INSTRUCTEVAL，我们致力于加深对指令微调模型的理解并推动其能力进步。INSTRUCTEVAL已在https://github.com/declare-lab/instruct-eval公开发布。