An emerging method to cheaply improve a weaker language model is to finetune it on outputs from a stronger model, such as a proprietary system like ChatGPT (e.g., Alpaca, Self-Instruct, and others). This approach looks to cheaply imitate the proprietary model's capabilities using a weaker open-source model. In this work, we critically analyze this approach. We first finetune a series of LMs that imitate ChatGPT using varying base model sizes (1.5B--13B), data sources, and imitation data amounts (0.3M--150M tokens). We then evaluate the models using crowd raters and canonical NLP benchmarks. Initially, we were surprised by the output quality of our imitation models -- they appear far better at following instructions, and crowd workers rate their outputs as competitive with ChatGPT. However, when conducting more targeted automatic evaluations, we find that imitation models close little to none of the gap from the base LM to ChatGPT on tasks that are not heavily supported in the imitation data. We show that these performance discrepancies may slip past human raters because imitation models are adept at mimicking ChatGPT's style but not its factuality. Overall, we conclude that model imitation is a false promise: there exists a substantial capabilities gap between open and closed LMs that, with current methods, can only be bridged using an unwieldy amount of imitation data or by using more capable base LMs. In turn, we argue that the highest leverage action for improving open-source models is to tackle the difficult challenge of developing better base LMs, rather than taking the shortcut of imitating proprietary systems.

翻译：一种新兴的低成本改进较弱语言模型的方法，是通过对更强模型（如ChatGPT等专有系统）的输出进行微调（例如Alpaca、Self-Instruct等）。这种方法看似通过使用较弱的开源模型低成本地模仿专有模型的能力。在本工作中，我们对此方法进行了批判性分析。首先，我们基于不同的基础模型规模（1.5B–13B）、数据源和模仿数据量（0.3M–150M个token），微调了一系列模拟ChatGPT的语言模型。随后，我们利用众包评分员和标准NLP基准对这些模型进行了评估。最初，我们对模仿模型的输出质量感到惊讶——它们在遵循指令方面表现出显著提升，且众包评分员认为其输出与ChatGPT具有竞争力。然而，在进行更有针对性的自动化评估时，我们发现，在模仿数据中支持不足的任务上，模仿模型几乎未缩小基础模型与ChatGPT之间的性能差距。我们证明，这些性能差异可能被人类评分员忽略，因为模仿模型擅长模仿ChatGPT的风格，而非其事实准确性。总体而言，我们得出结论：模型模仿是一个虚假承诺——开源与封闭语言模型之间存在巨大的能力鸿沟，在当前方法下，只有通过使用庞大的模仿数据量或更强大的基础模型才能弥合。据此，我们主张，改进开源模型的最具杠杆作用的行动是应对开发更优基础模型这一艰难挑战，而非采取模仿专有系统的捷径。