Large Language Models (LLMs) have demonstrated a remarkable ability to generalize zero-shot to various language-related tasks. This paper focuses on the study of exploring generative LLMs such as ChatGPT and GPT-4 for relevance ranking in Information Retrieval (IR). Surprisingly, our experiments reveal that properly instructed ChatGPT and GPT-4 can deliver competitive, even superior results than supervised methods on popular IR benchmarks. Notably, GPT-4 outperforms the fully fine-tuned monoT5-3B on MS MARCO by an average of 2.7 nDCG on TREC datasets, an average of 2.3 nDCG on eight BEIR datasets, and an average of 2.7 nDCG on ten low-resource languages Mr.TyDi. Subsequently, we delve into the potential for distilling the ranking capabilities of ChatGPT into a specialized model. Our small specialized model that trained on 10K ChatGPT generated data outperforms monoT5 trained on 400K annotated MS MARCO data on BEIR. The code to reproduce our results is available at www.github.com/sunnweiwei/RankGPT

翻译：大型语言模型（LLMs）在零样本泛化至各类语言相关任务中展现了卓越能力。本文聚焦于探究生成式LLMs（如ChatGPT和GPT-4）在信息检索（IR）中用于相关性排序的应用。令人惊讶的是，我们的实验表明，经过适当指导的ChatGPT和GPT-4能够在主流IR基准测试中取得与监督方法相媲美甚至更优的结果。值得注意的是，GPT-4在MS MARCO上的表现超越了完全微调的monoT5-3B：在TREC数据集上平均nDCG提升2.7，在八个BEIR数据集上平均nDCG提升2.3，在十个低资源语言Mr.TyDi数据集上平均nDCG提升2.7。随后，我们深入探讨了将ChatGPT的排序能力蒸馏至专用模型的潜力。我们使用10K条ChatGPT生成数据训练的小型专用模型，在BEIR上超越了基于400K条标注MS MARCO数据训练的monoT5。复现结果的代码已发布于www.github.com/sunnweiwei/RankGPT。