As a natural language assistant, ChatGPT is capable of performing various tasks, including but not limited to article generation, code completion, and data analysis. Furthermore, ChatGPT has consistently demonstrated a remarkable level of accuracy and reliability in terms of content evaluation, exhibiting the capability of mimicking human preferences. To further explore ChatGPT's potential in this regard, a study is conducted to assess its ability to rank content. In order to do so, a test set consisting of prompts is created, covering a wide range of use cases, and five models are utilized to generate corresponding responses. ChatGPT is then instructed to rank the responses generated by these models. The results on the test set show that ChatGPT's ranking preferences are consistent with human to a certain extent. This preliminary experimental finding implies that ChatGPT's zero-shot ranking capability could be used to reduce annotation pressure in a number of ranking tasks.

翻译：作为自然语言助手，ChatGPT能够执行多种任务，包括但不限于文章生成、代码补全和数据分析。此外，ChatGPT在内容评估方面始终展现出显著的准确性和可靠性，表现出模仿人类偏好的能力。为进一步探索ChatGPT在此方面的潜力，本研究评估了其内容排序能力。为此，构建了一个涵盖广泛用例的提示测试集，并利用五个模型生成对应响应。随后，指令ChatGPT对上述模型生成的响应进行排序。测试集结果显示，ChatGPT的排序偏好在一定程度上与人类一致。这一初步实验发现表明，ChatGPT的零样本排序能力可用于减轻多项排序任务中的标注压力。