The debut of ChatGPT has recently attracted the attention of the natural language processing (NLP) community and beyond. Existing studies have demonstrated that ChatGPT shows significant improvement in a range of downstream NLP tasks, but the capabilities and limitations of ChatGPT in terms of recommendations remain unclear. In this study, we aim to conduct an empirical analysis of ChatGPT's recommendation ability from an Information Retrieval (IR) perspective, including point-wise, pair-wise, and list-wise ranking. To achieve this goal, we re-formulate the above three recommendation policies into a domain-specific prompt format. Through extensive experiments on four datasets from different domains, we demonstrate that ChatGPT outperforms other large language models across all three ranking policies. Based on the analysis of unit cost improvements, we identify that ChatGPT with list-wise ranking achieves the best trade-off between cost and performance compared to point-wise and pair-wise ranking. Moreover, ChatGPT shows the potential for mitigating the cold start problem and interpretable recommendation. To facilitate further explorations in this area, the full code and detailed original results are open-sourced at https://github.com/rainym00d/LLM4RS.

翻译：ChatGPT的首次亮相近来引起了自然语言处理（NLP）社区及其他领域的关注。现有研究表明，ChatGPT在一系列下游NLP任务中表现出显著的改进，但在推荐方面，ChatGPT的能力和局限性仍不明确。在本研究中，我们旨在从信息检索（IR）的视角对ChatGPT的推荐能力进行实证分析，包括点级、对级和列表级排序。为实现这一目标，我们将上述三种推荐策略重新表述为领域特定的提示格式。通过对来自不同领域的四个数据集进行大量实验，我们证明ChatGPT在所有三种排序策略上均优于其他大型语言模型。基于单位成本改进的分析，我们确定与点级和对级排序相比，采用列表级排序的ChatGPT在成本与性能之间实现了最佳权衡。此外，ChatGPT在缓解冷启动问题和可解释推荐方面显示出潜力。为促进该领域的进一步探索，完整代码和详细原始结果已在https://github.com/rainym00d/LLM4RS上开源。