We propose a novel zero-shot document ranking approach based on Large Language Models (LLMs): the Setwise prompting approach. Our approach complements existing prompting approaches for LLM-based zero-shot ranking: Pointwise, Pairwise, and Listwise. Through the first-of-its-kind comparative evaluation within a consistent experimental framework and considering factors like model size, token consumption, latency, among others, we show that existing approaches are inherently characterised by trade-offs between effectiveness and efficiency. We find that while Pointwise approaches score high on efficiency, they suffer from poor effectiveness. Conversely, Pairwise approaches demonstrate superior effectiveness but incur high computational overhead. Our Setwise approach, instead, reduces the number of LLM inferences and the amount of prompt token consumption during the ranking procedure, compared to previous methods. This significantly improves the efficiency of LLM-based zero-shot ranking, while also retaining high zero-shot ranking effectiveness. We make our code and results publicly available at \url{https://github.com/ielab/llm-rankers}.

翻译：我们提出了一种基于大语言模型（LLMs）的新型零样本文档排序方法：集合式提示方法。该方法对现有基于LLM的零样本排序提示方法（逐点式、成对式、列表式）形成了补充。通过在一致的实验框架内进行首次比较评估，并综合考虑模型规模、令牌消耗、延迟等因素，我们发现现有方法本质上存在效果与效率之间的权衡。我们发现，逐点式方法虽然效率得分高，但效果较差；相反，成对式方法表现出优异的效果，但计算开销巨大。相比之下，我们的集合式方法在排序过程中减少了LLM推理次数和提示令牌消耗量。这显著提升了基于LLM的零样本排序效率，同时保持了较高的零样本排序效果。我们的代码与结果已公开于 \url{https://github.com/ielab/llm-rankers}。