Recent studies show that large language models (LLMs) can be instructed to effectively perform zero-shot passage re-ranking, in which the results of a first stage retrieval method, such as BM25, are rated and reordered to improve relevance. In this work, we improve LLM-based re-ranking by algorithmically selecting few-shot demonstrations to include in the prompt. Our analysis investigates the conditions where demonstrations are most helpful, and shows that adding even one demonstration is significantly beneficial. We propose a novel demonstration selection strategy based on difficulty rather than the commonly used semantic similarity. Furthermore, we find that demonstrations helpful for ranking are also effective at question generation. We hope our work will spur more principled research into question generation and passage ranking.

翻译：摘要：近期研究表明，大型语言模型（LLM）可通过指令有效执行零样本段落重排序，即对BM25等首阶段检索方法的结果进行评分并重新排序以提升相关性。本研究通过算法选取少量样例提示来改进基于LLM的重排序。我们的分析探究了样例最有效的条件，并表明即使仅添加一个样例也能显著提升效果。我们提出一种基于难度而非常用语义相似度的新颖样例选择策略。此外，我们发现有助于排序的样例同样能有效生成问题。期望本研究能推动问题生成与段落排序领域更规范的研究进展。