We study few-shot reranking for multi-hop QA with open-domain questions. To alleviate the need for a large number of labeled question-document pairs for retriever training, we propose PromptRank, which relies on large language models prompting for multi-hop path reranking. PromptRank first constructs an instruction-based prompt that includes a candidate document path and then computes the relevance score between a given question and the path based on the conditional likelihood of the question given the path prompt according to a language model. PromptRank yields strong retrieval performance on HotpotQA with only 128 training examples compared to state-of-the-art methods trained on thousands of examples -- 73.6 recall@10 by PromptRank vs. 77.8 by PathRetriever and 77.5 by multi-hop dense retrieval. Code available at https://github.com/mukhal/PromptRank

翻译：我们研究了面向开放域问题的多跳问答中的少样本重排序方法。为减少检索器训练所需的大量标注问答对，我们提出PromptRank方法，该方法利用大语言模型提示进行多跳路径重排序。PromptRank首先构建包含候选文档路径的指令型提示，然后根据给定问题在路径提示条件下的条件似然性（基于语言模型）计算问题与路径之间的相关性得分。在HotpotQA数据集上，PromptRank仅使用128个训练样本即展现出强大的检索性能，与使用数千样本训练的先进方法相比——PromptRank的召回率@10为73.6，而PathRetriever为77.8，多跳稠密检索为77.5。代码已开源：https://github.com/mukhal/PromptRank