Text ranking is a critical task in various information retrieval applications, and the recent success of pre-trained language models (PLMs), especially large language models (LLMs), has sparked interest in their application to text ranking. To eliminate the misalignment between PLMs and text ranking, fine-tuning with supervised ranking data has been widely explored. However, previous studies focus mainly on encoder-only and encoder-decoder PLMs, and decoder-only LLM research is still lacking. An exception to this is RankLLaMA, which suggests direct supervised fine-tuning (SFT) to explore LLaMA fully. In our work, we argue that a two-stage progressive paradigm would be more beneficial. First, we suggest continual pre-training (CPT) on LLMs by using a large-scale weakly-supervised corpus. Second, we perform SFT consistent with RankLLaMA, and propose an improved optimization strategy further. Our experimental results on multiple benchmarks demonstrate the superior performance of our method covering both in-domain and out-domain scenarios.

翻译：文本排序是信息检索应用中的关键任务，预训练语言模型（尤其是大语言模型）的最新成功激发了其在文本排序中的应用兴趣。为消除预训练语言模型与文本排序之间的错位现象，基于监督排序数据的微调方法已被广泛探索。然而，现有研究主要聚焦于编码器型和编码器-解码器型预训练语言模型，对解码器型大语言模型的研究仍显不足。其中的例外是RankLLaMA，该方法通过直接监督微调来充分探索LLaMA模型。在本文工作中，我们认为两阶段渐进式范式更具优势：首先，我们通过大规模弱监督语料库对大语言模型进行持续预训练；其次，我们采用与RankLLaMA一致的监督微调方法，并提出改进的优化策略。在多个基准测试上的实验结果表明，我们的方法在领域内和跨领域场景中均展现出优异性能。