Neural document rerankers are extremely effective in terms of accuracy. However, the best models require dedicated hardware for serving, which is costly and often not feasible. To avoid this serving-time requirement, we present a method of capturing up to 86% of the gains of a Transformer cross-attention model with a lexicalized scoring function that only requires 10-6% of the Transformer's FLOPs per document and can be served using commodity CPUs. When combined with a BM25 retriever, this approach matches the quality of a state-of-the art dual encoder retriever, that still requires an accelerator for query encoding. We introduce NAIL (Non-Autoregressive Indexing with Language models) as a model architecture that is compatible with recent encoder-decoder and decoder-only large language models, such as T5, GPT-3 and PaLM. This model architecture can leverage existing pre-trained checkpoints and can be fine-tuned for efficiently constructing document representations that do not require neural processing of queries.

翻译：神经文档重排序器在准确性上极为有效。然而，最佳模型需要专用硬件进行部署，这成本高昂且常常不可行。为避免此部署时需求，我们提出一种方法，利用仅需Transformer每文档计算量的10⁻⁶%、可在商用CPU上运行的词项化评分函数，捕获Transformer交叉注意力模型高达86%的性能提升。当与BM25检索器结合时，该方法可匹配仍需加速器进行查询编码的最先进双编码检索器的质量。我们提出NAIL（基于语言模型的非自回归索引）作为模型架构，该架构兼容T5、GPT-3和PaLM等近期编码器-解码器及仅解码器大型语言模型。此架构可利用现有预训练检查点，并通过微调高效构建无需对查询进行神经处理的文档表示。