Transformer-based document cross-encoder rerankers are a central component of modern information retrieval systems. Despite their success, these models suffer from high computational costs due to processing long query-document sequences at inference time. A known approach to improve efficiency is token compression, which consists of aggregating groups of tokens together in the initial embedding layer, reducing the effective number of tokens, and making the computation faster. While token compression has proven to be successful for bi-encoder retrievers, we empirically observed that this approach may be ineffective for cross-encoder rerankers. In this paper, we propose Layer-wise Token Compression (LTC), which applies adaptive token pooling at intermediate transformer layers. Through extensive ablation studies on MS MARCO passage and document ranking tasks, we demonstrate that compression at middle layers preserves ranking quality while increasing inference QPS by up to 25% for passage ranking and up to 116% for document ranking. We also extend LTC to listwise LLM rerankers and show that the same approach can be easily applied to long-context listwise reranking, where the QPS improvements are even greater. More surprisingly, when applying rerankers trained on short passages to long-document ranking tasks, models trained with compression outperform their uncompressed counterparts, suggesting that compression may act as a beneficial regularizer that encourages length-invariant representations.

翻译：基于Transformer的文档交叉编码器重排序模型是现代信息检索系统的核心组成部分。尽管这些模型取得了成功，但由于在推理时需要处理长查询-文档序列，它们面临着高计算成本的问题。一种已知的提升效率的方法是令牌压缩，该方法在初始嵌入层将令牌分组聚合，减少有效令牌数量，从而加快计算速度。尽管令牌压缩已被证明对双编码器检索器有效，但我们通过实证观察发现，这种方法对于交叉编码器重排序器可能并不有效。本文提出分层令牌压缩（LTC），该方法在中间Transformer层应用自适应令牌池化。通过在MS MARCO段落和文档排序任务上的大量消融研究，我们证明，中间层的压缩在保持排序质量的同时，能将段落排序的推理QPS提升至多25%，文档排序提升至多116%。我们还将LTC扩展至列表式LLM重排序器，并表明相同方法可轻松应用于长上下文列表式重排序，其中QPS提升更为显著。更令人惊讶的是，当将基于短段落训练的重排序器应用于长文档排序任务时，经过压缩训练的模型性能优于未压缩的对应模型，这表明压缩可能作为一种有益的正则化手段，促进长度不变性表示的学习。