Transformer-based document cross-encoder rerankers are a central component of modern information retrieval systems. Despite their success, these models suffer from high computational costs due to processing long query-document sequences at inference time. A known approach to improve efficiency is token compression, which consists of aggregating groups of tokens together in the initial embedding layer, reducing the effective number of tokens, and making the computation faster. While token compression has proven to be successful for bi-encoder retrievers, we empirically observed that this approach may be ineffective for cross-encoder rerankers. In this paper, we propose Layer-wise Token Compression (LTC), which applies adaptive token pooling at intermediate transformer layers. Through extensive ablation studies on MS MARCO passage and document ranking tasks, we demonstrate that compression at middle layers preserves ranking quality while increasing inference QPS by up to 25% for passage ranking and up to 116% for document ranking. We also extend LTC to listwise LLM rerankers and show that the same approach can be easily applied to long-context listwise reranking, where the QPS improvements are even greater. More surprisingly, when applying rerankers trained on short passages to long-document ranking tasks, models trained with compression outperform their uncompressed counterparts, suggesting that compression may act as a beneficial regularizer that encourages length-invariant representations.

翻译：基于Transformer的文档交叉编码器重排序模型是现代信息检索系统的核心组件。尽管这些模型取得了显著成功，但在推理阶段处理长查询-文档序列时，仍面临高计算成本的困境。词元压缩是一种已知的效率优化方法，其通过在初始嵌入层聚合词元组，减少有效词元数量，从而加速计算过程。虽然词元压缩已在双编码检索器中展现出有效性，但我们的实证观察发现，该方法对交叉编码器重排序模型可能并不奏效。本文提出层级式词元压缩（Layer-wise Token Compression, LTC），该方法在Transformer中间层应用自适应词元池化。通过在MS MARCO段落排序与文档排序任务上的充分消融研究，我们证明中间层的压缩能在保持排序质量的同时，使段落排序任务推理QPS提升高达25%，文档排序任务QPS提升达116%。我们还将LTC扩展到列表式LLM重排序模型，表明该方法可轻松应用于长上下文列表式重排序场景，其QPS提升更为显著。更令人惊讶的是，当将基于短段落训练的重排序模型应用于长文档排序任务时，采用压缩训练的模型表现优于未压缩基线，这表明压缩可能通过促进长度无关表征的生成，起到了有益的正则化作用。