In this paper, we consider the problem of improving the inference latency of language model-based dense retrieval systems by introducing structural compression and model size asymmetry between the context and query encoders. First, we investigate the impact of pre and post-training compression on the MSMARCO, Natural Questions, TriviaQA, SQUAD, and SCIFACT, finding that asymmetry in the dual encoders in dense retrieval can lead to improved inference efficiency. Knowing this, we introduce Kullback Leibler Alignment of Embeddings (KALE), an efficient and accurate method for increasing the inference efficiency of dense retrieval methods by pruning and aligning the query encoder after training. Specifically, KALE extends traditional Knowledge Distillation after bi-encoder training, allowing for effective query encoder compression without full retraining or index generation. Using KALE and asymmetric training, we can generate models which exceed the performance of DistilBERT despite having 3x faster inference.

翻译：本文研究如何通过引入结构压缩和查询编码器与上下文编码器的模型规模不对称性，来降低基于语言模型的密集检索系统的推理延迟。首先，我们在MSMARCO、Natural Questions、TriviaQA、SQUAD和SCIFACT数据集上探讨了训练前与训练后压缩的影响，发现密集检索中双编码器的非对称性能够提升推理效率。基于此，我们提出库尔贝克-莱布勒嵌入对齐方法（KALE），这是一种通过训练后剪枝和对齐查询编码器来提高密集检索方法推理效率的精确高效技术。具体而言，KALE在双编码器训练后扩展了传统知识蒸馏方法，无需完整重训练或索引生成即可实现有效的查询编码器压缩。利用KALE与非对称训练，我们生成的模型在推理速度提升3倍的情况下，性能仍超越DistilBERT。