Retrieval models based on dense representations in semantic space have become an indispensable branch for first-stage retrieval. These retrievers benefit from surging advances in representation learning towards compressive global sequence-level embeddings. However, they are prone to overlook local salient phrases and entity mentions in texts, which usually play pivot roles in first-stage retrieval. To mitigate this weakness, we propose to make a dense retriever align a well-performing lexicon-aware representation model. The alignment is achieved by weakened knowledge distillations to enlighten the retriever via two aspects -- 1) a lexicon-augmented contrastive objective to challenge the dense encoder and 2) a pair-wise rank-consistent regularization to make dense model's behavior incline to the other. We evaluate our model on three public benchmarks, which shows that with a comparable lexicon-aware retriever as the teacher, our proposed dense one can bring consistent and significant improvements, and even outdo its teacher. In addition, we found our improvement on the dense retriever is complementary to the standard ranker distillation, which can further lift state-of-the-art performance.

翻译：基于语义空间中密集表示的检索模型已成为第一阶段检索不可或缺的分支。这些检索器受益于表示学习在压缩性全局序列嵌入方面的迅猛进展，但常忽视文本中局部显著短语和实体提及——这些元素通常在第一阶段检索中起关键作用。为缓解此缺陷，我们提出使密集检索器与高性能词典感知表示模型对齐。该对齐通过弱化知识蒸馏实现，从两方面启发密集检索器：1）词典增强对比目标以挑战密集编码器；2）成对排序一致性正则化使密集模型行为倾向另一模型。我们在三个公开基准上评估模型，结果表明以可比的词典感知检索器作为教师模型时，所提出的密集检索器能带来持续显著提升，甚至超越教师模型。此外，我们发现对密集检索器的改进与标准排序器蒸馏具有互补性，可进一步提升当前最优性能。