Dense Retrieval Models (DRMs) are a prominent development in Information Retrieval (IR). A key challenge with these neural Transformer-based models is that they often struggle to generalize beyond the specific tasks and domains they were trained on. To address this challenge, prior research in IR incorporated the Mixture-of-Experts (MoE) framework within each Transformer layer of a DRM, which, though effective, substantially increased the number of additional parameters. In this paper, we propose a more efficient design, which introduces a single MoE block (SB-MoE) after the final Transformer layer. To assess the retrieval effectiveness of SB-MoE, we perform an empirical evaluation across three IR tasks. Our experiments involve two evaluation setups, aiming to assess both in-domain effectiveness and the model's zero-shot generalizability. In the first setup, we fine-tune SB-MoE with four different underlying DRMs on seven IR benchmarks and evaluate them on their respective test sets. In the second setup, we fine-tune SB-MoE on MSMARCO and perform zero-shot evaluation on thirteen BEIR datasets. Additionally, we perform further experiments to analyze the model's dependency on its hyperparameters (i.e., the number of employed and activated experts) and investigate how this variation affects SB-MoE's performance. The obtained results show that SB-MoE is particularly effective for DRMs with lightweight base models, such as TinyBERT and BERT-Small, consistently exceeding standard model fine-tuning across benchmarks. For DRMs with more parameters, such as BERT-Base and Contriever, our model requires a larger number of training samples to achieve improved retrieval performance. Our code is available online at: https://github.com/FaySokli/SB-MoE.

翻译：稠密检索模型是信息检索领域的一项重要进展。这类基于神经Transformer的模型面临的一个关键挑战是，它们往往难以泛化到训练任务和领域之外。为解决这一挑战，先前的信息检索研究在稠密检索模型的每个Transformer层中引入了专家混合框架，该方法虽有效但显著增加了额外参数量。本文提出一种更高效的设计方案，即在最终Transformer层后引入单一专家混合块。为评估SB-MoE的检索效能，我们在三个信息检索任务上进行了实证评估。实验包含两种评估设置，旨在同时评估模型在领域内的效能及其零样本泛化能力。在第一组设置中，我们基于七项信息检索基准测试，使用四种不同的底层稠密检索模型对SB-MoE进行微调，并在相应测试集上评估性能。在第二组设置中，我们在MSMARCO数据集上微调SB-MoE，并在十三组BEIR数据集上进行零样本评估。此外，我们通过进一步实验分析了模型对其超参数的依赖性，并探究了专家数量与激活专家数的变化如何影响SB-MoE的性能。实验结果表明，SB-MoE对于采用轻量级基础模型的稠密检索模型具有显著优势，在TinyBERT和BERT-Small等模型上持续超越标准微调方法。对于BERT-Base和Contriever等参数量更大的模型，我们的方法需要更多训练样本才能实现检索性能的提升。代码已开源：https://github.com/FaySokli/SB-MoE。