Regulatory documents are rich in nuanced terminology and specialized semantics. FRAG systems: Frozen retrieval-augmented generators utilizing pre-trained (or, frozen) components face consequent challenges with both retriever and answering performance. We present a system that adapts the retriever performance to the target domain using a multi-stage tuning (MST) strategy. Our retrieval approach, called MST-R (a) first fine-tunes encoders used in vector stores using hard negative mining, (b) then uses a hybrid retriever, combining sparse and dense retrievers using reciprocal rank fusion, and then (c) adapts the cross-attention encoder by fine-tuning only the top-k retrieved results. We benchmark the system performance on the dataset released for the RIRAG challenge (as part of the RegNLP workshop at COLING 2025). We achieve significant performance gains obtaining a top rank on the RegNLP challenge leaderboard. We also show that a trivial answering approach games the RePASs metric outscoring all baselines and a pre-trained Llama model. Analyzing this anomaly, we present important takeaways for future research.

翻译：监管文件富含细微差别的术语和专业化语义。采用预训练（或冻结）组件的冻结检索增强生成器（FRAG）系统因此面临检索器与回答性能的双重挑战。我们提出一种系统，通过多阶段调优（MST）策略使检索器性能适应目标领域。我们的检索方法MST-R（a）首先通过困难负例挖掘对向量数据库中使用的编码器进行微调，（b）随后采用混合检索器，通过逆序位融合结合稀疏与稠密检索器，（c）最后通过仅对top-k检索结果进行微调来适配交叉注意力编码器。我们在RIRAG挑战赛（作为COLING 2025 RegNLP研讨会组成部分）发布的数据集上对系统性能进行基准测试，取得了显著的性能提升，并在RegNLP挑战赛排行榜上获得首位。我们还证明，一种简单的回答方法能够操纵RePASs指标，其得分超越所有基线及预训练的Llama模型。通过分析这一异常现象，我们为未来研究提出了重要启示。