Answer Sentence Selection (AS2) is a core component for building an accurate Question Answering pipeline. AS2 models rank a set of candidate sentences based on how likely they answer a given question. The state of the art in AS2 exploits pre-trained transformers by transferring them on large annotated datasets, while using local contextual information around the candidate sentence. In this paper, we propose three pre-training objectives designed to mimic the downstream fine-tuning task of contextual AS2. This allows for specializing LMs when fine-tuning for contextual AS2. Our experiments on three public and two large-scale industrial datasets show that our pre-training approaches (applied to RoBERTa and ELECTRA) can improve baseline contextual AS2 accuracy by up to 8% on some datasets.

翻译：答案句子选择（AS2）是构建精确问答系统的核心组件。AS2模型根据候选句子回答给定问题的可能性对其排序。当前AS2领域最先进的方法利用预训练Transformer，通过在大规模标注数据集上进行迁移学习，并结合候选句子周围的局部上下文信息。本文提出了三种预训练目标，旨在模拟上下文感知AS2的下游微调任务。这可以在针对上下文AS2进行微调时实现对语言模型的专门化。我们在三个公开数据集和两个大规模工业数据集上的实验表明，我们的预训练方法（应用于RoBERTa和ELECTRA）在某些数据集上可将基线上下文AS2的准确率最高提升8%。