Language models like BERT and SpanBERT pretrained on open-domain data have obtained impressive gains on various NLP tasks. In this paper, we probe the effectiveness of domain-adaptive pretraining objectives on downstream tasks. In particular, three objectives, including a novel objective focusing on modeling predicate-argument relations, are evaluated on two challenging dialogue understanding tasks. Experimental results demonstrate that domain-adaptive pretraining with proper objectives can significantly improve the performance of a strong baseline on these tasks, achieving the new state-of-the-art performances.

翻译：BERT和SpanBERT等语言模型对开放域数据进行了预先培训,这些语言模型在各项国家劳工政策任务方面取得了令人印象深刻的成果。在本文件中,我们探讨了下游任务领域适应性培训前目标的有效性。特别是,根据两项具有挑战性的对话理解任务,评估了三项目标,包括一个侧重于模拟上游-争论关系的新目标。实验结果表明,具有适当目标的适应性培训前培训能够大大改善这些任务的强有力基线的绩效,实现新的最新业绩。