Language use differs between domains and even within a domain, language use changes over time. Previous work shows that adapting pretrained language models like BERT to domain through continued pretraining improves performance on in-domain downstream tasks. In this article, we investigate whether adapting BERT to time in addition to domain can increase performance even further. For this purpose, we introduce a benchmark corpus of social media comments sampled over three years. The corpus consists of 36.36m unlabelled comments for adaptation and evaluation on an upstream masked language modelling task as well as 0.9m labelled comments for finetuning and evaluation on a downstream document classification task. We find that temporality matters for both tasks: temporal adaptation improves upstream task performance and temporal finetuning improves downstream task performance. However, we do not find clear evidence that adapting BERT to time and domain improves downstream task performance over just adapting to domain. Temporal adaptation captures changes in language use in the downstream task, but not those changes that are actually relevant to performance on it.

翻译：以往的工作表明,通过持续的培训前培训,使诸如BERT等预先培训语言模型的域名改成域名,可以提高内部下游任务的业绩。在本条中,我们调查的是,除域名外,调整BERT的时间是否还能进一步提高业绩。为此,我们引入了三年来抽样的社交媒体评论基准集。该基准集包括36.36米未贴标签的关于上游遮盖语言建模任务适应和评价的评论,以及0.9米贴标签的评论,以对下游文件分类任务进行微调和评价。我们发现,这两项任务的时间性关系:时间性适应可以改善上游任务的业绩,时间性调整可以改善下游任务的业绩。然而,我们没有找到明确的证据,证明使BERT适应时间和领域能提高下游任务业绩,而只是适应领域。时间性适应可以捕捉下游任务语言使用的变化,但与下游任务实际相关的变化并不重要。