Salient Span Masking (SSM) has shown itself to be an effective strategy to improve closed-book question answering performance. SSM extends general masked language model pretraining by creating additional unsupervised training sentences that mask a single entity or date span, thus oversampling factual information. Despite the success of this paradigm, the span types and sampling strategies are relatively arbitrary and not widely studied for other tasks. Thus, we investigate SSM from the perspective of temporal tasks, where learning a good representation of various temporal expressions is important. To that end, we introduce Temporal Span Masking (TSM) intermediate training. First, we find that SSM alone improves the downstream performance on three temporal tasks by an avg. +5.8 points. Further, we are able to achieve additional improvements (avg. +0.29 points) by adding the TSM task. These comprise the new best reported results on the targeted tasks. Our analysis suggests that the effectiveness of SSM stems from the sentences chosen in the training data rather than the mask choice: sentences with entities frequently also contain temporal expressions. Nonetheless, the additional targeted spans of TSM can still improve performance, especially in a zero-shot context.

翻译：显著跨度掩码（SSM）已被证明是提升闭卷问答性能的有效策略。SSM通过创建额外的无监督训练语句来扩展通用掩码语言模型预训练，这些语句掩码单个实体或日期跨度，从而对事实信息进行过采样。尽管该范式取得了成功，但其跨度类型和采样策略相对随意，且未在其他任务中得到广泛研究。因此，我们从时间任务的角度研究SSM，其中学习各种时间表达的良好表示至关重要。为此，我们引入时间跨度掩码（TSM）中间训练。首先，我们发现仅使用SSM即可使三项时间任务的下游性能平均提升5.8个百分点。进一步地，通过添加TSM任务，我们实现了额外提升（平均0.29个百分点）。这些构成了所针对任务的最新最佳报告结果。我们的分析表明，SSM的有效性源于训练数据中选择的句子，而非掩码选择：包含实体的句子通常也包含时间表达。尽管如此，TSM的额外目标跨度仍能提升性能，尤其是在零样本场景下。