Pretrained language models (PLMs) have produced substantial improvements in discourse-aware neural machine translation (NMT), for example, improved coherence in spoken language translation. However, the underlying reasons for their strong performance have not been well explained. To bridge this gap, we introduce a probing task to interpret the ability of PLMs to capture discourse relation knowledge. We validate three state-of-the-art PLMs across encoder-, decoder-, and encoder-decoder-based models. The analysis shows that (1) the ability of PLMs on discourse modelling varies from architecture and layer; (2) discourse elements in a text lead to different learning difficulties for PLMs. Besides, we investigate the effects of different PLMs on spoken language translation. Through experiments on IWSLT2017 Chinese-English dataset, we empirically reveal that NMT models initialized from different layers of PLMs exhibit the same trends with the probing task. Our findings are instructive to understand how and when discourse knowledge in PLMs should work for downstream tasks.

翻译：预训练语言模型（PLMs）显著提升了语篇感知神经机器翻译（NMT）的性能，例如在口语翻译中增强了连贯性。然而，其强大性能的潜在原因尚未得到充分解释。为填补这一空白，我们引入了一项探针任务，用以阐释PLMs捕捉语篇关系知识的能力。我们在基于编码器、解码器及编码器-解码器的模型中验证了三种最先进的PLMs。分析表明：（1）PLMs的语篇建模能力随架构和层的不同而变化；（2）文本中的语篇元素会导致PLMs面临不同的学习难度。此外，我们研究了不同PLMs对口语翻译的影响。通过在IWSLT2017中英数据集上的实验，我们实证发现，从PLMs不同层初始化的NMT模型在探针任务中表现出相同趋势。我们的发现对于理解PLMs中的语篇知识如何及何时为下游任务发挥作用具有指导意义。