Transformer-based pretrained language models (PLMs) have achieved great success in modern NLP. An important advantage of PLMs is good out-of-distribution (OOD) robustness. Recently, diffusion models have attracted a lot of work to apply diffusion to PLMs. It remains under-explored how diffusion influences PLMs on OOD data. The core of diffusion models is a forward diffusion process which gradually applies Gaussian noise to inputs, and a reverse denoising process which removes noise. The noised input reconstruction is a fundamental ability of diffusion models. We directly analyze OOD robustness by measuring the reconstruction loss, including testing the abilities to reconstruct OOD data, and to detect OOD samples. Experiments are conducted by analyzing different training parameters and data statistical features on eight datasets. It shows that finetuning PLMs with diffusion degrades the reconstruction ability on OOD data. The comparison also shows that diffusion models can effectively detect OOD samples, achieving state-of-the-art performance in most of the datasets with an absolute accuracy improvement up to 18%. These results indicate that diffusion reduces OOD robustness of PLMs.

翻译：基于Transformer的预训练语言模型（PLMs）在现代自然语言处理中取得了巨大成功。PLMs的一个重要优势是具有良好的分布外（OOD）鲁棒性。近年来，扩散模型吸引了大量工作将其应用于PLMs。然而，扩散如何影响PLMs在OOD数据上的表现仍鲜有研究。扩散模型的核心是一个前向扩散过程，该过程逐步向输入添加高斯噪声，以及一个反向去噪过程用于去除噪声。带噪输入的重建是扩散模型的基本能力。我们通过测量重建损失直接分析OOD鲁棒性，包括测试重建OOD数据的能力以及检测OOD样本的能力。实验基于八个数据集，分析了不同训练参数和数据统计特征。结果表明，使用扩散微调PLMs会降低其在OOD数据上的重建能力。对比还显示，扩散模型能有效检测OOD样本，在大多数数据集上实现了最先进的性能，绝对准确率提升高达18%。这些结果说明，扩散降低了PLMs的OOD鲁棒性。