This paper presents a new supervised representation learning framework, namely structured probabilistic coding (SPC), to learn compact and informative representations from input related to the target task. SPC is an encoder-only probabilistic coding technology with a structured regularization from the target space. It can enhance the generalization ability of pre-trained language models for better language understanding. Specifically, our probabilistic coding simultaneously performs information encoding and task prediction in one module to more fully utilize the effective information from input data. It uses variational inference in the output space to reduce randomness and uncertainty. Besides, to better control the learning process of probabilistic representations, a structured regularization is proposed to promote uniformity across classes in the latent space. With the regularization term, SPC can preserve the Gaussian structure of the latent code and achieve better coverage of the hidden space with class uniformly. Experimental results on 12 natural language understanding tasks demonstrate that our SPC effectively improves the performance of pre-trained language models for classification and regression. Extensive experiments show that SPC can enhance the generalization capability, robustness to label noise, and clustering quality of output representations.

翻译：本文提出了一种新的监督表示学习框架——结构化概率编码（SPC），旨在从与目标任务相关的输入中学习紧凑且信息丰富的表示。SPC是一种仅编码器的概率编码技术，并带有来自目标空间的结构化正则化。它可以增强预训练语言模型的泛化能力，以更好地进行语言理解。具体而言，我们的概率编码在一个模块中同时进行信息编码和任务预测，从而更充分地利用输入数据中的有效信息。该方法在输出空间中利用变分推理来减少随机性和不确定性。此外，为了更好地控制概率表示的学习过程，我们提出了一种结构化正则化，以促进潜在空间中类别间的均匀性。借助该正则化项，SPC能够保持潜在码的高斯结构，并通过类别均匀性实现对隐空间的更好覆盖。在12个自然语言理解任务上的实验结果表明，我们的SPC有效提升了预训练语言模型在分类和回归任务中的性能。大量实验显示，SPC能够增强输出表示的泛化能力、对标签噪声的鲁棒性以及聚类质量。