This paper presents a new supervised representation learning framework, namely structured probabilistic coding (SPC), to learn compact and informative representations from input related to the target task. SPC is an encoder-only probabilistic coding technology with a structured regularization from the target space. It can enhance the generalization ability of pre-trained language models for better language understanding. Specifically, our probabilistic coding simultaneously performs information encoding and task prediction in one module to more fully utilize the effective information from input data. It uses variational inference in the output space to reduce randomness and uncertainty. Besides, to better control the learning process of probabilistic representations, a structured regularization is proposed to promote uniformity across classes in the latent space. With the regularization term, SPC can preserve the Gaussian structure of the latent code and achieve better coverage of the hidden space with class uniformly. Experimental results on 12 natural language understanding tasks demonstrate that our SPC effectively improves the performance of pre-trained language models for classification and regression. Extensive experiments show that SPC can enhance the generalization capability, robustness to label noise, and clustering quality of output representations.

翻译：本文提出了一种新的监督表示学习框架——结构化概率编码（SPC），旨在从与目标任务相关的输入中学习紧凑且信息丰富的表示。SPC是一种基于编码器的概率编码技术，通过目标空间的结构化正则化实现。它能增强预训练语言模型的泛化能力，以提升语言理解性能。具体而言，我们的概率编码在一个模块中同时进行信息编码和任务预测，以更充分地利用输入数据中的有效信息，并在输出空间中采用变分推断以降低随机性和不确定性。此外，为更好地控制概率表示的学习过程，我们提出了一种结构化正则化方法，以促进潜在空间中类别间的均匀性。借助该正则化项，SPC能够保持潜在编码的高斯结构，并通过类别均匀性实现隐藏空间的更优覆盖。在12项自然语言理解任务上的实验结果表明，我们的SPC有效提升了预训练语言模型在分类和回归任务中的性能。大量实验证明，SPC增强了输出表示的泛化能力、标签噪声鲁棒性以及聚类质量。