Self-training (ST) has prospered again in language understanding by augmenting the fine-tuning of pre-trained language models when labeled data is insufficient. However, it remains challenging to incorporate ST into attribute-controllable language generation. Augmented by only self-generated pseudo text, generation models over-emphasize exploitation of the previously learned space, suffering from a constrained generalization boundary. We revisit ST and propose a novel method, DuNST to alleviate this problem. DuNST jointly models text generation and classification with a shared Variational AutoEncoder and corrupts the generated pseudo text by two kinds of flexible noise to disturb the space. In this way, our model could construct and utilize both pseudo text from given labels and pseudo labels from available unlabeled text, which are gradually refined during the ST process. We theoretically demonstrate that DuNST can be regarded as enhancing exploration towards the potential real text space, providing a guarantee of improved performance. Experiments on three controllable generation tasks show that DuNST could significantly boost control accuracy while maintaining comparable generation fluency and diversity against several strong baselines.

翻译：自训练（ST）在标注数据不足时通过增强预训练语言模型的微调，在语言理解领域再次取得成功。然而，将自训练应用于属性可控语言生成仍具挑战性。仅依靠自生成的伪文本进行增强时，生成模型过度强调对先前已学习空间的利用，导致泛化边界受限。我们重新审视自训练方法，并提出了一种新型方法DuNST来缓解该问题。DuNST通过共享变分自编码器联合建模文本生成与分类，并利用两种灵活噪声对生成的伪文本进行扰动，从而扰乱已有空间。通过这种方式，我们的模型能够构建并利用来自给定标签的伪文本和来自可用无标签文本的伪标签，并在自训练过程中逐步优化这些数据。我们从理论上证明，DuNST可视为增强对潜在真实文本空间的探索，为性能提升提供保证。在三个可控生成任务上的实验表明，与多个强基线相比，DuNST能够在保持相当生成流畅性与多样性的同时，显著提升控制准确率。