Controllable text generation (CTG) aims to generate text with desired attributes, and decoding-time-based methods have shown promising performance on this task. However, in this paper, we identify the phenomenon of Attribute Collapse for the first time. It causes the fluency of generated text to rapidly decrease when the control strength exceeds a critical value, rendering the text completely unusable. This limitation hinders the effectiveness of decoding methods in achieving high levels of controllability. To address this problem, we propose a novel lightweight decoding framework named Air-Decoding. Its main idea is reconstructing the attribute distributions to balance the weights between attribute words and non-attribute words to generate more fluent text. Specifically, we train prefixes by prefix-tuning to obtain attribute distributions. Then we design a novel attribute distribution reconstruction method to balance the obtained distributions and use the reconstructed distributions to guide language models for generation, effectively avoiding the issue of Attribute Collapse. Experiments on multiple CTG tasks prove that our method achieves a new state-of-the-art control performance.

翻译：可控文本生成（CTG）旨在生成具有期望属性的文本，基于解码时的方法在此任务上展现出良好性能。然而，本文首次揭示了属性坍塌现象。当控制强度超过临界值时，该现象会导致生成文本的流畅度急剧下降，使文本完全不可用。这一局限阻碍了解码方法实现高可控性的有效性。为解决该问题，我们提出一种名为Air-Decoding的新型轻量级解码框架。其核心思想是通过重构属性分布来平衡属性词与非属性词的权重，从而生成更流畅的文本。具体而言，我们利用前缀微调训练前缀以获取属性分布，随后设计一种新颖的属性分布重构方法对所得分布进行平衡处理，并利用重构后的分布引导语言模型生成文本，有效避免了属性坍塌问题。在多个CTG任务上的实验证明，我们的方法实现了新的最优控制性能。