Controllable text generation (CTG) aims to generate text with desired attributes, and decoding-time-based methods have shown promising performance on this task. However, in this paper, we identify the phenomenon of Attribute Collapse for the first time. It causes the fluency of generated text to rapidly decrease when the control strength exceeds a critical value, rendering the text completely unusable. This limitation hinders the effectiveness of decoding methods in achieving high levels of controllability. To address this problem, we propose a novel lightweight decoding framework named Air-Decoding. Its main idea is reconstructing the attribute distributions to balance the weights between attribute words and non-attribute words to generate more fluent text. Specifically, we train prefixes by prefix-tuning to obtain attribute distributions. Then we design a novel attribute distribution reconstruction method to balance the obtained distributions and use the reconstructed distributions to guide language models for generation, effectively avoiding the issue of Attribute Collapse. Experiments on multiple CTG tasks prove that our method achieves a new state-of-the-art control performance.

翻译：可控文本生成旨在生成具有期望属性的文本，基于解码时控制的方法在该任务上已展现出良好性能。然而，本文首次揭示了属性坍缩现象——当控制强度超过临界值时，生成文本的流畅度会急剧下降，导致文本完全无法使用。这一缺陷限制了解码方法在实现高控制度方面的有效性。针对该问题，我们提出了一种名为Air-Decoding的新型轻量级解码框架，其核心思想是通过重构属性分布来平衡属性词与非属性词的权重，从而生成更流畅的文本。具体而言，我们利用前缀微调训练前缀以获取属性分布，随后设计了一种新颖的属性分布重构方法来平衡所获分布，并利用重构后的分布引导语言模型进行生成，从而有效规避属性坍缩问题。在多个可控文本生成任务上的实验证明，该方法实现了当前最优的控制性能。