In recent years, large language models (LMs) have achieved remarkable progress across various natural language processing tasks. As pre-training and fine-tuning are costly and might negatively impact model performance, it is desired to efficiently adapt an existing model to different conditions such as styles, sentiments or narratives, when facing different audiences or scenarios. However, efficient adaptation of a language model to diverse conditions remains an open challenge. This work is inspired by the observation that text conditions are often associated with selection of certain words in a context. Therefore we introduce LM-Switch, a theoretically grounded, lightweight and simple method for generative language model conditioning. We begin by investigating the effect of conditions in Hidden Markov Models (HMMs), and establish a theoretical connection with language model. Our finding suggests that condition shifts in HMMs are associated with linear transformations in word embeddings. LM-Switch is then designed to deploy a learnable linear factor in the word embedding space for language model conditioning. We show that LM-Switch can model diverse tasks, and achieves comparable or better performance compared with state-of-the-art baselines in LM detoxification and generation control, despite requiring no more than 1% of parameters compared with baselines and little extra time overhead compared with base LMs. It is also able to learn from as few as a few sentences or one document. Moreover, a learned LM-Switch can be transferred to other LMs of different sizes, achieving a detoxification performance similar to the best baseline. We will make our code available to the research community following publication.

翻译：近年来，大型语言模型在各种自然语言处理任务中取得了显著进展。由于预训练和微调成本高昂且可能对模型性能产生负面影响，因此在面对不同受众或场景时，如何高效地将现有模型适配到不同条件（如风格、情感或叙事）成为迫切需求。然而，高效适配语言模型以应对多样条件仍是一个开放挑战。本研究受以下观察启发：文本条件通常与上下文中特定词汇的选择相关联。因此，我们提出LM-Switch——一种理论完备、轻量且简单的生成式语言模型条件控制方法。我们首先在隐马尔可夫模型（HMM）中研究条件效应，并建立与语言模型的理论联系。研究发现，HMM中的条件偏移与词嵌入中的线性变换相关。据此设计的LM-Switch通过在词嵌入空间部署可学习的线性因子实现语言模型条件控制。实验表明，LM-Switch能建模多样任务，在语言模型去毒化和生成控制任务中取得与现有最优基线相当或更好的性能，且参数量仅为基线的1%以下，额外时间开销相比基础语言模型微乎其微。它还能从寥寥数句或单篇文档中学习。此外，训练后的LM-Switch可迁移至不同规模的其他语言模型，实现与最佳基线相近的去毒化效果。论文发表后，我们将向研究社区公开代码。