Deploying LLMs in real-world applications requires controllable output that satisfies multiple desiderata at the same time. While existing work extensively addresses LLM steering for a single behavior, \textit{compositional steering} -- i.e., steering LLMs simultaneously towards multiple behaviors -- remains an underexplored problem. In this work, we propose \emph{compositional steering tokens} for multi-behavior steering. We first embed individual behaviors, expressed as natural language instructions, into dedicated tokens via self-distillation. Contrary to most prior work, which operates in the activation space, our behavior steers live in the space of input tokens, enabling more effective zero-shot composition. We then train a dedicated \textit{composition token} on pairs of behaviors and show that it successfully captures the notion of composition: it generalizes well to \textit{unseen} compositions, including those with unseen behaviors as well as those with an unseen \textit{number} of behaviors. Our experiments across different LLM architectures show that steering tokens lead to superior multi-behavior control compared to competing approaches (instructions, activation steering, and LoRA merging). Moreover, we show that steering tokens complement natural language instructions, with their combination resulting in further gains.

翻译：在现实应用中部署大型语言模型需要生成满足多重期望的可控输出。现有研究已广泛探讨针对单一行为的LLM引导，然而\textit{组合式引导}——即同时引导LLM实现多种行为——仍是一个尚未充分探索的问题。本研究提出用于多行为引导的\emph{组合式引导令牌}方法。我们首先通过自蒸馏将自然语言指令描述的个体行为嵌入专用令牌。与多数在激活空间操作的前沿研究不同，我们的行为引导器存在于输入令牌空间，从而实现更有效的零样本组合。随后我们在行为对上训练专用的\textit{组合令牌}，并证明其成功捕获了组合语义：该令牌能良好泛化至\textit{未见}组合，包括含未见行为的组合以及含未见\textit{数量}行为的组合。在不同LLM架构上的实验表明，相较于竞争方法（指令调优、激活引导与LoRA融合），引导令牌能实现更优的多行为控制。此外，我们证明引导令牌可与自然语言指令形成互补，二者结合能产生进一步的性能提升。