Understanding and manipulating the causal generation mechanisms in language models is essential for controlling their behavior. Previous work has primarily relied on techniques such as representation surgery -- e.g., model ablations or manipulation of linear subspaces tied to specific concepts -- to intervene on these models. To understand the impact of interventions precisely, it is useful to examine counterfactuals -- e.g., how a given sentence would have appeared had it been generated by the model following a specific intervention. We highlight that counterfactual reasoning is conceptually distinct from interventions, as articulated in Pearl's causal hierarchy. Based on this observation, we propose a framework for generating true string counterfactuals by reformulating language models as Generalized Structural-equation. Models using the Gumbel-max trick. This allows us to model the joint distribution over original strings and their counterfactuals resulting from the same instantiation of the sampling noise. We develop an algorithm based on hindsight Gumbel sampling that allows us to infer the latent noise variables and generate counterfactuals of observed strings. Our experiments demonstrate that the approach produces meaningful counterfactuals while at the same time showing that commonly used intervention techniques have considerable undesired side effects.

翻译：理解并操控语言模型中的因果生成机制对于控制其行为至关重要。先前的研究主要依赖于表示手术等技术——例如模型消融或操控与特定概念绑定的线性子空间——来干预这些模型。为了精确理解干预的影响，考察反事实是有益的——例如，一个给定的句子在模型经过特定干预后生成时会呈现何种样貌。我们强调，反事实推理在概念上不同于干预，正如珀尔的因果层次理论所阐明的那样。基于这一观察，我们提出了一个通过将语言模型重新表述为使用Gumbel-max技巧的广义结构方程模型来生成真实字符串反事实的框架。这使我们能够对原始字符串及其由相同采样噪声实例化产生的反事实的联合分布进行建模。我们开发了一种基于后见之明Gumbel采样的算法，该算法允许我们推断潜在噪声变量并生成观测字符串的反事实。我们的实验表明，该方法能够产生有意义的反事实，同时揭示了常用干预技术存在显著的意外副作用。