We present Causal-Adapter, a modular framework that adapts frozen text-to-image diffusion backbones for counterfactual image generation. Our method supports causal interventions on target attributes and consistently propagates their effects to causal dependents while preserving the core identity of the image. Unlike prior approaches that rely on prompt engineering without explicit causal structure, Causal-Adapter leverages structural causal modeling with two attribute-regularization strategies: (i) prompt-aligned injection, which aligns causal attributes with textual embeddings for precise semantic control, and (ii) a conditioned token contrastive loss that disentangles attribute factors and reduces spurious correlations. Causal-Adapter achieves state-of-the-art performance on both synthetic and real-world datasets, including up to a 91% reduction in MAE on Pendulum for accurate attribute control and up to an 87% reduction in FID on ADNI for high-fidelity MRI generation. These results demonstrate robust, generalizable counterfactual editing with faithful attribute modification and strong identity preservation. Code and models will be released at: https://leitong02.github.io/causaladapter/.

翻译：我们提出Causal-Adapter，一种模块化框架，用于适配冻结的文本到图像扩散主干网络以进行反事实图像生成。我们的方法支持对目标属性进行因果干预，并一致地将干预效应传播至因果依赖项，同时保持图像的核心身份特征。与先前依赖提示工程而缺乏显式因果结构的方法不同，Causal-Adapter利用结构因果建模，并采用两种属性正则化策略：(i) 提示对齐注入，将因果属性与文本嵌入对齐以实现精确的语义控制；(ii) 条件化标记对比损失，用于解耦属性因子并减少虚假相关性。Causal-Adapter在合成和真实世界数据集上均实现了最先进的性能，包括在Pendulum数据集上实现高达91%的MAE降低以实现精确属性控制，在ADNI数据集上实现高达87%的FID降低以生成高保真MRI图像。这些结果证明了该方法具有鲁棒性、可泛化的反事实编辑能力，能够实现忠实的属性修改和强大的身份保持。代码和模型将在以下地址发布：https://leitong02.github.io/causaladapter/。