In designing generative models, it is commonly believed that in order to learn useful latent structure, we face a fundamental tension between expressivity and structure. In this paper we challenge this view by proposing a new approach to training arbitrarily expressive generative models that simultaneously learn causally disentangled concepts. This is accomplished by adding a simple context module to an arbitrarily complex black-box model, which learns to process concept information by implicitly inverting linear representations from the model's encoder. Inspired by the notion of intervention in a causal model, our module selectively modifies its architecture during training, allowing it to learn a compact joint model over different contexts. We show how adding this module leads to causally disentangled representations that can be composed for out-of-distribution generation on both real and simulated data. The resulting models can be trained end-to-end or fine-tuned from pre-trained models. To further validate our proposed approach, we prove a new identifiability result that extends existing work on identifying structured representations.

翻译：在设计生成模型时，普遍认为为了学习有用的潜在结构，我们面临表达能力与结构之间的根本权衡。本文通过提出一种训练任意表达能力生成模型的新方法挑战这一观点，该方法能够同时学习因果解耦的概念。通过在任意复杂的黑箱模型中加入一个简单的上下文模块实现，该模块通过隐式反转模型编码器中的线性表征来学习处理概念信息。受因果模型中干预概念的启发，该模块在训练过程中选择性修改其架构，使其能够学习不同上下文下的紧凑联合模型。我们展示了添加该模块如何产生可组合的因果解耦表征，以用于真实数据和模拟数据的分布外生成。所得模型可进行端到端训练或从预训练模型微调。为进一步验证所提方法，我们证明了一个新的可辨识性结果，该结果扩展了现有关于识别结构化表征的研究。