Text-to-image diffusion models have demonstrated an unparalleled ability to generate high-quality, diverse images from a textual concept (e.g., "a doctor", "love"). However, the internal process of mapping text to a rich visual representation remains an enigma. In this work, we tackle the challenge of understanding concept representations in text-to-image models by decomposing an input text prompt into a small set of interpretable elements. This is achieved by learning a pseudo-token that is a sparse weighted combination of tokens from the model's vocabulary, with the objective of reconstructing the images generated for the given concept. Applied over the state-of-the-art Stable Diffusion model, this decomposition reveals non-trivial and surprising structures in the representations of concepts. For example, we find that some concepts such as "a president" or "a composer" are dominated by specific instances (e.g., "Obama", "Biden") and their interpolations. Other concepts, such as "happiness" combine associated terms that can be concrete ("family", "laughter") or abstract ("friendship", "emotion"). In addition to peering into the inner workings of Stable Diffusion, our method also enables applications such as single-image decomposition to tokens, bias detection and mitigation, and semantic image manipulation. Our code will be available at: https://hila-chefer.github.io/Conceptor/

翻译：文本到图像扩散模型展现了从文本概念（例如“医生”、“爱”）生成高质量、多样化图像的无与伦比的能力。然而，将文本映射到丰富视觉表示的内部过程仍然是一个谜。在本文中，我们通过将输入文本提示分解为一小组可解释元素，来应对理解文本到图像模型中概念表示的挑战。这是通过学习一个伪标记实现的，该伪标记是模型词汇表中标记的稀疏加权组合，其目标是重建为给定概念生成的图像。将该方法应用于最先进的稳定扩散模型，分解揭示了概念表示中非平凡且令人惊讶的结构。例如，我们发现诸如“总统”或“作曲家”等概念由特定实例（例如“奥巴马”、“拜登”）及其插值主导。其他概念，如“幸福”，则结合了具体的关联词（“家庭”、“笑声”）或抽象的关联词（“友谊”、“情感”）。除了洞察稳定扩散的内部工作机制，我们的方法还实现了应用，例如将单张图像分解为标记、偏差检测与缓解以及语义图像操作。我们的代码将在 https://hila-chefer.github.io/Conceptor/ 提供。