Text-to-image diffusion models have made significant progress in generating naturalistic images from textual inputs, and demonstrate the capacity to learn and represent complex visual-semantic relationships. While these diffusion models have achieved remarkable success, the underlying mechanisms driving their performance are not yet fully accounted for, with many unanswered questions surrounding what they learn, how they represent visual-semantic relationships, and why they sometimes fail to generalize. Our work presents Diffusion Partial Information Decomposition (DiffusionPID), a novel technique that applies information-theoretic principles to decompose the input text prompt into its elementary components, enabling a detailed examination of how individual tokens and their interactions shape the generated image. We introduce a formal approach to analyze the uniqueness, redundancy, and synergy terms by applying PID to the denoising model at both the image and pixel level. This approach enables us to characterize how individual tokens and their interactions affect the model output. We first present a fine-grained analysis of characteristics utilized by the model to uniquely localize specific concepts, we then apply our approach in bias analysis and show it can recover gender and ethnicity biases. Finally, we use our method to visually characterize word ambiguity and similarity from the model's perspective and illustrate the efficacy of our method for prompt intervention. Our results show that PID is a potent tool for evaluating and diagnosing text-to-image diffusion models.

翻译：文本到图像扩散模型在根据文本输入生成逼真图像方面取得了显著进展，并展现出学习和表征复杂视觉-语义关系的能力。尽管这些扩散模型已取得显著成功，但其性能背后的驱动机制尚未得到充分阐释，关于模型学习了什么、如何表征视觉-语义关系以及为何有时泛化能力不足等问题仍悬而未决。本研究提出扩散部分信息分解（DiffusionPID），这是一种基于信息论原理的新技术，可将输入文本提示分解为基本组成单元，从而细致考察单个词汇标记及其交互作用如何塑造生成图像。我们通过将PID应用于图像级和像素级的去噪模型，提出了分析唯一性、冗余性和协同性指标的规范化方法。该方法使我们能够刻画单个词汇标记及其交互作用对模型输出的影响机制。我们首先对模型用于精确定位特定概念的特征进行了细粒度分析，随后将本方法应用于偏差分析，证明其能够有效识别性别与种族偏见。最后，我们利用该方法从模型视角对词汇歧义性和相似性进行可视化表征，并展示了本方法在提示干预中的有效性。实验结果表明，PID是评估和诊断文本到图像扩散模型的强效工具。