Understanding the structure of real data is paramount in advancing modern deep-learning methodologies. Natural data such as images are believed to be composed of features organised in a hierarchical and combinatorial manner, which neural networks capture during learning. Recent advancements show that diffusion models can generate high-quality images, hinting at their ability to capture this underlying structure. We study this phenomenon in a hierarchical generative model of data. We find that the backward diffusion process acting after a time $t$ is governed by a phase transition at some threshold time, where the probability of reconstructing high-level features, like the class of an image, suddenly drops. Instead, the reconstruction of low-level features, such as specific details of an image, evolves smoothly across the whole diffusion process. This result implies that at times beyond the transition, the class has changed but the generated sample may still be composed of low-level elements of the initial image. We validate these theoretical insights through numerical experiments on class-unconditional ImageNet diffusion models. Our analysis characterises the relationship between time and scale in diffusion models and puts forward generative models as powerful tools to model combinatorial data properties.

翻译：理解真实数据的结构对于推进现代深度学习方法论至关重要。诸如图像等自然数据被认为是由以层次化和组合方式组织的特征构成，神经网络在学习过程中捕获这些特征。最新进展表明，扩散模型能够生成高质量图像，暗示其具备捕获此类底层结构的能力。我们基于数据的分层生成模型对此现象展开研究。研究发现，在时间$t$后运行的反向扩散过程受阈值时间处的相变支配：当通过该阈值时，重构高层特征（如图像类别）的概率突然下降；而重构低层特征（如图像细节）的概率则在整个扩散过程中平滑演化。这一结果表明，在相变时间点之后，图像类别虽已改变，但生成样本可能仍由原始图像的低层元素构成。我们通过对无类别条件的ImageNet扩散模型进行数值实验验证了这些理论洞见。本分析刻画了扩散模型中时间与尺度的关系，并提出生成模型可作为建模组合数据特性的有力工具。