Understanding the structure of real data is paramount in advancing modern deep-learning methodologies. Natural data such as images are believed to be composed of features organised in a hierarchical and combinatorial manner, which neural networks capture during learning. Recent advancements show that diffusion models can generate high-quality images, hinting at their ability to capture this underlying structure. We study this phenomenon in a hierarchical generative model of data. We find that the backward diffusion process acting after a time $t$ is governed by a phase transition at some threshold time, where the probability of reconstructing high-level features, like the class of an image, suddenly drops. Instead, the reconstruction of low-level features, such as specific details of an image, evolves smoothly across the whole diffusion process. This result implies that at times beyond the transition, the class has changed but the generated sample may still be composed of low-level elements of the initial image. We validate these theoretical insights through numerical experiments on class-unconditional ImageNet diffusion models. Our analysis characterises the relationship between time and scale in diffusion models and puts forward generative models as powerful tools to model combinatorial data properties.

翻译：理解真实数据的结构对于推进现代深度学习方法论至关重要。诸如图像等自然数据被认为由以分层和组合方式组织的特征构成，神经网络在学习过程中会捕获这些特征。最新进展表明，扩散模型能够生成高质量图像，这暗示它们具备捕捉这种底层结构的能力。我们在一个分层生成数据模型中研究了这一现象。研究发现，在时间 $t$ 之后进行的反向扩散过程受某个阈值时间处的相变主导，此时重建高层特征（如图像类别）的概率会突然下降。相反，低层特征（如图像的特定细节）的重建则在扩散过程中平滑演变。这一结果意味着，在超过相变时间后，类别虽已改变，但生成的样本可能仍由原始图像的低层元素构成。我们通过对无类别条件ImageNet扩散模型的数值实验验证了这些理论见解。我们的分析刻画了扩散模型中时间与尺度之间的关系，并提出了生成模型作为模拟组合数据特性的强大工具。