A Phase Transition in Diffusion Models Reveals the Hierarchical Nature of Data
CoRR(2024)
摘要
Understanding the structure of real data is paramount in advancing modern
deep-learning methodologies. Natural data such as images are believed to be
composed of features organised in a hierarchical and combinatorial manner,
which neural networks capture during learning. Recent advancements show that
diffusion models can generate high-quality images, hinting at their ability to
capture this underlying structure. We study this phenomenon in a hierarchical
generative model of data. We find that the backward diffusion process acting
after a time t is governed by a phase transition at some threshold time,
where the probability of reconstructing high-level features, like the class of
an image, suddenly drops. Instead, the reconstruction of low-level features,
such as specific details of an image, evolves smoothly across the whole
diffusion process. This result implies that at times beyond the transition, the
class has changed but the generated sample may still be composed of low-level
elements of the initial image. We validate these theoretical insights through
numerical experiments on class-unconditional ImageNet diffusion models. Our
analysis characterises the relationship between time and scale in diffusion
models and puts forward generative models as powerful tools to model
combinatorial data properties.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要