Neural Network-Based Score Estimation in Diffusion Models: Optimization and Generalization
CoRR(2024)
摘要
Diffusion models have emerged as a powerful tool rivaling GANs in generating
high-quality samples with improved fidelity, flexibility, and robustness. A key
component of these models is to learn the score function through score
matching. Despite empirical success on various tasks, it remains unclear
whether gradient-based algorithms can learn the score function with a provable
accuracy. As a first step toward answering this question, this paper
establishes a mathematical framework for analyzing score estimation using
neural networks trained by gradient descent. Our analysis covers both the
optimization and the generalization aspects of the learning procedure. In
particular, we propose a parametric form to formulate the denoising
score-matching problem as a regression with noisy labels. Compared to the
standard supervised learning setup, the score-matching problem introduces
distinct challenges, including unbounded input, vector-valued output, and an
additional time variable, preventing existing techniques from being applied
directly. In this paper, we show that with a properly designed neural network
architecture, the score function can be accurately approximated by a
reproducing kernel Hilbert space induced by neural tangent kernels.
Furthermore, by applying an early-stopping rule for gradient descent and
leveraging certain coupling arguments between neural network training and
kernel regression, we establish the first generalization error (sample
complexity) bounds for learning the score function despite the presence of
noise in the observations. Our analysis is grounded in a novel parametric form
of the neural network and an innovative connection between score matching and
regression analysis, facilitating the application of advanced statistical and
optimization techniques.
更多查看译文
关键词
diffusion models,score estimation,neural networks,neural tangent kernels
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要