TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models
CVPR 2024(2024)
摘要
Recent advances in text-to-video generation have demonstrated the utility of
powerful diffusion models. Nevertheless, the problem is not trivial when
shaping diffusion models to animate static image (i.e., image-to-video
generation). The difficulty originates from the aspect that the diffusion
process of subsequent animated frames should not only preserve the faithful
alignment with the given image but also pursue temporal coherence among
adjacent frames. To alleviate this, we present TRIP, a new recipe of
image-to-video diffusion paradigm that pivots on image noise prior derived from
static image to jointly trigger inter-frame relational reasoning and ease the
coherent temporal modeling via temporal residual learning. Technically, the
image noise prior is first attained through one-step backward diffusion process
based on both static image and noised video latent codes. Next, TRIP executes a
residual-like dual-path scheme for noise prediction: 1) a shortcut path that
directly takes image noise prior as the reference noise of each frame to
amplify the alignment between the first frame and subsequent frames; 2) a
residual path that employs 3D-UNet over noised video and static image latent
codes to enable inter-frame relational reasoning, thereby easing the learning
of the residual noise for each frame. Furthermore, both reference and residual
noise of each frame are dynamically merged via attention mechanism for final
video generation. Extensive experiments on WebVid-10M, DTDB and MSR-VTT
datasets demonstrate the effectiveness of our TRIP for image-to-video
generation. Please see our project page at https://trip-i2v.github.io/TRIP/.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要