Diffusion Models are Geometry Critics: Single Image 3D Editing Using Pre-Trained Diffusion Priors
arxiv(2024)
摘要
We propose a novel image editing technique that enables 3D manipulations on
single images, such as object rotation and translation. Existing 3D-aware image
editing approaches typically rely on synthetic multi-view datasets for training
specialized models, thus constraining their effectiveness on open-domain images
featuring significantly more varied layouts and styles. In contrast, our method
directly leverages powerful image diffusion models trained on a broad spectrum
of text-image pairs and thus retain their exceptional generalization abilities.
This objective is realized through the development of an iterative novel view
synthesis and geometry alignment algorithm. The algorithm harnesses diffusion
models for dual purposes: they provide appearance prior by predicting novel
views of the selected object using estimated depth maps, and they act as a
geometry critic by correcting misalignments in 3D shapes across the sampled
views. Our method can generate high-quality 3D-aware image edits with large
viewpoint transformations and high appearance and shape consistency with the
input image, pushing the boundaries of what is possible with single-image
3D-aware editing.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要