Bootstrap3D: Improving 3D Content Creation with Synthetic Data
CoRR(2024)
Abstract
Recent years have witnessed remarkable progress in multi-view diffusion
models for 3D content creation. However, there remains a significant gap in
image quality and prompt-following ability compared to 2D diffusion models. A
critical bottleneck is the scarcity of high-quality 3D assets with detailed
captions. To address this challenge, we propose Bootstrap3D, a novel framework
that automatically generates an arbitrary quantity of multi-view images to
assist in training multi-view diffusion models. Specifically, we introduce a
data generation pipeline that employs (1) 2D and video diffusion models to
generate multi-view images based on constructed text prompts, and (2) our
fine-tuned 3D-aware MV-LLaVA for filtering high-quality data and rewriting
inaccurate captions. Leveraging this pipeline, we have generated 1 million
high-quality synthetic multi-view images with dense descriptive captions to
address the shortage of high-quality 3D data. Furthermore, we present a
Training Timestep Reschedule (TTR) strategy that leverages the denoising
process to learn multi-view consistency while maintaining the original 2D
diffusion prior. Extensive experiments demonstrate that Bootstrap3D can
generate high-quality multi-view images with superior aesthetic quality,
image-text alignment, and maintained view consistency.
MoreTranslated text
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined