3D-Scenedreamer: Text-Driven 3D-Consistent Scene Generation

Text-driven 3D scene generation techniques have made rapid progress in recentyears. Their success is mainly attributed to using existing generative modelsto iteratively perform image warping and inpainting to generate 3D scenes.However, these methods heavily rely on the outputs of existing models, leadingto error accumulation in geometry and appearance that prevent the models frombeing used in various scenarios (e.g., outdoor and unreal scenarios). Toaddress this limitation, we generatively refine the newly generated local viewsby querying and aggregating global 3D information, and then progressivelygenerate the 3D scene. Specifically, we employ a tri-plane features-based NeRFas a unified representation of the 3D scene to constrain global 3D consistency,and propose a generative refinement network to synthesize new contents withhigher quality by exploiting the natural image prior from 2D diffusion model aswell as the global 3D information of the current scene. Our extensiveexperiments demonstrate that, in comparison to previous methods, our approachsupports wide variety of scene generation and arbitrary camera trajectorieswith improved visual quality and 3D consistency.