Talk3D: High-Fidelity Talking Portrait Synthesis via Personalized 3D Generative Prior
CoRR(2024)
摘要
Recent methods for audio-driven talking head synthesis often optimize neural
radiance fields (NeRF) on a monocular talking portrait video, leveraging its
capability to render high-fidelity and 3D-consistent novel-view frames.
However, they often struggle to reconstruct complete face geometry due to the
absence of comprehensive 3D information in the input monocular videos. In this
paper, we introduce a novel audio-driven talking head synthesis framework,
called Talk3D, that can faithfully reconstruct its plausible facial geometries
by effectively adopting the pre-trained 3D-aware generative prior. Given the
personalized 3D generative model, we present a novel audio-guided attention
U-Net architecture that predicts the dynamic face variations in the NeRF space
driven by audio. Furthermore, our model is further modulated by audio-unrelated
conditioning tokens which effectively disentangle variations unrelated to audio
features. Compared to existing methods, our method excels in generating
realistic facial geometries even under extreme head poses. We also conduct
extensive experiments showing our approach surpasses state-of-the-art
benchmarks in terms of both quantitative and qualitative evaluations.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要