A Unified Editing Method for Co-Speech Gesture Generation via Diffusion Inversion
arxiv(2024)
摘要
Diffusion models have shown great success in generating high-quality
co-speech gestures for interactive humanoid robots or digital avatars from
noisy input with the speech audio or text as conditions. However, they rarely
focus on providing rich editing capabilities for content creators other than
high-level specialized measures like style conditioning. To resolve this, we
propose a unified framework utilizing diffusion inversion that enables
multi-level editing capabilities for co-speech gesture generation without
re-training. The method takes advantage of two key capabilities of invertible
diffusion models. The first is that through inversion, we can reconstruct the
intermediate noise from gestures and regenerate new gestures from the noise.
This can be used to obtain gestures with high-level similarities to the
original gestures for different speech conditions. The second is that this
reconstruction reduces activation caching requirements during gradient
calculation, making the direct optimization on input noises possible on current
hardware with limited memory. With different loss functions designed for, e.g.,
joint rotation or velocity, we can control various low-level details by
automatically tweaking the input noises through optimization. Extensive
experiments on multiple use cases show that this framework succeeds in unifying
high-level and low-level co-speech gesture editing.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要