Music to Dance as Language Translation using Sequence Models
CoRR(2024)
摘要
Synthesising appropriate choreographies from music remains an open problem.
We introduce MDLT, a novel approach that frames the choreography generation
problem as a translation task. Our method leverages an existing data set to
learn to translate sequences of audio into corresponding dance poses. We
present two variants of MDLT: one utilising the Transformer architecture and
the other employing the Mamba architecture. We train our method on AIST++ and
PhantomDance data sets to teach a robotic arm to dance, but our method can be
applied to a full humanoid robot. Evaluation metrics, including Average Joint
Error and Frechet Inception Distance, consistently demonstrate that, when given
a piece of music, MDLT excels at producing realistic and high-quality
choreography. The code can be found at github.com/meowatthemoon/MDLT.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要