CMMD: Contrastive Multi-Modal Diffusion for Video-Audio Conditional Modeling
CoRR(2023)
摘要
We introduce a multi-modal diffusion model tailored for the bi-directional
conditional generation of video and audio. Recognizing the importance of
accurate alignment between video and audio events in multi-modal generation
tasks, we propose a joint contrastive training loss to enhance the
synchronization between visual and auditory occurrences. Our research
methodology involves conducting comprehensive experiments on multiple datasets
to thoroughly evaluate the efficacy of our proposed model. The assessment of
generation quality and alignment performance is carried out from various
angles, encompassing both objective and subjective metrics. Our findings
demonstrate that the proposed model outperforms the baseline, substantiating
its effectiveness and efficiency. Notably, the incorporation of the contrastive
loss results in improvements in audio-visual alignment, particularly in the
high-correlation video-to-audio generation task. These results indicate the
potential of our proposed model as a robust solution for improving the quality
and alignment of multi-modal generation, thereby contributing to the
advancement of video and audio conditional generation systems.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要