A SOUND APPROACH: Using Large Language Models to generate audio descriptions for egocentric text-audio retrieval
ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)(2024)
摘要
Video databases from the internet are a valuable source of text-audio
retrieval datasets. However, given that sound and vision streams represent
different "views" of the data, treating visual descriptions as audio
descriptions is far from optimal. Even if audio class labels are present, they
commonly are not very detailed, making them unsuited for text-audio retrieval.
To exploit relevant audio information from video-text datasets, we introduce a
methodology for generating audio-centric descriptions using Large Language
Models (LLMs). In this work, we consider the egocentric video setting and
propose three new text-audio retrieval benchmarks based on the EpicMIR and
EgoMCQ tasks, and on the EpicSounds dataset. Our approach for obtaining
audio-centric descriptions gives significantly higher zero-shot performance
than using the original visual-centric descriptions. Furthermore, we show that
using the same prompts, we can successfully employ LLMs to improve the
retrieval on EpicSounds, compared to using the original audio class labels of
the dataset. Finally, we confirm that LLMs can be used to determine the
difficulty of identifying the action associated with a sound.
更多查看译文
关键词
text-audio retrieval,large language models,generated audio descriptions,egocentric data
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要