MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokens
arxiv(2024)
摘要
This paper introduces MiniGPT4-Video, a multimodal Large Language Model (LLM)
designed specifically for video understanding. The model is capable of
processing both temporal visual and textual data, making it adept at
understanding the complexities of videos. Building upon the success of
MiniGPT-v2, which excelled in translating visual features into the LLM space
for single images and achieved impressive results on various image-text
benchmarks, this paper extends the model's capabilities to process a sequence
of frames, enabling it to comprehend videos. MiniGPT4-video does not only
consider visual content but also incorporates textual conversations, allowing
the model to effectively answer queries involving both visual and text
components. The proposed model outperforms existing state-of-the-art methods,
registering gains of 4.22
and TVQA benchmarks respectively. Our models and code have been made publicly
available here https://vision-cair.github.io/MiniGPT4-video/
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要