Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understanding
arxiv(2024)
摘要
Understanding videos is one of the fundamental directions in computer vision
research, with extensive efforts dedicated to exploring various architectures
such as RNN, 3D CNN, and Transformers. The newly proposed architecture of state
space model, e.g., Mamba, shows promising traits to extend its success in long
sequence modeling to video modeling. To assess whether Mamba can be a viable
alternative to Transformers in the video understanding domain, in this work, we
conduct a comprehensive set of studies, probing different roles Mamba can play
in modeling videos, while investigating diverse tasks where Mamba could exhibit
superiority. We categorize Mamba into four roles for modeling videos, deriving
a Video Mamba Suite composed of 14 models/modules, and evaluating them on 12
video understanding tasks. Our extensive experiments reveal the strong
potential of Mamba on both video-only and video-language tasks while showing
promising efficiency-performance trade-offs. We hope this work could provide
valuable data points and insights for future research on video understanding.
Code is public: https://github.com/OpenGVLab/video-mamba-suite.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要