LoSA: Long-Short-range Adapter for Scaling End-to-End Temporal Action Localization
arxiv(2024)
摘要
Temporal Action Localization (TAL) involves localizing and classifying action
snippets in an untrimmed video. The emergence of large video foundation models
has led RGB-only video backbones to outperform previous methods needing both
RGB and optical flow modalities. Leveraging these large models is often limited
to training only the TAL head due to the prohibitively large GPU memory
required to adapt the video backbone for TAL. To overcome this limitation, we
introduce LoSA, the first memory-and-parameter-efficient backbone adapter
designed specifically for TAL to handle untrimmed videos. LoSA specializes for
TAL by introducing Long-Short-range Adapters that adapt the intermediate layers
of the video backbone over different temporal ranges. These adapters run
parallel to the video backbone to significantly reduce memory footprint. LoSA
also includes Long-Short-range Fusion that strategically combines the output of
these adapters from the video backbone layers to enhance the video features
provided to the TAL head. Experiments show that LoSA significantly outperforms
all existing methods on standard TAL benchmarks, THUMOS-14 and
ActivityNet-v1.3, by scaling end-to-end backbone adaptation to
billion-parameter-plus models like VideoMAEv2 (ViT-g) and leveraging them
beyond head-only transfer learning.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要