MTGA: Multi-view Temporal Granularity aligned Aggregation for Event-based Lip-reading
CoRR(2024)
摘要
Lip-reading is to utilize the visual information of the speaker's lip
movements to recognize words and sentences. Existing event-based lip-reading
solutions integrate different frame rate branches to learn spatio-temporal
features of varying granularities. However, aggregating events into event
frames inevitably leads to the loss of fine-grained temporal information within
frames. To remedy this drawback, we propose a novel framework termed Multi-view
Temporal Granularity aligned Aggregation (MTGA). Specifically, we first present
a novel event representation method, namely time-segmented voxel graph list,
where the most significant local voxels are temporally connected into a graph
list. Then we design a spatio-temporal fusion module based on temporal
granularity alignment, where the global spatial features extracted from event
frames, together with the local relative spatial and temporal features
contained in voxel graph list are effectively aligned and integrated. Finally,
we design a temporal aggregation module that incorporates positional encoding,
which enables the capture of local absolute spatial and global temporal
information. Experiments demonstrate that our method outperforms both the
event-based and video-based lip-reading counterparts. Our code will be publicly
available.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要