Temporal Action Localization (TAL) aims to identify actions' start, end, and class labels in untrimmed videos. While recent advancements using transformer networks and Feature Pyramid Networks (FPN) have enhanced visual feature recognition in TAL tasks, less progress has been made in the integration of audio features into such frameworks. This paper introduces the Multi-Resolution Audio-Visual Feature Fusion (MRAV-FF), an innovative method to merge audio-visual data across different temporal resolutions. Central to our approach is a hierarchical gated cross-attention mechanism, which discerningly weighs the importance of audio information at diverse temporal scales. Such a technique not only refines the precision of regression boundaries but also bolsters classification confidence. Importantly, MRAV-FF is versatile, making it compatible with existing FPN TAL architectures and offering a significant enhancement in performance when audio data is available.

翻译：时间动作定位（Temporal Action Localization, TAL）旨在从未剪辑视频中识别动作的开始时间、结束时间及类别标签。尽管近期基于Transformer网络和特征金字塔网络（Feature Pyramid Networks, FPN）的研究进展提升了TAL任务中的视觉特征识别能力，但在将音频特征集成到此类框架方面进展有限。本文提出多分辨率音视频特征融合方法（Multi-Resolution Audio-Visual Feature Fusion, MRAV-FF），这是一种创新的跨不同时间分辨率融合音视频数据的技术。该方法的核心是分层门控交叉注意力机制，能够有区别地权衡不同时间尺度上音频信息的重要性。该技术不仅提升了回归边界的精度，还增强了分类置信度。重要的是，MRAV-FF具有通用性，可兼容现有基于FPN的TAL架构，并在提供音频数据时显著提升性能。