Audio-visual event (AVE) localization has attracted much attention in recent years. Most existing methods are often limited to independently encoding and classifying each video segment separated from the full video (which can be regarded as the segment-level representations of events). However, they ignore the semantic consistency of the event within the same full video (which can be considered as the video-level representations of events). In contrast to existing methods, we propose a novel video-level semantic consistency guidance network for the AVE localization task. Specifically, we propose an event semantic consistency modeling (ESCM) module to explore video-level semantic information for semantic consistency modeling. It consists of two components: a cross-modal event representation extractor (CERE) and an intra-modal semantic consistency enhancer (ISCE). CERE is proposed to obtain the event semantic information at the video level. Furthermore, ISCE takes video-level event semantics as prior knowledge to guide the model to focus on the semantic continuity of an event within each modality. Moreover, we propose a new negative pair filter loss to encourage the network to filter out the irrelevant segment pairs and a new smooth loss to further increase the gap between different categories of events in the weakly-supervised setting. We perform extensive experiments on the public AVE dataset and outperform the state-of-the-art methods in both fully- and weakly-supervised settings, thus verifying the effectiveness of our method.The code is available at https://github.com/Bravo5542/VSCG.

翻译：音视频事件（AVE）定位近年来备受关注。现有方法大多局限于对从完整视频中分离出的每个视频片段进行独立编码和分类（可视为事件的片段级表示），却忽视了同一完整视频内事件的语义一致性（即事件的视频级表示）。与现有方法不同，我们提出一种新颖的视频级语义一致性引导网络用于AVE定位任务。具体而言，我们设计了一个事件语义一致性建模（ESCM）模块，用于探索视频级语义信息以实现语义一致性建模。该模块包含两个组成部分：跨模态事件表征提取器（CERE）与模态内语义一致性增强器（ISCE）。CERE用于获取视频级的事件语义信息，而ISCE则将视频级事件语义作为先验知识，引导模型关注每个模态内事件的语义连续性。此外，我们提出一种新的负样本对过滤损失以鼓励网络过滤无关片段对，并提出一种新的平滑损失以进一步增大弱监督设置下不同类别事件之间的间隔。我们在公开AVE数据集上进行了大量实验，在完全监督和弱监督设置下均取得了优于现有最优方法的结果，验证了所提方法的有效性。代码已开源在 https://github.com/Bravo5542/VSCG。