Multi-modal fusion is proven to be an effective method to improve the accuracy and robustness of speaker tracking, especially in complex scenarios. However, how to combine the heterogeneous information and exploit the complementarity of multi-modal signals remains a challenging issue. In this paper, we propose a novel Multi-modal Perception Tracker (MPT) for speaker tracking using both audio and visual modalities. Specifically, a novel acoustic map based on spatial-temporal Global Coherence Field (stGCF) is first constructed for heterogeneous signal fusion, which employs a camera model to map audio cues to the localization space consistent with the visual cues. Then a multi-modal perception attention network is introduced to derive the perception weights that measure the reliability and effectiveness of intermittent audio and video streams disturbed by noise. Moreover, a unique cross-modal self-supervised learning method is presented to model the confidence of audio and visual observations by leveraging the complementarity and consistency between different modalities. Experimental results show that the proposed MPT achieves 98.6% and 78.3% tracking accuracy on the standard and occluded datasets, respectively, which demonstrates its robustness under adverse conditions and outperforms the current state-of-the-art methods.

翻译：事实证明,多式聚合是提高发言者跟踪准确性和稳健性的有效方法,特别是在复杂情景中。然而,如何将各种信息结合起来并利用多式信号的互补性仍然是一个挑战性问题。在本文件中,我们提议采用视听模式和视觉模式为发言者跟踪提供新型的多式感知跟踪器(MPT),具体地说,根据时空空间全球一致性域(stGCF),首次为混杂信号聚合制作了一部新的声学地图,该图使用照相机模型绘制与视觉提示一致的本地化空间音频提示。随后,引入了多式感知关注网络,以得出测量噪音干扰的间歇音频和视频流的可靠性和有效性的感知力重。此外,还介绍了一种独特的跨式自我监督的自闭式学习方法,通过利用不同模式之间的互补性和一致性来模拟声学和视觉观测的信心。实验结果表明,拟议的MPT达到98.6%和78.3%的感应跟踪标准和隐蔽数据集的准确性,这分别表明了其在当前不利条件下和状态下的稳性。