People can easily imagine the potential sound while seeing an event. This natural synchronization between audio and visual signals reveals their intrinsic correlations. To this end, we propose to learn the audio-visual correlations from the perspective of cross-modal generation in a self-supervised manner, the learned correlations can be then readily applied in multiple downstream tasks such as the audio-visual cross-modal localization and retrieval. We introduce a novel Variational AutoEncoder (VAE) framework that consists of Multiple encoders and a Shared decoder (MS-VAE) with an additional Wasserstein distance constraint to tackle the problem. Extensive experiments demonstrate that the optimized latent representation of the proposed MS-VAE can effectively learn the audio-visual correlations and can be readily applied in multiple audio-visual downstream tasks to achieve competitive performance even without any given label information during training.

翻译：人们在看到一个事件时可以很容易地想象出潜在的声音。 这种视听信号之间的自然同步性揭示了它们内在的关联性。 为此,我们提议从跨现代一代的角度以自我监督的方式学习视听相关关系,然后可以很容易地在多种下游任务中应用所学到的关联性,例如视听跨模式本地化和检索。我们引入了一个由多个编码器和共享解码器(MS-VAE)组成的新的动态自动编码器框架,为解决这一问题增加了瓦塞尔斯坦距离限制。广泛的实验表明,拟议的MS-VAE的优化潜在代表性可以有效地学习视听相关关系,并且可以很容易地应用于多个视听下游任务中,即使没有在培训期间提供任何标签信息,也能实现竞争性业绩。