The images and sounds that we perceive undergo subtle but geometrically consistent changes as we rotate our heads. In this paper, we use these cues to solve a problem we call Sound Localization from Motion (SLfM): jointly estimating camera rotation and localizing sound sources. We learn to solve these tasks solely through self-supervision. A visual model predicts camera rotation from a pair of images, while an audio model predicts the direction of sound sources from binaural sounds. We train these models to generate predictions that agree with one another. At test time, the models can be deployed independently. To obtain a feature representation that is well-suited to solving this challenging problem, we also propose a method for learning an audio-visual representation through cross-view binauralization: estimating binaural sound from one view, given images and sound from another. Our model can successfully estimate accurate rotations on both real and synthetic scenes, and localize sound sources with accuracy competitive with state-of-the-art self-supervised approaches. Project site: https://ificl.github.io/SLfM/

翻译：我们感知到的图像和声音会随着头部旋转而产生细微但几何一致的变化。本文利用这些线索解决了一个我们称之为“基于运动的声源定位”（SLfM）的问题：联合估计相机旋转并定位声源。我们仅通过自监督学习来求解这些任务。视觉模型从图像对中预测相机旋转，而音频模型从双耳声音中预测声源方向。我们训练这些模型生成相互一致的预测。在测试时，这些模型可以独立部署。为了获得适合解决这一挑战性问题的特征表示，我们还提出了一种通过跨视角双耳化学习视听表示的方法：给定另一视角的图像和声音，估计当前视角的双耳声音。我们的模型能够在真实和合成场景中成功估计准确的旋转，并以与最先进自监督方法相竞争的精确定位声源。项目网站：https://ificl.github.io/SLfM/