Existing homography and optical flow methods are erroneous in challenging scenes, such as fog, rain, night, and snow because the basic assumptions such as brightness and gradient constancy are broken. To address this issue, we present an unsupervised learning approach that fuses gyroscope into homography and optical flow learning. Specifically, we first convert gyroscope readings into motion fields named gyro field. Second, we design a self-guided fusion module (SGF) to fuse the background motion extracted from the gyro field with the optical flow and guide the network to focus on motion details. Meanwhile, we propose a homography decoder module (HD) to combine gyro field and intermediate results of SGF to produce the homography. To the best of our knowledge, this is the first deep learning framework that fuses gyroscope data and image content for both deep homography and optical flow learning. To validate our method, we propose a new dataset that covers regular and challenging scenes. Experiments show that our method outperforms the state-of-the-art methods in both regular and challenging scenes.

翻译：现有单应性与光流方法在雾、雨、夜、雪等挑战性场景中存在误差，原因是亮度恒定和梯度恒定等基本假设被破坏。为解决此问题，我们提出一种将陀螺仪融合进单应性与光流学习的无监督学习方法。具体而言，首先将陀螺仪读数转换为名为陀螺场（gyro field）的运动场；其次设计自引导融合模块（SGF），将陀螺场中提取的背景运动与光流融合，引导网络聚焦于运动细节；同时提出单应性解码模块（HD），结合陀螺场与SGF中间结果生成单应性。据我们所知，这是首个将陀螺仪数据与图像内容融合用于深度单应性与光流学习的深度学习框架。为验证方法有效性，我们构建了包含常规与挑战性场景的新数据集。实验表明，本方法在常规与挑战性场景中均优于现有最先进方法。