Deep learning has revolutionized the field of computer vision by introducing large scale neural networks with millions of parameters. Training these networks requires massive datasets and leads to intransparent models that can fail to generalize. At the other extreme, models designed from partial differential equations (PDEs) embed specialized domain knowledge into mathematical equations and usually rely on few manually chosen hyperparameters. This makes them transparent by construction and if designed and calibrated carefully, they can generalize well to unseen scenarios. In this paper, we show how to bring model- and data-driven approaches together by combining the explicit PDE-based approaches with convolutional neural networks to obtain the best of both worlds. We illustrate a joint architecture for the task of inpainting optical flow fields and show that the combination of model- and data-driven modeling leads to an effective architecture. Our model outperforms both fully explicit and fully data-driven baselines in terms of reconstruction quality, robustness and amount of required training data. Averaging the endpoint error across different mask densities, our method outperforms the explicit baselines by 11-27%, the GAN baseline by 47% and the Probabilisitic Diffusion baseline by 42%. With that, our method sets a new state of the art for inpainting of optical flow fields from random masks.

翻译：深度学习通过引入具有数百万参数的大规模神经网络，彻底改变了计算机视觉领域。训练这些网络需要海量数据集，并导致模型缺乏透明度，可能无法良好泛化。另一方面，基于偏微分方程设计的模型将特定领域知识嵌入数学方程，通常仅依赖少量手动选择的超参数。这使得它们在结构上具有透明度，若经过精心设计和校准，能够对未见场景表现出良好的泛化能力。本文展示了如何通过将显式PDE方法与卷积神经网络相结合，融合模型驱动与数据驱动方法的优势。我们针对光流场修复任务提出了一种联合架构，并证明模型驱动与数据驱动建模的结合能够形成高效架构。在重建质量、鲁棒性和所需训练数据量方面，我们的模型均优于完全显式与完全数据驱动的基线方法。通过平均不同掩码密度下的端点误差，我们的方法较显式基线提升11-27%，较GAN基线提升47%，较概率扩散基线提升42%。由此，我们的方法为随机掩码下的光流场修复任务确立了新的技术标杆。