We present a pipeline for realistic embedding of virtual objects into footage of indoor scenes with focus on real-time AR applications. Our pipeline consists of two main components: A light estimator and a neural soft shadow texture generator. Our light estimation is based on deep neural nets and determines the main light direction, light color, ambient color and an opacity parameter for the shadow texture. Our neural soft shadow method encodes object-based realistic soft shadows as light direction dependent textures in a small MLP. We show that our pipeline can be used to integrate objects into AR scenes in a new level of realism in real-time. Our models are small enough to run on current mobile devices. We achieve runtimes of 9ms for light estimation and 5ms for neural shadows on an iPhone 11 Pro.

翻译：我们提出了一种面向室内场景视频的虚拟物体逼真嵌入流水线，重点面向实时增强现实应用。该流水线包含两个核心组件：光照估计器与神经软阴影纹理生成器。光照估计模块基于深度神经网络，可确定主光源方向、光源颜色、环境光颜色以及阴影纹理的不透明度参数。神经软阴影方法通过小型多层感知机（MLP）将基于物体的真实软阴影编码为与光源方向相关的纹理。实验表明，该流水线能够以全新层次的真实感将虚拟物体实时嵌入增强现实场景中。模型体积轻量，可运行于当前移动设备。在iPhone 11 Pro上，光照估计耗时9毫秒，神经阴影生成耗时5毫秒。