In this paper, we propose an approach for synthesizing novel view images from a single RGBD (Red Green Blue-Depth) input. Novel view synthesis (NVS) is an interesting computer vision task with extensive applications. Methods using multiple images has been well-studied, exemplary ones include training scene-specific Neural Radiance Fields (NeRF), or leveraging multi-view stereo (MVS) and 3D rendering pipelines. However, both are either computationally intensive or non-generalizable across different scenes, limiting their practical value. Conversely, the depth information embedded in RGBD images unlocks 3D potential from a singular view, simplifying NVS. The widespread availability of compact, affordable stereo cameras, and even LiDARs in contemporary devices like smartphones, makes capturing RGBD images more accessible than ever. In our method, we convert an RGBD image into a point cloud and render it from a different viewpoint, then formulate the NVS task into an image translation problem. We leveraged generative adversarial networks to style-transfer the rendered image, achieving a result similar to a photograph taken from the new perspective. We explore both unsupervised learning using CycleGAN and supervised learning with Pix2Pix, and demonstrate the qualitative results. Our method circumvents the limitations of traditional multi-image techniques, holding significant promise for practical, real-time applications in NVS.

翻译：本文提出了一种从单张RGBD（红绿蓝-深度）输入合成新视角图像的方法。新视角合成（NVS）是计算机视觉中一项具有广泛应用价值且颇具趣味性的任务。基于多张图像的方法已得到充分研究，典型方案包括训练场景特定的神经辐射场（NeRF），或利用多视图立体视觉（MVS）及三维渲染管线。然而，这两种方法要么计算开销大，要么无法跨场景泛化，限制了其实用价值。相比之下，RGBD图像中蕴含的深度信息可从单一视角解锁三维潜力，从而简化NVS任务。紧凑且价格低廉的立体相机，乃至智能手机等当代设备中集成的激光雷达（LiDAR）的广泛普及，使得RGBD图像的获取比以往任何时候都更加便捷。在我们的方法中，首先将RGBD图像转换为点云，并从新视角对其渲染，进而将NVS任务转化为图像翻译问题。我们利用生成对抗网络对渲染图像进行风格迁移，使其结果接近新视角下拍摄的照片。我们分别探索了基于CycleGAN的无监督学习方法和基于Pix2Pix的有监督学习方法，并展示了定性结果。该方法规避了传统多图像技术的局限性，为NVS在实际即时应用场景中展现出显著潜力。