We present 3D Cinemagraphy, a new technique that marries 2D image animation with 3D photography. Given a single still image as input, our goal is to generate a video that contains both visual content animation and camera motion. We empirically find that naively combining existing 2D image animation and 3D photography methods leads to obvious artifacts or inconsistent animation. Our key insight is that representing and animating the scene in 3D space offers a natural solution to this task. To this end, we first convert the input image into feature-based layered depth images using predicted depth values, followed by unprojecting them to a feature point cloud. To animate the scene, we perform motion estimation and lift the 2D motion into the 3D scene flow. Finally, to resolve the problem of hole emergence as points move forward, we propose to bidirectionally displace the point cloud as per the scene flow and synthesize novel views by separately projecting them into target image planes and blending the results. Extensive experiments demonstrate the effectiveness of our method. A user study is also conducted to validate the compelling rendering results of our method.

翻译：我们提出了三维动态摄影技术（3D Cinemagraphy），一种将二维图像动画与三维摄影相融合的新技术。以单张静态图像为输入，我们的目标是生成同时包含视觉内容动画与相机运动的视频。实验发现，简单组合现有二维图像动画与三维摄影方法会导致明显伪影或动画不一致。我们的核心见解在于：在三维空间中表示并驱动场景能为该任务提供自然解决方案。为此，我们首先利用预测深度值将输入图像转换为基于特征的分层深度图像，随后通过反投影将其转换为特征点云。为实现场景动画，我们执行运动估计并将二维运动提升至三维场景流。最后，为解决点云向前运动时产生的空洞问题，我们提出沿场景流方向双向位移点云，并通过分别投影至目标图像平面并融合结果来合成新视角。大量实验验证了该方法的有效性，同时通过用户研究证明了本方法生成的渲染结果具有说服力。