Recent advances in neural rendering have improved both training and rendering times by orders of magnitude. While these methods demonstrate state-of-the-art quality and speed, they are designed for photogrammetry of static scenes and do not generalize well to freely moving humans in the environment. In this work, we introduce Human Gaussian Splats (HUGS) that represents an animatable human together with the scene using 3D Gaussian Splatting (3DGS). Our method takes only a monocular video with a small number of (50-100) frames, and it automatically learns to disentangle the static scene and a fully animatable human avatar within 30 minutes. We utilize the SMPL body model to initialize the human Gaussians. To capture details that are not modeled by SMPL (e.g. cloth, hairs), we allow the 3D Gaussians to deviate from the human body model. Utilizing 3D Gaussians for animated humans brings new challenges, including the artifacts created when articulating the Gaussians. We propose to jointly optimize the linear blend skinning weights to coordinate the movements of individual Gaussians during animation. Our approach enables novel-pose synthesis of human and novel view synthesis of both the human and the scene. We achieve state-of-the-art rendering quality with a rendering speed of 60 FPS while being ~100x faster to train over previous work. Our code will be announced here: https://github.com/apple/ml-hugs

翻译：摘要：神经渲染领域的最新进展将训练和渲染时间提升了数个数量级。虽然这些方法在质量和速度上达到了顶尖水平，但它们专为静态场景的摄影测量设计，无法很好地推广到环境中自由移动的人体。本文提出人体高斯点云（HUGS），该方法利用三维高斯点云（3DGS）将可动画化的人体与场景一同表示。我们的方法仅需一段包含少量（50-100帧）帧的单目视频，即可在30分钟内自动学习分离静态场景与完全可动画化的人体化身。我们利用SMPL人体模型初始化人体高斯点，为捕获SMPL未建模的细节（如衣物、毛发），允许三维高斯点偏离人体模型。将三维高斯点用于动画人体带来了新挑战，包括在驱动高斯点关节时产生的伪影。我们提出联合优化线性混合蒙皮权重，以协调动画过程中各高斯点的运动。本方法能够实现人体的新姿态合成以及人与场景的新视角合成，在60 FPS的渲染速度下达到顶尖渲染质量，且训练速度比先前工作快约100倍。代码将发布于：https://github.com/apple/ml-hugs