Modeling animatable human avatars from RGB videos is a long-standing and challenging problem. Recent works usually adopt MLP-based neural radiance fields (NeRF) to represent 3D humans, but it remains difficult for pure MLPs to regress pose-dependent garment details. To this end, we introduce Animatable Gaussians, a new avatar representation that leverages powerful 2D CNNs and 3D Gaussian splatting to create high-fidelity avatars. To associate 3D Gaussians with the animatable avatar, we learn a parametric template from the input videos, and then parameterize the template on two front \& back canonical Gaussian maps where each pixel represents a 3D Gaussian. The learned template is adaptive to the wearing garments for modeling looser clothes like dresses. Such template-guided 2D parameterization enables us to employ a powerful StyleGAN-based CNN to learn the pose-dependent Gaussian maps for modeling detailed dynamic appearances. Furthermore, we introduce a pose projection strategy for better generalization given novel poses. Overall, our method can create lifelike avatars with dynamic, realistic and generalized appearances. Experiments show that our method outperforms other state-of-the-art approaches. Code: https://github.com/lizhe00/AnimatableGaussians

翻译：从RGB视频中建模可动人体化身是一个长期且具有挑战性的问题。现有工作通常采用基于MLP的神经辐射场（NeRF）来表示三维人体，但纯MLP网络难以回归出姿态相关的服装细节。为此，我们提出可动高斯体（Animatable Gaussians），一种利用强大二维CNN和三维高斯溅射技术构建高保真化身的新型表示方法。为将三维高斯体与可动化身关联，我们从输入视频中学习参数化模板，进而将该模板参数化为前、后两张规范高斯贴图，其中每个像素对应一个三维高斯体。该学习模板能自适应穿着衣物，以建模连衣裙等宽松服饰。这种模板引导的二维参数化方案使我们得以借助强大的StyleGAN架构CNN学习姿态相关的高斯贴图，从而建模精细的动态外观。此外，我们提出姿态投影策略以提升对未见姿态的泛化能力。总体而言，本方法可生成具有动态、逼真且泛化外观的逼真化身。实验表明，本方法性能优于其他现有最优方法。代码地址：https://github.com/lizhe00/AnimatableGaussians