In recent advancements in novel view synthesis, generalizable Neural Radiance Fields (NeRF) based methods applied to human subjects have shown remarkable results in generating novel views from few images. However, this generalization ability cannot capture the underlying structural features of the skeleton shared across all instances. Building upon this, we introduce HFNeRF: a novel generalizable human feature NeRF aimed at generating human biomechanic features using a pre-trained image encoder. While previous human NeRF methods have shown promising results in the generation of photorealistic virtual avatars, such methods lack underlying human structure or biomechanic features such as skeleton or joint information that are crucial for downstream applications including Augmented Reality (AR)/Virtual Reality (VR). HFNeRF leverages 2D pre-trained foundation models toward learning human features in 3D using neural rendering, and then volume rendering towards generating 2D feature maps. We evaluate HFNeRF in the skeleton estimation task by predicting heatmaps as features. The proposed method is fully differentiable, allowing to successfully learn color, geometry, and human skeleton in a simultaneous manner. This paper presents preliminary results of HFNeRF, illustrating its potential in generating realistic virtual avatars with biomechanic features using NeRF.

翻译：在新视角合成的最新进展中，基于可泛化神经辐射场（NeRF）的方法应用于人体对象时，已展现出从少量图像生成新视角的显著成果。然而，这种泛化能力无法捕捉所有实例共享的骨架底层结构特征。在此基础上，我们提出HFNeRF：一种新型可泛化人体特征NeRF，旨在利用预训练图像编码器生成人体生物力学特征。尽管此前的人体NeRF方法在生成逼真虚拟化身方面已取得可喜成果，但此类方法缺乏底层人体结构或生物力学特征（如骨架或关节信息），而这对于增强现实（AR）/虚拟现实（VR）等下游应用至关重要。HFNeRF利用二维预训练基础模型，通过神经渲染学习三维人体特征，进而采用体渲染生成二维特征图。我们通过预测热图作为特征，在骨架估计任务中评估HFNeRF。所提方法完全可微分，能够同时成功学习颜色、几何形状和人体骨架。本文展示了HFNeRF的初步结果，阐明了其利用NeRF生成具有生物力学特征的逼真虚拟化身的潜力。