Neural Radiance Fields (NeRFs) have demonstrated the remarkable potential of neural networks to capture the intricacies of 3D objects. By encoding the shape and color information within neural network weights, NeRFs excel at producing strikingly sharp novel views of 3D objects. Recently, numerous generalizations of NeRFs utilizing generative models have emerged, expanding its versatility. In contrast, Gaussian Splatting (GS) offers a similar renders quality with faster training and inference as it does not need neural networks to work. We encode information about the 3D objects in the set of Gaussian distributions that can be rendered in 3D similarly to classical meshes. Unfortunately, GS are difficult to condition since they usually require circa hundred thousand Gaussian components. To mitigate the caveats of both models, we propose a hybrid model that uses GS representation of the 3D object's shape and NeRF-based encoding of color and opacity. Our model uses Gaussian distributions with trainable positions (i.e. means of Gaussian), shape (i.e. covariance of Gaussian), color and opacity, and neural network, which takes parameters of Gaussian and viewing direction to produce changes in color and opacity. Consequently, our model better describes shadows, light reflections, and transparency of 3D objects.

翻译：神经辐射场（NeRFs）已展现出神经网络在捕捉三维物体细节方面的显著潜力。通过将形状与颜色信息编码至神经网络权重中，NeRFs能够生成令人惊叹的清晰三维物体新视角。近期，利用生成模型的NeRFs泛化方法层出不穷，进一步拓展了其应用范围。相比之下，高斯泼溅（GS）无需依赖神经网络即可实现相近的渲染质量，且训练与推理速度更快。该方法将三维物体的信息编码至高斯分布集合中，使其能够像传统网格一样进行三维渲染。然而，GS模型通常需要约十万个高斯分量，因此难以进行条件控制。为缓解两种模型的局限性，我们提出一种混合模型：采用GS表示三维物体形状，并基于NeRF编码颜色与不透明度。该模型使用具有可训练位置（即高斯均值）、形状（即高斯协方差）、颜色与不透明度的高斯分布，并引入神经网络，依据高斯参数与视角方向动态调整颜色与不透明度。由此，我们的模型能更精准地描述三维物体的阴影、光反射与透明效果。