Neural Radiance Fields (NeRFs) have demonstrated the remarkable potential of neural networks to capture the intricacies of 3D objects. By encoding the shape and color information within neural network weights, NeRFs excel at producing strikingly sharp novel views of 3D objects. Recently, numerous generalizations of NeRFs utilizing generative models have emerged, expanding its versatility. In contrast, Gaussian Splatting (GS) offers a similar renders quality with faster training and inference as it does not need neural networks to work. We encode information about the 3D objects in the set of Gaussian distributions that can be rendered in 3D similarly to classical meshes. Unfortunately, GS are difficult to condition since they usually require circa hundred thousand Gaussian components. To mitigate the caveats of both models, we propose a hybrid model that uses GS representation of the 3D object's shape and NeRF-based encoding of color and opacity. Our model uses Gaussian distributions with trainable positions (i.e. means of Gaussian), shape (i.e. covariance of Gaussian), color and opacity, and neural network, which takes parameters of Gaussian and viewing direction to produce changes in color and opacity. Consequently, our model better describes shadows, light reflections, and transparency of 3D objects.

翻译：神经辐射场（NeRFs）展现了神经网络在捕捉三维物体复杂性方面的巨大潜力。通过在神经网络权重中编码形状与颜色信息，NeRFs能够出色地生成三维物体极其清晰的新视角图像。近年来，基于生成模型的NeRFs泛化方法不断涌现，进一步拓展了其适用性。相比之下，高斯溅射（GS）可在不依赖神经网络的情况下实现相似渲染质量，且训练与推理速度更快。我们将三维物体信息编码为一组高斯分布，这些分布可像传统网格一样实现三维渲染。然而，GS模型通常需要数十万个高斯分量，导致其条件控制困难。为缓解两种模型的局限性，我们提出一种混合模型：采用GS表示三维物体形状，并基于NeRF编码颜色与透明度。该模型使用位置（高斯均值）、形状（高斯协方差）、颜色和透明度均可训练的高斯分布，同时通过神经网络根据高斯参数与视角方向动态调整颜色与透明度。因此，本模型能更精确地描述三维物体的阴影、光反射及透明度特性。