We introduce pixelSplat, a feed-forward model that learns to reconstruct 3D radiance fields parameterized by 3D Gaussian primitives from pairs of images. Our model features real-time and memory-efficient rendering for scalable training as well as fast 3D reconstruction at inference time. To overcome local minima inherent to sparse and locally supported representations, we predict a dense probability distribution over 3D and sample Gaussian means from that probability distribution. We make this sampling operation differentiable via a reparameterization trick, allowing us to back-propagate gradients through the Gaussian splatting representation. We benchmark our method on wide-baseline novel view synthesis on the real-world RealEstate10k and ACID datasets, where we outperform state-of-the-art light field transformers and accelerate rendering by 2.5 orders of magnitude while reconstructing an interpretable and editable 3D radiance field.

翻译：我们提出pixelSplat，一种前馈式模型，学习从图像对重建由3D高斯基元参数化的3D辐射场。该模型具备实时高效的内存渲染能力，支持可扩展训练以及快速推理时的3D重建。为克服稀疏且局部支持表征固有的局部最小值问题，我们预测3D空间的密集概率分布，并从中采样高斯均值。通过重参数化技巧使该采样操作可微分，从而能够将梯度反向传播至高斯泼溅表征。我们在真实场景的RealEstate10k和ACID数据集上进行宽基线新视角合成基准测试，性能超越最先进的光场变换器，渲染速度提升2.5个数量级，同时重建出可解释、可编辑的3D辐射场。