One of the most critical factors in achieving sharp Novel View Synthesis (NVS) using neural field methods like Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) is the quality of the training images. However, Conventional RGB cameras are susceptible to motion blur. In contrast, neuromorphic cameras like event and spike cameras inherently capture more comprehensive temporal information, which can provide a sharp representation of the scene as additional training data. Recent methods have explored the integration of event cameras to improve the quality of NVS. The event-RGB approaches have some limitations, such as high training costs and the inability to work effectively in the background. Instead, our study introduces a new method that uses the spike camera to overcome these limitations. By considering texture reconstruction from spike streams as ground truth, we design the Texture from Spike (TfS) loss. Since the spike camera relies on temporal integration instead of temporal differentiation used by event cameras, our proposed TfS loss maintains manageable training costs. It handles foreground objects with backgrounds simultaneously. We also provide a real-world dataset captured with our spike-RGB camera system to facilitate future research endeavors. We conduct extensive experiments using synthetic and real-world datasets to demonstrate that our design can enhance novel view synthesis across NeRF and 3DGS. The code and dataset will be made available for public access.

翻译：实现基于神经辐射场（NeRF）和三维高斯泼溅法（3DGS）等神经场方法的锐利新视角合成（NVS），最关键因素之一是训练图像的质量。然而，传统RGB相机易受运动模糊影响。相比之下，事件相机和脉冲相机等神经形态相机能够自然捕获更全面的时间信息，可作为附加训练数据提供场景的锐利表征。现有方法已探索利用事件相机提升NVS质量，但事件-RGB融合方法存在训练成本高、无法有效处理背景区域等局限性。本研究提出一种基于脉冲相机的新方法以克服上述局限。通过将脉冲流纹理重建结果视为地面真值，我们设计了脉冲纹理损失函数（TfS）。由于脉冲相机基于时间积分机制（而非事件相机采用的时间差分机制），所提出的TfS损失能维持可控的训练成本，并同步处理前景物体与背景。我们还提供了基于自主研发的脉冲-RGB相机系统采集的真实场景数据集，以推动未来研究。通过合成数据集与真实场景数据集的广泛实验证明，本设计可有效提升基于NeRF与3DGS的新视角合成质量。代码与数据集将开源发布。