Snapshot Compressive Imaging (SCI) offers a possibility for capturing information in high-speed dynamic scenes, requiring efficient reconstruction method to recover scene information. Despite promising results, current deep learning-based and NeRF-based reconstruction methods face challenges: 1) deep learning-based reconstruction methods struggle to maintain 3D structural consistency within scenes, and 2) NeRF-based reconstruction methods still face limitations in handling dynamic scenes. To address these challenges, we propose SCIGS, a variant of 3DGS, and develop a primitive-level transformation network that utilizes camera pose stamps and Gaussian primitive coordinates as embedding vectors. This approach resolves the necessity of camera pose in vanilla 3DGS and enhances multi-view 3D structural consistency in dynamic scenes by utilizing transformed primitives. Additionally, a high-frequency filter is introduced to eliminate the artifacts generated during the transformation. The proposed SCIGS is the first to reconstruct a 3D explicit scene from a single compressed image, extending its application to dynamic 3D scenes. Experiments on both static and dynamic scenes demonstrate that SCIGS not only enhances SCI decoding but also outperforms current state-of-the-art methods in reconstructing dynamic 3D scenes from a single compressed image. The code will be made available upon publication.
翻译:快照压缩成像(SCI)为捕获高速动态场景信息提供了可能,这需要高效的重建方法来恢复场景信息。尽管现有基于深度学习和基于NeRF的重建方法取得了有希望的结果,但仍面临挑战:1)基于深度学习的重建方法难以保持场景内的3D结构一致性;2)基于NeRF的重建方法在处理动态场景时仍存在局限。为解决这些挑战,我们提出了SCIGS——3DGS的变体,并开发了一个基元级变换网络,该网络利用相机姿态时间戳和高斯基元坐标作为嵌入向量。该方法解决了原始3DGS对相机姿态的依赖,并通过变换后的基元增强了动态场景中多视角的3D结构一致性。此外,我们引入了高频滤波器以消除变换过程中产生的伪影。所提出的SCIGS是首个从单张压缩图像重建3D显式场景的方法,将其应用扩展至动态3D场景。在静态和动态场景上的实验表明,SCIGS不仅提升了SCI解码性能,而且在从单张压缩图像重建动态3D场景方面超越了当前最先进的方法。代码将在论文发表时公开。