Neural Implicit Representation (NIR) has recently gained significant attention due to its remarkable ability to encode complex and high-dimensional data into representation space and easily reconstruct it through a trainable mapping function. However, NIR methods assume a one-to-one mapping between the target data and representation models regardless of data relevancy or similarity. This results in poor generalization over multiple complex data and limits their efficiency and scalability. Motivated by continual learning, this work investigates how to accumulate and transfer neural implicit representations for multiple complex video data over sequential encoding sessions. To overcome the limitation of NIR, we propose a novel method, Progressive Fourier Neural Representation (PFNR), that aims to find an adaptive and compact sub-module in Fourier space to encode videos in each training session. This sparsified neural encoding allows the neural network to hold free weights, enabling an improved adaptation for future videos. In addition, when learning a representation for a new video, PFNR transfers the representation of previous videos with frozen weights. This design allows the model to continuously accumulate high-quality neural representations for multiple videos while ensuring lossless decoding that perfectly preserves the learned representations for previous videos. We validate our PFNR method on the UVG8/17 and DAVIS50 video sequence benchmarks and achieve impressive performance gains over strong continual learning baselines. The PFNR code is available at https://github.com/ihaeyong/PFNR.git.

翻译：神经隐式表示（NIR）近期因其将复杂高维数据编码至表示空间并通过可训练映射函数轻松重建的卓越能力而受到广泛关注。然而，NIR方法假设目标数据与表示模型之间存在一对一映射关系，而忽略数据间的关联性或相似性。这导致其在处理多个复杂数据时泛化能力不足，限制了效率与可扩展性。受持续学习启发，本研究探索如何在序列编码会话中累积并迁移多个复杂视频数据的神经隐式表示。为突破NIR的局限性，我们提出新型方法——渐进式傅里叶神经表示（PFNR），旨在傅里叶空间中为每个训练会话的视频编码寻找自适应且紧凑的子模块。这种稀疏化神经编码使神经网络保留自由权重，从而提升对未来视频的自适应能力。此外，在学习新视频表示时，PFNR以冻结权重的方式迁移先前视频的表示。该设计使模型能够持续累积多个视频的高质量神经表示，同时确保无损解码，完美保留先前视频已习得的表示。我们在UVG8/17和DAVIS50视频序列基准上验证了PFNR方法，相较于强持续学习基线取得了显著性能提升。PFNR代码开源于https://github.com/ihaeyong/PFNR.git。