Neural Implicit Representation (NIR) has recently gained significant attention due to its remarkable ability to encode complex and high-dimensional data into representation space and easily reconstruct it through a trainable mapping function. However, NIR methods assume a one-to-one mapping between the target data and representation models regardless of data relevancy or similarity. This results in poor generalization over multiple complex data and limits their efficiency and scalability. Motivated by continual learning, this work investigates how to accumulate and transfer neural implicit representations for multiple complex video data over sequential encoding sessions. To overcome the limitation of NIR, we propose a novel method, Progressive Fourier Neural Representation (PFNR), that aims to find an adaptive and compact sub-module in Fourier space to encode videos in each training session. This sparsified neural encoding allows the neural network to hold free weights, enabling an improved adaptation for future videos. In addition, when learning a representation for a new video, PFNR transfers the representation of previous videos with frozen weights. This design allows the model to continuously accumulate high-quality neural representations for multiple videos while ensuring lossless decoding that perfectly preserves the learned representations for previous videos. We validate our PFNR method on the UVG8/17 and DAVIS50 video sequence benchmarks and achieve impressive performance gains over strong continual learning baselines. The PFNR code is available at https://github.com/ihaeyong/PFNR.git.

翻译：神经隐式表示（NIR）因其将复杂高维数据编码至表示空间并通过可训练映射函数轻松重构的卓越能力而近期备受关注。然而，NIR方法假设目标数据与表示模型之间存在一一对应关系，无需考虑数据关联性或相似性。这导致其对多类复杂数据的泛化能力不足，限制其效率与可扩展性。受持续学习启发，本研究探索如何在序列编码会话中累积并迁移多类复杂视频数据的神经隐式表示。为突破NIR的局限，我们提出新颖方法——渐进式傅立叶神经表示（PFNR），旨在每个训练会话中于傅立叶空间寻找自适应紧凑子模块来编码视频。这种稀疏化神经编码允许神经网络保留空闲权重，从而提升对后续视频的适应能力。此外，当学习新视频表示时，PFNR通过冻结权重迁移先前视频的表示。该设计使模型能够持续累积多段视频的高质量神经表示，同时确保无损解码，完美保留先前视频的已习得表示。我们在UVG8/17和DAVIS50视频序列基准上验证PFNR方法，相较于强持续学习基线取得了显著性能提升。PFNR代码已开源至https://github.com/ihaeyong/PFNR.git。