Neural Processes (NPs) are popular meta-learning methods for efficiently modelling predictive uncertainty. Recent state-of-the-art methods, however, leverage expensive attention mechanisms, limiting their applications, particularly in low-resource settings. In this work, we propose Constant Memory Attentive Neural Processes (CMANPs), an NP variant that only requires constant memory. To do so, we first propose an efficient update operation for Cross Attention. Leveraging the update operation, we propose Constant Memory Attention Block (CMAB), a novel attention block that (i) is permutation invariant, (ii) computes its output in constant memory, and (iii) performs constant computation updates. Finally, building on CMAB, we detail Constant Memory Attentive Neural Processes. Empirically, we show CMANPs achieve state-of-the-art results on popular NP benchmarks while being significantly more memory efficient than prior methods.

翻译：神经过程（NPs）是用于高效建模预测不确定性的流行元学习方法。然而，当前最先进的方法采用了计算代价高昂的注意力机制，限制了其应用场景，特别是在低资源环境中。本文提出恒定内存注意力神经过程（CMANPs），这是一种仅需恒定内存的NP变体。为此，我们首先为交叉注意力提出了一种高效的更新操作。基于该更新操作，我们提出了恒定内存注意力块（CMAB），这是一种新颖的注意力模块，其具有以下特性：（i）置换不变性；（ii）以恒定内存计算输出；（iii）执行恒定计算量的更新。最后，基于CMAB，我们详细阐述了恒定内存注意力神经过程。实验表明，CMANPs在主流NP基准测试中取得了最先进的性能，同时其内存效率显著优于现有方法。