We present a very simple algorithm for attention that requires $O(1)$ memory with respect to sequence length and an extension to self-attention that requires $O(\log n)$ memory. This is in contrast with the frequently stated belief that self-attention requires $O(n^2)$ memory. While the time complexity is still $O(n^2)$, device memory rather than compute capability is often the limiting factor on modern accelerators. Thus, reducing the memory requirements of attention allows processing of longer sequences than might otherwise be feasible. We provide a practical implementation for accelerators that requires $O(\sqrt{n})$ memory, is numerically stable, and is within a few percent of the runtime of the standard implementation of attention. We also demonstrate how to differentiate the function while remaining memory-efficient. For sequence length 16384, the memory overhead of self-attention is reduced by 59X for inference and by 32X for differentiation.

翻译：我们提出了一个非常简单的注意算法,在序列长度方面需要O(1)美元内存,自我注意的延伸需要O(log n)美元内存。这与经常表示的自留需要O(n)2美元内存的信念形成对照。虽然时间复杂性仍然是$(n)2美元,但设备内存而不是计算能力往往是现代加速器的限制因素。因此,减少对注意的内存要求使得对注意序列的处理比其他可能可行时要长。我们为需要O(sqrt{n})内存的加速器提供实际操作,因为需要O(sqrt{n)美元内存,数字稳定,在标准关注执行的运行时间的一小部分之内。我们还演示了如何在保持记忆效率的同时区分功能。对于第16384号序列,自留的内存管理费减少59X,用于推断的减少32X。