We propose Token Turing Machines (TTM), a sequential, autoregressive Transformer model with memory for real-world sequential visual understanding. Our model is inspired by the seminal Neural Turing Machine, and has an external memory consisting of a set of tokens which summarise the previous history (i.e., frames). This memory is efficiently addressed, read and written using a Transformer as the processing unit/controller at each step. The model's memory module ensures that a new observation will only be processed with the contents of the memory (and not the entire history), meaning that it can efficiently process long sequences with a bounded computational cost at each step. We show that TTM outperforms other alternatives, such as other Transformer models designed for long sequences and recurrent neural networks, on two real-world sequential visual understanding tasks: online temporal activity detection from videos and vision-based robot action policy learning. Code is publicly available at: https://github.com/google-research/scenic/tree/main/scenic/projects/token_turing

翻译：我们提出Token图灵机（Token Turing Machines, TTM），这是一种面向真实世界序列视觉理解的序列化自回归Transformer模型，具备记忆功能。该模型受经典神经图灵机启发，其外部记忆由一组Token构成，这些Token用于摘要先前历史（即帧）。该记忆通过Transformer作为处理单元/控制器，在每一步实现高效寻址、读取与写入。模型的记忆模块确保新观测仅与记忆内容（而非完整历史）进行交互，从而在每一步以有限计算成本高效处理长序列。在两项真实世界序列视觉理解任务——基于视频的在线时序行为检测与基于视觉的机器人动作策略学习——中，我们证明TTM的性能优于其他替代方案（如专为长序列设计的Transformer模型及循环神经网络）。代码已开源：https://github.com/google-research/scenic/tree/main/scenic/projects/token_turing