We propose MAMo, a novel memory and attention frame-work for monocular video depth estimation. MAMo can augment and improve any single-image depth estimation networks into video depth estimation models, enabling them to take advantage of the temporal information to predict more accurate depth. In MAMo, we augment model with memory which aids the depth prediction as the model streams through the video. Specifically, the memory stores learned visual and displacement tokens of the previous time instances. This allows the depth network to cross-reference relevant features from the past when predicting depth on the current frame. We introduce a novel scheme to continuously update the memory, optimizing it to keep tokens that correspond with both the past and the present visual information. We adopt attention-based approach to process memory features where we first learn the spatio-temporal relation among the resultant visual and displacement memory tokens using self-attention module. Further, the output features of self-attention are aggregated with the current visual features through cross-attention. The cross-attended features are finally given to a decoder to predict depth on the current frame. Through extensive experiments on several benchmarks, including KITTI, NYU-Depth V2, and DDAD, we show that MAMo consistently improves monocular depth estimation networks and sets new state-of-the-art (SOTA) accuracy. Notably, our MAMo video depth estimation provides higher accuracy with lower latency, when omparing to SOTA cost-volume-based video depth models.

翻译：我们提出MAMo，一种新颖的记忆与注意力框架，用于单目视频深度估计。MAMo能够增强和改善任何单图像深度估计网络，将其转化为视频深度估计模型，使其能够利用时序信息预测更精确的深度。在MAMo中，我们通过记忆增强模型，当模型处理视频流时，这些记忆有助于深度预测。具体来说，记忆存储了先前时间实例中学习到的视觉和位移标记。这使得深度网络在预测当前帧深度时，能够交叉引用过去的关联特征。我们引入了一种新颖的连续更新记忆方案，优化它以保留与过去和当前视觉信息相对应的标记。我们采用基于注意力的方法处理记忆特征，首先通过自注意力模块学习产生的视觉和位移记忆标记之间的时空关联。随后，自注意力的输出特征通过交叉注意力与当前视觉特征聚合。最后，交叉注意力特征输入解码器以预测当前帧的深度。通过多个基准测试（包括KITTI、NYU-Depth V2和DDAD）的大量实验，我们证明MAMO能够持续改进单目深度估计网络，并达到新的最先进精度。值得注意的是，与基于成本体积的最先进视频深度模型相比，我们的MAMO视频深度估计在提供更高精度的同时保持了更低的延迟。