Many online action prediction models observe complete frames to locate and attend to informative subregions in the frames called glimpses and recognize an ongoing action based on global and local information. However, in applications with constrained resources, an agent may not be able to observe the complete frame, yet must still locate useful glimpses to predict an incomplete action based on local information only. In this paper, we develop Glimpse Transformers (GliTr), which observe only narrow glimpses at all times, thus predicting an ongoing action and the following most informative glimpse location based on the partial spatiotemporal information collected so far. In the absence of a ground truth for the optimal glimpse locations for action recognition, we train GliTr using a novel spatiotemporal consistency objective: We require GliTr to attend to the glimpses with features similar to the corresponding complete frames (i.e. spatial consistency) and the resultant class logits at time $t$ equivalent to the ones predicted using whole frames up to $t$ (i.e. temporal consistency). Inclusion of our proposed consistency objective yields ~10% higher accuracy on the Something-Something-v2 (SSv2) dataset than the baseline cross-entropy objective. Overall, despite observing only ~33% of the total area per frame, GliTr achieves 53.02% and 93.91% accuracy on the SSv2 and Jester datasets, respectively.

翻译：许多在线动作预测模型通过观察完整帧来定位并关注帧中的信息子区域（称为glimpse），并基于全局与局部信息识别进行中的动作。然而，在资源受限的应用场景中，智能体可能无法观察完整帧，但仍需定位有效的glimpse以仅基于局部信息预测不完整动作。本文提出Glimpse Transformers（GliTr），该模型始终仅观察狭窄的glimpse，从而基于迄今收集的部分时空信息预测进行中的动作及下一步信息量最大的glimpse位置。由于缺乏动作识别中最优glimpse位置的真实标注，我们提出一种新颖的时空一致性目标训练GliTr：要求GliTr关注特征与对应完整帧相似的glimpse（即空间一致性），且其t时刻产生的类别logits与使用直至t时刻的完整帧预测得到的logits等价（即时间一致性）。引入所提一致性目标后，在Something-Something-v2（SSv2）数据集上的准确率比基线交叉熵目标提升约10%。总体而言，尽管每帧仅观察约33%的区域，GliTr在SSv2与Jester数据集上分别达到53.02%和93.91%的准确率。