Recently, there has been a growing trend toward feature-based approaches for Online Action Detection (OAD). However, these approaches have limitations due to their fixed backbone design, which ignores the potential capability of a trainable backbone. In this paper, we propose the first end-to-end OAD model, termed E2E-LOAD, designed to address the major challenge of OAD, namely, long-term understanding and efficient online reasoning. Specifically, our proposed approach adopts an initial spatial model that is shared by all frames and maintains a long sequence cache for inference at a low computational cost. We also advocate an asymmetric spatial-temporal model for long-form and short-form modeling effectively. Furthermore, we propose a novel and efficient inference mechanism that accelerates heavy spatial-temporal exploration. Extensive ablation studies and experiments demonstrate the effectiveness and efficiency of our proposed method. Notably, we achieve 17.3 (+12.6) FPS for end-to-end OAD with 72.4%~(+1.2%), 90.3%~(+0.7%), and 48.1%~(+26.0%) mAP on THMOUS14, TVSeries, and HDD, respectively, which is 3x faster than previous approaches. The source code will be made publicly available.

翻译：近年来，基于特征的在线动作检测方法日益增多。然而，这些方法受限于固定主干网络设计，忽略了可训练主干网络的潜在能力。本文提出首个端到端在线动作检测模型E2E-LOAD，旨在解决在线动作检测的主要挑战——即长期理解与高效在线推理。具体而言，所提方法采用所有帧共享的初始空间模型，并维护长序列缓存以降低推理计算成本。同时，我们提出非对称时空模型以有效进行长形式和短形式建模。此外，我们设计了一种新颖高效的推理机制，能够加速繁重的时空探索。广泛的消融实验与定量分析验证了所提方法的有效性与高效性。值得注意的是，在THMOUS14、TVSeries和HDD数据集上，我们分别实现了72.4%（+1.2%）、90.3%（+0.7%）和48.1%（+26.0%）的mAP指标，端到端在线动作检测帧率达到17.3 FPS（+12.6），较以往方法提速3倍。源代码将公开发布。