The most performant spatio-temporal action localisation models use external person proposals and complex external memory banks. We propose a fully end-to-end, purely-transformer based model that directly ingests an input video, and outputs tubelets -- a sequence of bounding boxes and the action classes at each frame. Our flexible model can be trained with either sparse bounding-box supervision on individual frames, or full tubelet annotations. And in both cases, it predicts coherent tubelets as the output. Moreover, our end-to-end model requires no additional pre-processing in the form of proposals, or post-processing in terms of non-maximal suppression. We perform extensive ablation experiments, and significantly advance the state-of-the-art results on four different spatio-temporal action localisation benchmarks with both sparse keyframes and full tubelet annotations.

翻译：最具性能的时空动作定位模型通常依赖外部人物候选框和复杂的外部记忆库。我们提出了一种完全端到端、基于纯Transformer的模型，该模型直接输入视频，并输出管状序列——即每帧的边界框序列及动作类别。这一灵活模型可通过单帧稀疏边界框监督或完整管状标注进行训练。两种情况下，它都能预测连贯的管状序列作为输出。此外，我们的端到端模型无需预处理的候选框生成，也无需后处理的非极大值抑制。通过大量消融实验，我们在四个不同的时空动作定位基准上（同时包含稀疏关键帧和完整管状标注）显著提升了现有最优结果。