Temporal action detection (TAD) is challenging, yet fundamental for real-world video applications. Recently, DETR-based models for TAD have been prevailing thanks to their unique benefits. However, transformers demand a huge dataset, and unfortunately data scarcity in TAD causes a severe degeneration. In this paper, we identify two crucial problems from data scarcity: attention collapse and imbalanced performance. To this end, we propose a new pre-training strategy, Long-Term Pre-training (LTP), tailored for transformers. LTP has two main components: 1) class-wise synthesis, 2) long-term pretext tasks. Firstly, we synthesize long-form video features by merging video snippets of a target class and non-target classes. They are analogous to untrimmed data used in TAD, despite being created from trimmed data. In addition, we devise two types of long-term pretext tasks to learn long-term dependency. They impose long-term conditions such as finding second-to-fourth or short-duration actions. Our extensive experiments show state-of-the-art performances in DETR-based methods on ActivityNet-v1.3 and THUMOS14 by a large margin. Moreover, we demonstrate that LTP significantly relieves the data scarcity issues in TAD.

翻译：时序动作检测（TAD）具有挑战性，但对现实世界的视频应用至关重要。近年来，基于DETR的TAD模型因其独特优势而日益流行。然而，Transformer需要大量数据，而TAD领域的数据稀缺性不幸地导致了严重的性能退化。本文中，我们识别出数据稀缺引发的两个关键问题：注意力崩溃与性能不均衡。为此，我们提出一种专为Transformer设计的新预训练策略——长期预训练（LTP）。LTP包含两个核心组件：1）类别合成，2）长期预训练任务。首先，我们通过合并目标类别与非目标类别的视频片段来合成长视频特征。这些合成特征虽源自已裁剪数据，但其形态类似于TAD中使用的未裁剪数据。此外，我们设计了两种类型的长期预训练任务以学习长期依赖关系。这些任务施加了长期约束条件，例如要求定位第二至第四个动作或短时动作。我们在ActivityNet-v1.3和THUMOS14数据集上进行的广泛实验表明，该方法在基于DETR的方法中取得了显著领先的先进性能。更重要的是，我们验证了LTP能有效缓解TAD中的数据稀缺问题。