Procedure step recognition (PSR) aims to identify all correctly completed steps and their sequential order in videos of procedural tasks. The existing state-of-the-art models rely solely on detecting assembly object states in individual video frames. By neglecting temporal features, model robustness and accuracy are limited, especially when objects are partially occluded. To overcome these limitations, we propose Spatio-Temporal Occlusion-Resilient Modeling for Procedure Step Recognition (STORM-PSR), a dual-stream framework for PSR that leverages both spatial and temporal features. The assembly state detection stream operates effectively with unobstructed views of the object, while the spatio-temporal stream captures both spatial and temporal features to recognize step completions even under partial occlusion. This stream includes a spatial encoder, pre-trained using a novel weakly supervised approach to capture meaningful spatial representations, and a transformer-based temporal encoder that learns how these spatial features relate over time. STORM-PSR is evaluated on the MECCANO and IndustReal datasets, reducing the average delay between actual and predicted assembly step completions by 11.2% and 26.1%, respectively, compared to prior methods. We demonstrate that this reduction in delay is driven by the spatio-temporal stream, which does not rely on unobstructed views of the object to infer completed steps. The code for STORM-PSR, along with the newly annotated MECCANO labels, is made publicly available at https://timschoonbeek.github.io/stormpsr .

翻译：工序步骤识别（PSR）旨在识别程序性任务视频中所有正确完成的步骤及其顺序。现有最先进模型仅依赖于检测单个视频帧中的装配对象状态。由于忽略时序特征，模型的鲁棒性和准确性受到限制，尤其在物体部分遮挡时更为明显。为克服这些局限性，我们提出用于工序步骤识别的时空遮挡鲁棒建模（STORM-PSR），这是一种利用空间和时序特征的双流PSR框架。装配状态检测流在物体视野无遮挡时高效运作，而时空流则同时捕获空间和时序特征，即使在部分遮挡条件下也能识别步骤完成状态。该流包含一个空间编码器（采用新型弱监督预训练方法以捕获有意义的空间表征）和一个基于Transformer的时序编码器（学习这些空间特征随时间变化的关联规律）。STORM-PSR在MECCANO和IndustReal数据集上进行评估，相较于现有方法，其装配步骤完成时间与实际时间的平均延迟分别降低11.2%和26.1%。我们证明这种延迟减少主要得益于时空流，该流无需依赖物体的无遮挡视野即可推断完成步骤。STORM-PSR的代码及新标注的MECCANO标签已公开于https://timschoonbeek.github.io/stormpsr。