Batch Normalization's (BN) unique property of depending on other samples in a batch is known to cause problems in several tasks, including sequence modeling. Yet, BN-related issues are hardly studied for long video understanding, despite the ubiquitous use of BN in CNNs (Convolutional Neural Networks) for feature extraction. Especially in surgical workflow analysis, where the lack of pretrained feature extractors has led to complex, multi-stage training pipelines, limited awareness of BN issues may have hidden the benefits of training CNNs and temporal models end to end. In this paper, we analyze pitfalls of BN in video learning, including issues specific to online tasks such as a 'cheating' effect in anticipation. We observe that BN's properties create major obstacles for end-to-end learning. However, using BN-free backbones, even simple CNN-LSTMs beat the state of the art {\color{\colorrevtwo}on three surgical workflow benchmarks} by utilizing adequate end-to-end training strategies which maximize temporal context. We conclude that awareness of BN's pitfalls is crucial for effective end-to-end learning in surgical tasks. By reproducing results on natural-video datasets, we hope our insights will benefit other areas of video learning as well. Code is available at: \url{https://gitlab.com/nct_tso_public/pitfalls_bn}

翻译：批归一化（Batch Normalization, BN）的独特性质——依赖于批次内其他样本——已知会在多个任务中引发问题，包括序列建模。然而，尽管卷积神经网络（CNN）在特征提取中广泛使用BN，但针对长视频理解中BN相关问题的研究仍十分匮乏。特别是在手术工作流分析中，由于缺乏预训练特征提取器，导致需采用复杂的多阶段训练流程，而对BN问题的有限认知可能掩盖了端到端训练CNN与时序模型的优势。本文分析了视频学习中BN的陷阱，包括在线任务特有的“作弊”效应（如预测中的提前泄露）。我们观察到，BN的性质为端到端学习设置了重大障碍。然而，使用无BN骨干网络时，即便简单的CNN-LSTM结构，通过充分利用最大化时序上下文的端到端训练策略，也能在三个手术工作流基准上超越当前最优方法。我们得出结论：认知BN的陷阱对手术任务中的高效端到端学习至关重要。通过在自然视频数据集上复现实验结果，我们希望本文的见解能惠及其他视频学习领域。代码见：\url{https://gitlab.com/nct_tso_public/pitfalls_bn}