Batch Normalization's (BN) unique property of depending on other samples in a batch is known to cause problems in several tasks, including sequential modeling. Yet, BN-related issues are hardly studied for long video understanding, despite the ubiquitous use of BN in CNNs (Convolutional Neural Networks) for feature extraction. Especially in surgical workflow analysis, where the lack of pretrained feature extractors has led to complex, multi-stage training pipelines, limited awareness of BN issues may have hidden the benefits of training CNNs and temporal models end to end. In this paper, we analyze pitfalls of BN in video learning, including issues specific to online tasks such as a 'cheating' effect in anticipation. We observe that BN's properties create major obstacles for end-to-end learning. However, using BN-free backbones, even simple CNN-LSTMs beat the state of the art {\color{\colorrevtwo}on three surgical workflow benchmarks} by utilizing adequate end-to-end training strategies which maximize temporal context. We conclude that awareness of BN's pitfalls is crucial for effective end-to-end learning in surgical tasks. By reproducing results on natural-video datasets, we hope our insights will benefit other areas of video learning as well. Code is available at: \url{https://gitlab.com/nct_tso_public/pitfalls_bn}

翻译：批归一化（BN）依赖于批次中其他样本的独特性质，已知会在包括序列建模在内的多项任务中引发问题。然而，尽管CNN（卷积神经网络）在特征提取中普遍使用BN，但针对长视频理解中由BN引起的相关问题却鲜有研究。尤其是在手术工作流分析中，由于缺乏预训练特征提取器，导致采用了复杂的多阶段训练流程，而对BN问题的认识不足可能掩盖了端到端训练CNN与时序模型的价值。本文分析了视频学习中BN的陷阱，包括在线任务特有的“作弊”效应（如预期任务中的作弊现象）。我们观察到，BN的特性为端到端学习设置了重大障碍。然而，通过采用无BN的骨干网络，即使是简单的CNN-LSTM模型，也能利用最大化时序上下文的充分端到端训练策略，在三个手术工作流基准上超越当前最先进水平。我们得出结论：对于外科任务中的有效端到端学习，认识BN的陷阱至关重要。通过在自然视频数据集上复现结果，我们希望我们的见解也能惠及其他视频学习领域。代码地址：\url{https://gitlab.com/nct_tso_public/pitfalls_bn}