Deep learning models face persistent challenges in training, particularly due to internal covariate shift and label shift. While single-mode normalization methods like Batch Normalization partially address these issues, they are constrained by batch size dependencies and limiting distributional assumptions. Multi-mode normalization techniques mitigate these limitations but struggle with computational demands when handling diverse Gaussian distributions. In this paper, we introduce a new approach to multi-mode normalization that leverages prior knowledge to improve neural network representations. Our method organizes data into predefined structures, or "contexts", prior to training and normalizes based on these contexts, with two variants: Context Normalization (CN) and Context Normalization - Extended (CN-X). When contexts are unavailable, we introduce Adaptive Context Normalization (ACN), which dynamically builds contexts in the latent space during training. Across tasks in image classification, domain adaptation, and image generation, our methods demonstrate superior convergence and performance.

翻译：深度学习模型在训练过程中持续面临挑战，特别是由于内部协变量偏移和标签偏移。尽管像批归一化这样的单模态归一化方法部分解决了这些问题，但它们受限于批次大小依赖性和有限的分布假设。多模态归一化技术缓解了这些限制，但在处理多样化的高斯分布时面临计算需求过大的难题。本文提出一种新的多模态归一化方法，利用先验知识来改进神经网络表示。我们的方法在训练前将数据组织到预定义的结构（即“上下文”）中，并基于这些上下文进行归一化，包含两种变体：上下文归一化（CN）和扩展上下文归一化（CN-X）。当上下文不可用时，我们引入自适应上下文归一化（ACN），该方法在训练期间于隐空间中动态构建上下文。在图像分类、领域适应和图像生成等任务中，我们的方法展现出更优的收敛性和性能。