Deep learning grapples with challenges in training neural networks, notably internal covariate shift and label shift. Conventional normalization techniques like Batch Normalization (BN) partially mitigate these issues but are hindered by constraints such as dependency on batch size and distribution assumptions. Similarly, mixture normalization (MN) encounters computational barriers in handling diverse Gaussian distributions. This paper introduces Cluster-based Normalization (CB-Norm), presenting two variants: Supervised Cluster-based Normalization (SCB-Norm) and Unsupervised Cluster-based Normalization (UCB-Norm), offering a pioneering single-step normalization strategy. CB-Norm employs a Gaussian mixture model to address gradient stability and learning acceleration challenges. SCB-Norm utilizes predefined data partitioning, termed clusters, for supervised normalization, while UCB-Norm adaptively clusters neuron activations during training, eliminating reliance on predefined partitions. This approach simultaneously tackles clustering and resolution tasks within neural networks, reducing computational complexity compared to existing methods. CB-Norm outperforms traditional techniques like BN and MN, enhancing neural network performance across diverse learning scenarios.

翻译：深度学习在训练神经网络时面临诸多挑战，尤其是内部协变量偏移和标签偏移。传统的归一化技术（如批量归一化）虽能部分缓解这些问题，但受限于批量大小依赖性和分布假设等约束。同样，混合归一化在处理不同高斯分布时遭遇计算障碍。本文提出基于聚类的归一化（CB-Norm），并给出两种变体：监督式聚类归一化（SCB-Norm）和无监督式聚类归一化（UCB-Norm），开创性地实现了单步归一化策略。CB-Norm采用高斯混合模型来解决梯度稳定性与学习加速的挑战。SCB-Norm利用预定义的数据划分（称为聚类）进行监督式归一化，而UCB-Norm在训练过程中自适应地对神经元激活值进行聚类，消除了对预定义划分的依赖。该方法在神经网络内同时处理聚类与分辨率任务，相比现有方法降低了计算复杂度。CB-Norm在多种学习场景中均优于批量归一化和混合归一化等传统技术，显著提升了神经网络性能。