Deep learning faces significant challenges during the training of neural networks, including internal covariate shift, label shift, vanishing/exploding gradients, overfitting, and computational complexity. While conventional normalization methods, such as Batch Normalization, aim to tackle some of these issues, they often depend on assumptions that constrain their adaptability. Mixture Normalization faces computational hurdles in its pursuit of handling multiple Gaussian distributions. This paper introduces Cluster-Based Normalization (CB-Norm) in two variants - Supervised Cluster-Based Normalization (SCB-Norm) and Unsupervised Cluster-Based Normalization (UCB-Norm) - proposing a groundbreaking one-step normalization approach. CB-Norm leverages a Gaussian mixture model to specifically address challenges related to gradient stability and learning acceleration. For SCB-Norm, a supervised variant, the novel mechanism involves introducing predefined data partitioning, termed clusters, to normalize activations based on the assigned cluster. This cluster-driven approach creates a space that conforms to a Gaussian mixture model. On the other hand, UCB-Norm, an unsupervised counterpart, dynamically clusters neuron activations during training, adapting to task-specific challenges without relying on predefined data partitions (clusters). This dual approach ensures flexibility in addressing diverse learning scenarios. CB-Norm innovatively uses a one-step normalization approach, where parameters of each mixture component (cluster in activation space) serve as weights for deep neural networks. This adaptive clustering process tackles both clustering and resolution of deep neural network tasks concurrently during training, signifying a notable advancement in the field.

翻译：深度学习在神经网络训练过程中面临诸多挑战，包括内部协变量偏移、标签偏移、梯度消失/爆炸、过拟合以及计算复杂性。传统的归一化方法（如批归一化）旨在解决部分问题，但往往依赖于限制其适应性的假设。混合归一化在处理多个高斯分布时面临计算障碍。本文提出两种变体的基于聚类的归一化（CB-Norm）——有监督聚类归一化（SCB-Norm）与无监督聚类归一化（UCB-Norm）——开创性地提出一种单步归一化方法。CB-Norm利用高斯混合模型，专门应对梯度稳定性与学习加速方面的挑战。对于有监督变体SCB-Norm，其新颖机制在于引入预定义数据分区（称为簇），根据所分配的簇对激活值进行归一化。这种基于簇的方法构建了一个符合高斯混合模型的空间。而无监督变体UCB-Norm则在训练过程中动态地对神经元激活值进行聚类，无需依赖预定义数据分区（簇），自适应地解决任务特定挑战。这种双轨方法确保了应对多样化学习场景的灵活性。CB-Norm创新性地采用单步归一化，其中每个混合分量（激活空间中的簇）的参数作为深度神经网络的权重。这一自适应聚类过程在训练过程中同时处理深度神经网络任务的聚类与求解，标志着该领域的显著进步。