Data imbalance is a common problem in machine learning that can have a critical effect on the performance of a model. Various solutions exist but their impact on the convergence of the learning dynamics is not understood. Here, we elucidate the significant negative impact of data imbalance on learning, showing that the learning curves for minority and majority classes follow sub-optimal trajectories when training with a gradient-based optimizer. This slowdown is related to the imbalance ratio and can be traced back to a competition between the optimization of different classes. Our main contribution is the analysis of the convergence of full-batch (GD) and stochastic gradient descent (SGD), and of variants that renormalize the contribution of each per-class gradient. We find that GD is not guaranteed to decrease the loss for each class but that this problem can be addressed by performing a per-class normalization of the gradient. With SGD, class imbalance has an additional effect on the direction of the gradients: the minority class suffers from a higher directional noise, which reduces the effectiveness of the per-class gradient normalization. Our findings not only allow us to understand the potential and limitations of strategies involving the per-class gradients, but also the reason for the effectiveness of previously used solutions for class imbalance such as oversampling.

翻译：数据不平衡是机器学习中的常见问题，对模型性能具有关键影响。现有多种解决方案，但其对学习动力学收敛性的影响尚未明晰。本研究阐明了数据不平衡对学习的显著负面影响，揭示了在基于梯度的优化器训练时，少数类与多数类的学习曲线会遵循次优轨迹。这种减速效应与不平衡比例相关，可追溯至不同类别优化之间的竞争。本研究的核心贡献在于分析了全批次梯度下降（GD）与随机梯度下降（SGD）的收敛性，以及通过重新归一化各类别梯度贡献的变体方法。我们发现GD无法保证每类损失均下降，但可通过逐类梯度归一化解决此问题。对于SGD，类别不平衡对梯度方向具有额外影响：少数类遭受更高的方向性噪声，这削弱了逐类梯度归一化的效果。我们的发现不仅揭示了基于逐类梯度的策略的潜力与局限，更阐明了过采样等现有类别不平衡解决方案的有效性根源。