We study whether and how the choice of optimization algorithm can impact group fairness in deep neural networks. Through stochastic differential equation analysis of optimization dynamics in an analytically tractable setup, we demonstrate that the choice of optimization algorithm indeed influences fairness outcomes, particularly under severe imbalance. Furthermore, we show that when comparing two categories of optimizers, adaptive methods and stochastic methods, RMSProp (from the adaptive category) has a higher likelihood of converging to fairer minima than SGD (from the stochastic category). Building on this insight, we derive two new theoretical guarantees showing that, under appropriate conditions, RMSProp exhibits fairer parameter updates and improved fairness in a single optimization step compared to SGD. We then validate these findings through extensive experiments on three publicly available datasets, namely CelebA, FairFace, and MS-COCO, across different tasks as facial expression recognition, gender classification, and multi-label classification, using various backbones. Considering multiple fairness definitions including equalized odds, equal opportunity, and demographic parity, adaptive optimizers like RMSProp and Adam consistently outperform SGD in terms of group fairness, while maintaining comparable predictive accuracy. Our results highlight the role of adaptive updates as a crucial yet overlooked mechanism for promoting fair outcomes. We release the source code at: https://github.com/Mkolahdoozi/Some-Optimizers-Are-More-Equal.

翻译：我们研究优化算法的选择是否以及如何影响深度神经网络中的群体公平性。通过对一个可解析处理设置中优化动态的随机微分方程分析，我们证明优化算法的选择确实会影响公平性结果，尤其在严重不平衡的情况下。此外，我们表明，在比较自适应方法与随机方法这两类优化器时，RMSProp（属于自适应类别）比SGD（属于随机类别）更有可能收敛到更公平的极小值。基于这一洞见，我们推导出两个新的理论保证，证明在适当条件下，与SGD相比，RMSProp在单次优化步骤中展现出更公平的参数更新和改善的公平性。随后，我们在三个公开数据集（CelebA、FairFace和MS-COCO）上，针对不同任务（如面部表情识别、性别分类和多标签分类），使用多种骨干网络，通过大量实验验证了这些发现。考虑到包括均衡几率、均衡机会和人口统计均等在内的多种公平性定义，自适应优化器（如RMSProp和Adam）在群体公平性方面始终优于SGD，同时保持可比的预测准确性。我们的结果突显了自适应更新作为一种关键但被忽视的机制，对于促进公平结果的重要性。我们在https://github.com/Mkolahdoozi/Some-Optimizers-Are-More-Equal发布了源代码。