A central problem in machine learning theory is to characterize how learning dynamics select particular solutions among the many compatible with the training objective, a phenomenon, called implicit bias, which remains only partially characterized. In the present work, we identify a general mechanism, in terms of an explicit geometric correction of the learning dynamics, for the emergence of implicit biases, arising from the interaction between continuous symmetries in the model's parametrization and stochasticity in the optimization process. Our viewpoint is constructive in two complementary directions: given model symmetries, one can derive the implicit bias they induce; conversely, one can inverse-design a wide class of different implicit biases by computing specific redundant parameterizations. More precisely, we show that, when the dynamics is expressed in the quotient space obtained by factoring out the symmetry group of the parameterization, the resulting stochastic differential equation gains a closed form geometric correction in the stationary distribution of the optimizer dynamics favoring orbits with small local volume. We compute the resulting symmetry induced bias for a range of architectures, showing how several well known results fit into a single unified framework. The approach also provides a practical methodology for deriving implicit biases in new settings, and it yields concrete, testable predictions that we confirm by numerical simulations on toy models trained on synthetic data, leaving more complex scenarios for future work. Finally, we test the implicit bias inverse-design procedure in notable cases, including biases toward sparsity in linear features or in spectral properties of the model parameters.

翻译：机器学习理论中的一个核心问题是刻画学习动态如何从众多与训练目标兼容的解中选择特定解，这种现象被称为隐式偏差，目前仅得到部分表征。在本工作中，我们提出了一种基于学习动态显式几何修正的通用机制，用于解释隐式偏差的出现，该机制源于模型参数化中的连续对称性与优化过程中随机性之间的相互作用。我们的观点在两个互补方向上具有建设性：给定模型对称性，可以推导其诱导的隐式偏差；反之，可以通过计算特定的冗余参数化来逆向设计多种不同的隐式偏差。更精确地说，我们证明当动态在通过分解参数化对称群得到的商空间中表达时，所得随机微分方程会在优化器动态的平稳分布中获得闭合形式的几何修正，该修正倾向于局部体积较小的轨道。我们计算了一系列架构中由此产生的对称性诱导偏差，展示了多个已知结果如何融入统一的理论框架。该方法还为推导新场景中的隐式偏差提供了实用方法论，并产生了可通过数值模拟验证的具体预测——我们在合成数据训练的玩具模型上确认了这些预测，更复杂的场景留待未来研究。最后，我们在若干典型案例中测试了隐式偏差逆向设计流程，包括对线性特征稀疏性或模型参数谱特性的偏好偏差。