In this paper, we consider Discretized Neural Networks (DNNs) consisting of low-precision weights and activations, which suffer from either infinite or zero gradients caused by the non-differentiable discrete function in the training process. In this case, most training-based DNNs use the standard Straight-Through Estimator (STE) to approximate the gradient w.r.t. discrete value. However, the standard STE will cause the gradient mismatch problem, i.e., the approximated gradient direction may deviate from the steepest descent direction. In other words, the gradient mismatch implies the approximated gradient with perturbations. To address this problem, we introduce the duality theory to regard the perturbation of the approximated gradient as the perturbation of the metric in Linearly Nearly Euclidean (LNE) manifolds. Simultaneously, under the Ricci-DeTurck flow, we prove the dynamical stability and convergence of the LNE metric with the $L^2$-norm perturbation, which can provide a theoretical solution for the gradient mismatch problem. In practice, we also present the steepest descent gradient flow for DNNs on LNE manifolds from the viewpoints of the information geometry and mirror descent. The experimental results on various datasets demonstrate that our method achieves better and more stable performance for DNNs than other representative training-based methods.

翻译：本文研究由低精度权重和激活值组成的离散化神经网络（DNN），该类网络在训练过程中因非可微离散函数导致梯度为零或无穷大。在此情形下，多数基于训练的离散神经网络采用标准直通估计器（STE）来近似计算关于离散值的梯度。然而，标准STE会引发梯度失配问题，即近似梯度方向可能偏离最速下降方向。换言之，梯度失配意味着近似梯度存在扰动。为解决该问题，我们引入对偶理论，将近似梯度的扰动视为线性近欧几里得（LNE）流形上度量的扰动。同时，在Ricci-DeTurck流框架下，我们证明了具有$L^2$范数扰动的LNE度量的动力学稳定性与收敛性，这为梯度失配问题提供了理论解决方案。在实践层面，我们还从信息几何和镜像下降视角给出了LNE流形上DNN的最速下降梯度流。多数据集上的实验结果表明，相较于其他代表性基于训练的方法，我们的方法能使DNN获得更优且更稳定的性能。