Quantized neural network training optimizes a discrete, non-differentiable objective. The straight-through estimator (STE) enables backpropagation through surrogate gradients and is widely used. While previous studies have primarily focused on the properties of surrogate gradients and their convergence, the influence of quantization hyperparameters, such as bit width and quantization range, on learning dynamics remains largely unexplored. We theoretically show that in the high-dimensional limit, STE dynamics converge to a deterministic ordinary differential equation. This reveals that STE training exhibits a plateau followed by a sharp drop in generalization error, with plateau length depending on the quantization range. A fixed-point analysis quantifies the asymptotic deviation from the unquantized linear model. We also extend analytical techniques for stochastic gradient descent to nonlinear transformations of weights and inputs.

翻译：量化神经网络训练优化的是一个离散且不可微的目标函数。直通估计器（STE）通过替代梯度实现反向传播，并得到广泛应用。虽然先前研究主要关注替代梯度的性质及其收敛性，但量化超参数（如比特宽度和量化范围）对学习动力学的影响在很大程度上仍未得到探索。我们从理论上证明，在高维极限下，STE动力学收敛于一个确定性常微分方程。这表明STE训练在泛化误差上呈现先平台期后骤降的现象，且平台期长度取决于量化范围。通过定点分析，我们量化了其与未量化线性模型的渐近偏差。此外，我们将随机梯度下降的分析技术扩展至权重与输入的非线性变换情形。