Does the use of auto-differentiation yield reasonable updates to deep neural networks that represent neural ODEs? Through mathematical analysis and numerical evidence, we find that when the neural network employs high-order forms to approximate the underlying ODE flows (such as the Linear Multistep Method (LMM)), brute-force computation using auto-differentiation often produces non-converging artificial oscillations. In the case of Leapfrog, we propose a straightforward post-processing technique that effectively eliminates these oscillations, rectifies the gradient computation and thus respects the updates of the underlying flow.

翻译：通过数学分析与数值实验验证，本研究探讨了在表示神经常微分方程（neural ODE）的深度神经网络训练中，直接使用自动微分（auto-differentiation）能否产生合理的参数更新。研究结果表明：当神经网络采用高阶格式（如线性多步法（LMM））逼近底层ODE流时，基于自动微分的暴力计算常导致非收敛性的人工振荡。针对跳点法（Leapfrog）案例，我们提出了一种简洁的后处理技术，该技术能有效消除此类振荡，修正梯度计算方式，从而确保参数更新与底层ODE流的动力学特性保持一致。