Real-world systems are often formulated as constrained optimization problems. Techniques to incorporate constraints into Neural Networks (NN), such as Neural Ordinary Differential Equations (Neural ODEs), have been used. However, these introduce hyperparameters that require manual tuning through trial and error, raising doubts about the successful incorporation of constraints into the generated model. This paper describes in detail the two-stage training method for Neural ODEs, a simple, effective, and penalty parameter-free approach to model constrained systems. In this approach the constrained optimization problem is rewritten as two unconstrained sub-problems that are solved in two stages. The first stage aims at finding feasible NN parameters by minimizing a measure of constraints violation. The second stage aims to find the optimal NN parameters by minimizing the loss function while keeping inside the feasible region. We experimentally demonstrate that our method produces models that satisfy the constraints and also improves their predictive performance. Thus, ensuring compliance with critical system properties and also contributing to reducing data quantity requirements. Furthermore, we show that the proposed method improves the convergence to an optimal solution and improves the explainability of Neural ODE models. Our proposed two-stage training method can be used with any NN architectures.

翻译：现实世界中的系统通常被表述为约束优化问题。将约束融入神经网络的技术（如神经常微分方程）已被采用。然而，这些方法引入的超参数需要通过反复试错进行手动调整，这引发了人们对约束能否成功融入生成模型的疑虑。本文详细描述了神经常微分方程的两阶段训练方法——一种简单、有效且无需惩罚参数的方法，用于对约束系统建模。该方法将约束优化问题重写为两个无约束子问题，分两个阶段求解：第一阶段通过最小化约束违反度量来寻找可行的神经网络参数；第二阶段在保持可行域内的前提下通过最小化损失函数来寻找最优神经网络参数。我们通过实验证明，该方法生成的模型既能满足约束条件，又能提升预测性能，从而确保关键系统属性得到满足，同时有助于降低数据量需求。此外，我们证明该方法能改善最优解的收敛性，并提升神经常微分方程模型的可解释性。我们提出的两阶段训练方法可适用于任何神经网络架构。