Deep Equilibrium Models (DEQs) and Neural Ordinary Differential Equations (Neural ODEs) are two branches of implicit models that have achieved remarkable success owing to their superior performance and low memory consumption. While both are implicit models, DEQs and Neural ODEs are derived from different mathematical formulations. Inspired by homotopy continuation, we establish a connection between these two models and illustrate that they are actually two sides of the same coin. Homotopy continuation is a classical method of solving nonlinear equations based on a corresponding ODE. Given this connection, we proposed a new implicit model called HomoODE that inherits the property of high accuracy from DEQs and the property of stability from Neural ODEs. Unlike DEQs, which explicitly solve an equilibrium-point-finding problem via Newton's methods in the forward pass, HomoODE solves the equilibrium-point-finding problem implicitly using a modified Neural ODE via homotopy continuation. Further, we developed an acceleration method for HomoODE with a shared learnable initial point. It is worth noting that our model also provides a better understanding of why Augmented Neural ODEs work as long as the augmented part is regarded as the equilibrium point to find. Comprehensive experiments with several image classification tasks demonstrate that HomoODE surpasses existing implicit models in terms of both accuracy and memory consumption.

翻译：深度平衡模型（DEQs）和神经常微分方程（Neural ODEs）是隐式模型的两个分支，因其卓越性能和低内存消耗而取得显著成功。尽管同属隐式模型，DEQs与Neural ODEs却源自不同的数学形式。受同伦延拓启发，我们在两者之间建立了联系，并阐明它们实为同一枚硬币的两面。同伦延拓是一种基于对应常微分方程求解非线性方程的经典方法。基于这一联系，我们提出了一种名为HomoODE的新型隐式模型，它继承了DEQs的高精度特性和Neural ODEs的稳定性特性。与DEQs在前向传播中通过牛顿方法显式求解平衡点寻找问题不同，HomoODE通过同伦延拓利用修正的Neural ODE隐式求解该问题。此外，我们开发了一种基于共享可学习初始点的HomoODE加速方法。值得注意的是，该模型还提供了对增广神经常微分方程（Augmented Neural ODEs）有效性的更深理解：只要将增广部分视为待求解的平衡点。多项图像分类任务的全面实验表明，HomoODE在准确率和内存消耗方面均超越了现有隐式模型。