Deep Equilibrium Models (DEQs) and Neural Ordinary Differential Equations (Neural ODEs) are two branches of implicit models that have achieved remarkable success owing to their superior performance and low memory consumption. While both are implicit models, DEQs and Neural ODEs are derived from different mathematical formulations. Inspired by homotopy continuation, we establish a connection between these two models and illustrate that they are actually two sides of the same coin. Homotopy continuation is a classical method of solving nonlinear equations based on a corresponding ODE. Given this connection, we proposed a new implicit model called HomoODE that inherits the property of high accuracy from DEQs and the property of stability from Neural ODEs. Unlike DEQs, which explicitly solve an equilibrium-point-finding problem via Newton's methods in the forward pass, HomoODE solves the equilibrium-point-finding problem implicitly using a modified Neural ODE via homotopy continuation. Further, we developed an acceleration method for HomoODE with a shared learnable initial point. It is worth noting that our model also provides a better understanding of why Augmented Neural ODEs work as long as the augmented part is regarded as the equilibrium point to find. Comprehensive experiments with several image classification tasks demonstrate that HomoODE surpasses existing implicit models in terms of both accuracy and memory consumption.

翻译：深度均衡模型（DEQs）与神经常微分方程（Neural ODEs）是隐式模型的两个分支，因其卓越性能与低内存消耗而取得了显著成功。尽管同为隐式模型，DEQs和Neural ODEs源于不同的数学形式。受同伦延拓启发，我们建立了这两种模型之间的关联，并阐明它们实为同一硬币的两面。同伦延拓是一种基于相应常微分方程（ODE）求解非线性方程的经典方法。基于此关联，我们提出一种名为HomoODE的新型隐式模型，它继承了DEQs的高精度特性与Neural ODEs的稳定性特性。与DEQs在前向传播中通过牛顿方法显式求解平衡点发现问题不同，HomoODE通过同伦延拓利用改进的Neural ODE隐式求解该问题。进一步地，我们为HomoODE开发了一种基于共享可学习初始点的加速方法。值得注意的是，我们的模型还为增广神经常微分方程（Augmented Neural ODEs）的工作原理提供了更深刻的理解——只要将增广部分视为待求的平衡点。多项图像分类任务的综合实验表明，HomoODE在准确率和内存消耗方面均超越了现有隐式模型。