Implicit models such as Deep Equilibrium Models (DEQs) have garnered significant attention in the community for their ability to train infinite layer models with elegant solution-finding procedures and constant memory footprint. However, despite several attempts, these methods are heavily constrained by model inefficiency and optimization instability. Furthermore, fair benchmarking across relevant methods for vision tasks is missing. In this work, we revisit the line of implicit models and trace them back to the original weight-tied models. Surprisingly, we observe that weight-tied models are more effective, stable, as well as efficient on vision tasks, compared to the DEQ variants. Through the lens of these simple-yet-clean weight-tied models, we further study the fundamental limits in the model capacity of such models and propose the use of distinct sparse masks to improve the model capacity. Finally, for practitioners, we offer design guidelines regarding the depth, width, and sparsity selection for weight-tied models, and demonstrate the generalizability of our insights to other learning paradigms.

翻译：隐式模型（如深度平衡模型DEQ）因其能够通过优雅的求解过程训练无限层模型且内存占用恒定而备受学界关注。然而，尽管已有诸多尝试，这类方法仍受限于模型低效与优化不稳定性。此外，视觉任务相关方法的公平基准测试尚付阙如。本文重新审视隐式模型的发展脉络，将其追溯至原始的权值共享模型。我们惊奇地发现，与DEQ变体相比，权值共享模型在视觉任务中更有效、更稳定且更高效。通过这一简洁而清晰的权值共享模型视角，我们进一步研究了此类模型容量的理论极限，并提出采用差异化稀疏掩码来提升模型容量。最终，我们为实践者提供了关于权值共享模型深度、宽度与稀疏性选择的设计准则，并论证了相关见解对其他学习范式的普适性。