In recent years, by utilizing optimization techniques to formulate the propagation of deep model, a variety of so-called Optimization-Derived Learning (ODL) approaches have been proposed to address diverse learning and vision tasks. Although having achieved relatively satisfying practical performance, there still exist fundamental issues in existing ODL methods. In particular, current ODL methods tend to consider model construction and learning as two separate phases, and thus fail to formulate their underlying coupling and depending relationship. In this work, we first establish a new framework, named Hierarchical ODL (HODL), to simultaneously investigate the intrinsic behaviors of optimization-derived model construction and its corresponding learning process. Then we rigorously prove the joint convergence of these two sub-tasks, from the perspectives of both approximation quality and stationary analysis. To our best knowledge, this is the first theoretical guarantee for these two coupled ODL components: optimization and learning. We further demonstrate the flexibility of our framework by applying HODL to challenging learning tasks, which have not been properly addressed by existing ODL methods. Finally, we conduct extensive experiments on both synthetic data and real applications in vision and other learning tasks to verify the theoretical properties and practical performance of HODL in various application scenarios.

翻译：近年来，通过利用优化技术来构建深度模型的传播机制，一系列被称为优化衍生学习（ODL）的方法被提出，以应对多样化的学习与视觉任务。尽管在实践应用中取得了相对令人满意的性能，现有ODL方法仍存在一些根本性问题。特别是，当前ODL方法倾向于将模型构建与学习视为两个分离的阶段，从而未能刻画二者之间的内在耦合与依赖关系。本工作中，我们首先建立了一个名为分层优化衍生学习（HODL）的新框架，以同步探究优化驱动下模型构建及其对应学习过程的本质行为。随后，我们从逼近质量和平稳性分析两个角度严格证明了这两个子任务的联合收敛性。据我们所知，这是针对优化与学习这两个耦合的ODL组件首次提供的理论保证。我们进一步通过将HODL应用于现有ODL方法未能妥善处理的挑战性学习任务，展示了该框架的灵活性。最后，我们在合成数据及视觉与其他学习任务的实际应用上开展了广泛实验，以验证HODL在不同应用场景下的理论性质与实用性能。