This work formulates the machine learning mechanism as a bi-level optimization problem. The inner level optimization loop entails minimizing a properly chosen loss function evaluated on the training data. This is nothing but the well-studied training process in pursuit of optimal model parameters. The outer level optimization loop is less well-studied and involves maximizing a properly chosen performance metric evaluated on the validation data. This is what we call the "iteration process", pursuing optimal model hyper-parameters. Among many other degrees of freedom, this process entails model engineering (e.g., neural network architecture design) and management, experiment tracking, dataset versioning and augmentation. The iteration process could be automated via Automatic Machine Learning (AutoML) or left to the intuitions of machine learning students, engineers, and researchers. Regardless of the route we take, there is a need to reduce the computational cost of the iteration step and as a direct consequence reduce the carbon footprint of developing artificial intelligence algorithms. Despite the clean and unified mathematical formulation of the iteration step as a bi-level optimization problem, its solutions are case specific and complex. This work will consider such cases while increasing the level of complexity from supervised learning to semi-supervised, self-supervised, unsupervised, few-shot, federated, reinforcement, and physics-informed learning. As a consequence of this exercise, this proposal surfaces a plethora of open problems in the field, many of which can be addressed in parallel.

翻译：本文将机器学习机制形式化为一个双层优化问题。内层优化循环涉及最小化在训练数据上评估的适当损失函数，这不过是追求最优模型参数中已深入研究的训练过程。外层优化循环的研究较少，涉及最大化在验证数据上评估的适当性能指标，我们称之为"迭代过程"，旨在寻求最优模型超参数。在众多自由度中，此过程涵盖模型工程（如神经网络架构设计）与管理、实验跟踪、数据集版本控制及增强。迭代过程可通过自动机器学习（AutoML）实现自动化，或交由机器学习学生、工程师和研究者的直觉处理。无论采取何种路径，都需要降低迭代步骤的计算成本，从而直接减少开发人工智能算法的碳足迹。尽管迭代步骤的数学形式化表达为简洁统一的双层优化问题，但其解决方案具有案例特异性且复杂。本文将考虑此类案例，同时将复杂度从监督学习逐步提升至半监督、自监督、无监督、少样本、联邦、强化及物理信息学习。通过这一实践，本研究揭示了该领域涌现的大量开放问题，其中许多问题可并行解决。