This work formulates the machine learning mechanism as a bi-level optimization problem. The inner level optimization loop entails minimizing a properly chosen loss function evaluated on the training data. This is nothing but the well-studied training process in pursuit of optimal model parameters. The outer level optimization loop is less well-studied and involves maximizing a properly chosen performance metric evaluated on the validation data. This is what we call the "iteration process", pursuing optimal model hyper-parameters. Among many other degrees of freedom, this process entails model engineering (e.g., neural network architecture design) and management, experiment tracking, dataset versioning and augmentation. The iteration process could be automated via Automatic Machine Learning (AutoML) or left to the intuitions of machine learning students, engineers, and researchers. Regardless of the route we take, there is a need to reduce the computational cost of the iteration step and as a direct consequence reduce the carbon footprint of developing artificial intelligence algorithms. Despite the clean and unified mathematical formulation of the iteration step as a bi-level optimization problem, its solutions are case specific and complex. This work will consider such cases while increasing the level of complexity from supervised learning to semi-supervised, self-supervised, unsupervised, few-shot, federated, reinforcement, and physics-informed learning. As a consequence of this exercise, this proposal surfaces a plethora of open problems in the field, many of which can be addressed in parallel.

翻译：这项工作将机器学习机制设计成双级优化问题。 内部一级优化循环意味着最大限度地减少对培训数据进行评估的正确选择的损失功能。 这只不过是为追求最佳模型参数而研究周密的培训过程而已。 外部一级优化循环没有很好地研究, 涉及对验证数据进行最佳选择的性能衡量尺度。 这就是我们称之为“ 填补过程”, 追求最佳模型超参数。 在许多其他自由度中, 这一过程包括模型工程( 例如神经网络结构设计)和管理、 实验跟踪、 数据设置和增强。 这只不过是通过自动机器学习( 自动学习) 或留待机器学习学生、 工程师和研究人员的直觉来自动操作。 无论我们走哪条路线, 都需要降低循环步骤的计算成本, 并直接降低开发人工智能算法的碳足迹。 尽管在双级优化问题中, 循环步骤的简单和统一的数学配置, 其解决方案是具体和复杂的。 这项工作将考虑通过自动机器学习( 自动学习) 精细的实地学习过程, 将这种精细的自我学习过程从不精细的复杂程度,, 将考虑一个不精确的实地学习过程 。