We introduce EV3, a novel meta-optimization framework designed to efficiently train scalable machine learning models through an intuitive explore-assess-adapt protocol. In each iteration of EV3, we explore various model parameter updates, assess them using pertinent evaluation methods, and adapt the model based on the optimal updates and previous progress history. EV3 offers substantial flexibility without imposing stringent constraints like differentiability on the key objectives relevant to the tasks of interest. Moreover, this protocol welcomes updates with biased gradients and allows for the use of a diversity of losses and optimizers. Additionally, in scenarios with multiple objectives, it can be used to dynamically prioritize tasks. With inspiration drawn from evolutionary algorithms, meta-learning, and neural architecture search, we investigate an application of EV3 to knowledge distillation. Our experimental results illustrate EV3's capability to safely explore model spaces, while hinting at its potential applicability across numerous domains due to its inherent flexibility and adaptability.

翻译：我们提出EV3，一种新颖的元优化框架，通过直观的探索-评估-适应协议高效训练可扩展机器学习模型。在EV3的每次迭代中，我们探索多种模型参数更新方案，利用相关评估方法对其进行评估，并根据最优更新方案及历史进度调整模型。EV3具有高度灵活性，无需对目标任务相关的关键目标施加可微性等严格约束。此外，该协议兼容含偏梯度的更新，并允许使用多样化的损失函数与优化器。在多目标场景下，它还能动态调整任务优先级。受进化算法、元学习及神经架构搜索的启发，我们探索了EV3在知识蒸馏中的应用。实验结果表明，EV3具备安全探索模型空间的能力，同时因其固有灵活性与适应性，暗示其在众多领域的潜在应用价值。