In this paper, we formulate the hyperparameter tuning problem in machine learning as a bilevel program. The bilevel program is solved using a micro genetic algorithm that is enhanced with a linear program. While the genetic algorithm searches over discrete hyperparameters, the linear program enhancement allows hyper local search over continuous hyperparameters. The major contribution in this paper is the formulation of a linear program that supports fast search over continuous hyperparameters, and can be integrated with any hyperparameter search technique. It can also be applied directly on any trained machine learning or deep learning model for the purpose of fine-tuning. We test the performance of the proposed approach on two datasets, MNIST and CIFAR-10. Our results clearly demonstrate that using the linear program enhancement offers significant promise when incorporated with any population-based approach for hyperparameter tuning.

翻译：本文提出将机器学习中的超参数调优问题建模为双层规划问题。该双层规划通过一种经线性规划增强的微遗传算法进行求解。遗传算法负责在离散超参数空间中进行搜索，而线性规划增强则支持在连续超参数空间进行超局部搜索。本文的主要贡献在于构建了一种能够支持连续超参数快速搜索的线性规划模型，该模型可与任意超参数搜索技术相结合，亦可直接应用于任何已训练的机器学习或深度学习模型以实现精细调优。我们在MNIST和CIFAR-10两个数据集上测试了所提方法的性能。实验结果表明，将线性规划增强模块融入基于种群的超参数调优方法能带来显著的性能提升。