Classification model selection is a process of identifying a suitable model class for a given classification task on a dataset. Traditionally, model selection is based on cross-validation, meta-learning, and user preferences, which are often time-consuming and resource-intensive. The performance of any machine learning classification task depends on the choice of the model class, the learning algorithm, and the dataset's characteristics. Our work proposes a novel method for automatic classification model selection from a set of candidate model classes by determining the empirical model-fitness for a dataset based only on its clustering indices. Clustering Indices measure the ability of a clustering algorithm to induce good quality neighborhoods with similar data characteristics. We propose a regression task for a given model class, where the clustering indices of a given dataset form the features and the dependent variable represents the expected classification performance. We compute the dataset clustering indices and directly predict the expected classification performance using the learned regressor for each candidate model class to recommend a suitable model class for dataset classification. We evaluate our model selection method through cross-validation with 60 publicly available binary class datasets and show that our top3 model recommendation is accurate for over 45 of 60 datasets. We also propose an end-to-end Automated ML system for data classification based on our model selection method. We evaluate our end-to-end system against popular commercial and noncommercial Automated ML systems using a different collection of 25 public domain binary class datasets. We show that the proposed system outperforms other methods with an excellent average rank of 1.68.

翻译：分类模型选择是在给定数据集上为分类任务识别合适模型类别的过程。传统模型选择基于交叉验证、元学习和用户偏好，这些方法通常耗时且资源密集。任何机器学习分类任务的性能都取决于模型类别、学习算法和数据集特征的选择。本文提出一种新颖方法，通过仅基于数据集聚类指标确定经验模型适配度，从候选模型类别中自动完成分类模型选择。聚类指标衡量聚类算法诱导具有相似数据特征的高质量邻域的能力。我们针对给定模型类别提出回归任务，其中数据集聚类指标构成特征，因变量表示预期分类性能。我们计算数据集聚类指标，并利用为每个候选模型类别学习到的回归器直接预测预期分类性能，从而推荐适合数据集分类的模型类别。通过包含60个公开二分类数据集的交叉验证评估，本方法的前三模型推荐在超过45个数据集上表现准确。我们还基于该模型选择方法构建了端到端自动化机器学习系统。采用另一组25个公开领域二分类数据集，通过与主流商业及非商业自动化机器学习系统对比评估，本系统以1.68的优异平均排名优于其他方法。