Mutation validation (MV) is a recently proposed approach for model selection, garnering significant interest due to its unique characteristics and potential benefits compared to the widely used cross-validation (CV) method. In this study, we empirically compared MV and $k$-fold CV using benchmark and real-world datasets. By employing Bayesian tests, we compared generalization estimates yielding three posterior probabilities: practical equivalence, CV superiority, and MV superiority. We also evaluated the differences in the capacity of the selected models and computational efficiency. We found that both MV and CV select models with practically equivalent generalization performance across various machine learning algorithms and the majority of benchmark datasets. MV exhibited advantages in terms of selecting simpler models and lower computational costs. However, in some cases MV selected overly simplistic models leading to underfitting and showed instability in hyperparameter selection. These limitations of MV became more evident in the evaluation of a real-world neuroscientific task of predicting sex at birth using brain functional connectivity.

翻译：突变验证（MV）是近期提出的一种模型选择方法，因其独特特性及相较于广泛使用的交叉验证（CV）方法的潜在优势而备受关注。本研究使用基准数据集和真实世界数据集，对MV与$k$折CV进行了实证比较。通过采用贝叶斯检验，我们比较了泛化估计结果，得出三种后验概率：实际等价性、CV优越性及MV优越性。同时评估了所选模型的容量差异与计算效率。结果表明，在各种机器学习算法及大多数基准数据集上，MV与CV所选模型的泛化性能实际相当。MV在选择更简单模型和降低计算成本方面展现出优势，但在某些情况下会因选择过于简化的模型而导致欠拟合，并在超参数选择上表现出不稳定性。在利用大脑功能连接预测出生性别的真实神经科学任务评估中，MV的这些局限性愈发显著。