Mutation validation (MV) is a recently proposed approach for model selection, garnering significant interest due to its unique characteristics and potential benefits compared to the widely used cross-validation (CV) method. In this study, we empirically compared MV and $k$-fold CV using benchmark and real-world datasets. By employing Bayesian tests, we compared generalization estimates yielding three posterior probabilities: practical equivalence, CV superiority, and MV superiority. We also evaluated the differences in the capacity of the selected models and computational efficiency. We found that both MV and CV select models with practically equivalent generalization performance across various machine learning algorithms and the majority of benchmark datasets. MV exhibited advantages in terms of selecting simpler models and lower computational costs. However, in some cases MV selected overly simplistic models leading to underfitting and showed instability in hyperparameter selection. These limitations of MV became more evident in the evaluation of a real-world neuroscientific task of predicting sex at birth using brain functional connectivity.

翻译：突变验证（MV）是近期提出的一种模型选择方法，因其相较于广泛使用的交叉验证（CV）具有独特特性和潜在优势而受到广泛关注。本研究利用基准数据集和真实数据集，对MV方法与$k$折CV进行了实证比较。通过采用贝叶斯检验，我们比较了泛化估计结果，得出三种后验概率：实际等价、CV优势与MV优势。同时，我们还评估了所选模型的能力差异及计算效率。研究发现，在多种机器学习算法及大部分基准数据集上，MV与CV所选模型的泛化性能在实践中相当。MV在选择更简洁模型和降低计算成本方面展现出优势。然而，在某些情况下，MV可能选择过于简化的模型导致欠拟合，并在超参数选择中表现出不稳定性。在利用脑功能连接预测出生性别的真实神经科学任务评估中，MV的这些局限性更为凸显。