Cross-validation (CV) is routinely used across the sciences to select models and tune parameters, and the resulting choices are often interpreted as substantive scientific conclusions (e.g., which variables, mechanisms, or risk factors are ``supported by the data''). A key part of the CV procedure -- the hold-out size, or equivalently the fold count $K$ -- is typically set by convention (e.g., 80/20, $K=5$) rather than by a principled criterion. Central to the issue is the tradeoff between training and testing: increasing the training sample size improves model accuracy, while sacrificing certainty around the accuracy itself. We formalize the tradeoff by targeting predictive performance and explicitly penalizing evaluation uncertainty, which cannot be identified from the data without additional assumptions. We derive finite-sample expressions of this evaluation uncertainty under symmetric errors and general upper bounds under broader error conditions, yielding a transparent utility-based rule for selecting the hold-out size as a function of an irreducible-noise parameter. Empirical analyses with linear regression and random forests across multiple domains, and a high-dimensional genomics application, show that (i) the choice of $K$ is dependent on the data and model. (ii) the optimal $K$ varies based on the assumption on the irreducible error, and (iii) the implied inferential conclusions can change materially as the irreducible error, and thus $K$, varies. The resulting framework replaces a one-size-fits-all convention with a context-specific, assumption-explicit choice of $K$, enabling more reliable model comparisons and downstream scientific inference.

翻译：交叉验证（CV）在科学领域被常规用于模型选择和参数调优，其选择结果常被解释为实质性的科学结论（例如，哪些变量、机制或风险因素“得到数据支持”）。CV流程的一个关键部分——保留集大小，或等效的折数$K$——通常依据惯例设定（例如80/20划分、$K=5$），而非基于有原则的标准。此问题的核心在于训练与测试之间的权衡：增加训练样本量可提升模型精度，但会牺牲对精度本身估计的确定性。我们通过以预测性能为目标并显式惩罚评估不确定性来形式化这一权衡，而评估不确定性在没有额外假设的情况下无法仅从数据中识别。我们在对称误差条件下推导了该评估不确定性的有限样本表达式，并在更广泛的误差条件下给出了一般上界，从而得到一个基于效用、透明的规则，用于根据不可约噪声参数选择保留集大小。通过线性回归和随机森林在多领域进行的实证分析，以及一个高维基因组学应用表明：（i）$K$的选择依赖于数据和模型；（ii）最优$K$随关于不可约误差的假设而变化；（iii）随着不可约误差及相应$K$的变化，所隐含的推断结论可能发生实质性改变。所提出的框架用特定情境下、假设明确的$K$选择替代了“一刀切”的惯例，从而能够实现更可靠的模型比较及后续科学推断。