The question of whether $Y$ can be predicted based on $X$ often arises and while a well adjusted model may perform well on observed data, the risk of overfitting always exists, leading to poor generalization error on unseen data. This paper proposes a rigorous permutation test to assess the credibility of high $R^2$ values in regression models, which can also be applied to any measure of goodness of fit, without the need for sample splitting, by generating new pairings of $(X_i, Y_j)$ and providing an overall interpretation of the model's accuracy. It introduces a new formulation of the null hypothesis and justification for the test, which distinguishes it from previous literature. The theoretical findings are applied to both simulated data and sensor data of tennis serves in an experimental context. The simulation study underscores how the available information affects the test, showing that the less informative the predictors, the lower the probability of rejecting the null hypothesis, and emphasizing that detecting weaker dependence between variables requires a sufficient sample size.

翻译：预测变量$Y$能否基于$X$进行预测是一个常见问题。尽管调整良好的模型在观测数据上可能表现优异，但过拟合风险始终存在，导致对未见数据的泛化误差较高。本文提出一种严格的置换检验方法，用于评估回归模型中高$R^2$值的可信度。该方法无需样本分割，通过生成$(X_i, Y_j)$的新配对，即可适用于任意拟合优度指标，并提供对模型准确性的整体解释。本文提出了新的零假设表述形式及其检验依据，这与现有文献存在显著区别。理论成果通过模拟数据及实验场景中的网球发球传感器数据得到应用。模拟研究揭示了可用信息对检验效力的影响：预测变量所含信息越少，拒绝零假设的概率越低，并强调检测变量间较弱依赖性需要足够的样本量。