Adaptive specification search generates statistically significant backtests even under martingale-difference nulls. We introduce a falsification audit testing complete predictive workflows against synthetic reference classes, including zero-predictability environments and microstructure placebos. Workflows generating significant walk-forward evidence in these environments are falsified. For passing workflows, we quantify selection-induced performance inflation using an absolute magnitude gap linking optimized in-sample evidence to disjoint walk-forward realizations, adjusted for effective multiplicity. Simulations validate extreme-value scaling under correlated searches and demonstrate detection power under genuine structure. Empirical case studies confirm that many apparent findings represent methodological artifacts rather than genuine predictability.
翻译:自适应规范搜索即便在鞅差原假设下也能产生统计上显著的回测结果。我们引入了一种伪造审计方法,针对合成参考类别(包括零可预测性环境与微观结构安慰剂)测试完整的预测工作流。在这些环境中产生显著滚动前向证据的工作流被判定为伪证。对于通过验证的工作流,我们通过绝对幅度差距量化选择导致的性能膨胀,该差距将优化的样本内证据与独立的滚动前向实现相联系,并针对有效多重性进行调整。模拟实验验证了相关搜索下的极值缩放特性,并展示了在真实结构下的检测能力。实证案例研究确认,许多显著发现实为方法论伪像而非真实可预测性。