We study a statistical framework for replicability based on a recently proposed quantitative measure of replication success, the sceptical $p$-value. A recalibration is proposed to obtain exact overall Type-I error control if the effect is null in both studies and additional bounds on the partial and conditional Type-I error rate, which represent the case where only one study has a null effect. The approach avoids the double dichotomization for significance of the two-trials rule and has larger project power to detect existing effects over both studies in combination. It can also be used for power calculations and requires a smaller replication sample size than the two-trials rule for already convincing original studies. We illustrate the performance of the proposed methodology in an application to data from the Experimental Economics Replication Project.
翻译:我们研究了一种基于最新提出的量化重复成功度量——怀疑p值的统计框架,用于评估可重复性。提出了一种重新校准方法,以实现两个研究效应均为零时的精确总体I类错误控制,并对部分和条件I类错误率(仅一个研究效应为零的情况)进行额外约束。该方法避免了双重二分法对两试验规则显著性的划分,在联合检测两个研究中存在的效应时具有更高的项目检验功效。它还可用于功效计算,且对于已有说服力的原始研究,其所需的重复样本量小于两试验规则。我们通过实验经济学重复项目的数据应用,展示了所提出方法的性能。