Underpowered studies (below 50%) suffer from the winner's curse: A statistically significant result must exaggerate the true treatment effect to meet the significance threshold. A study by Dipayan Biswas, Annika Abell, and Roger Chacko published in the Journal of Consumer Research (2023) reported that in an A/B test simply rounding the corners of square buttons increased the online click-through rate by 55% (p-value 0.037)$\unicode{x2014}$a striking finding with potentially wide-ranging implications for a digital industry that is seeking to enhance consumer engagement. Drawing on our experience with tens of thousands of A/B tests, many involving similar user interface modifications, we found this dramatic claim implausibly large. To evaluate the claim, and provide a more accurate estimate of the treatment effect, we conducted three high-powered A/B tests, each involving over two thousand times more users than the original study. All three experiments yielded effect size estimates that were approximately two orders of magnitude smaller than initially reported, with 95% confidence intervals that include zero, that is, not statistically significant at the 0.05 level. Two additional independent replications by Evidoo found similarly small effects. These findings underscore the critical importance of power analysis and experimental design in increasing trust and reproducibility of results.

翻译：功效不足的研究（低于50%）会遭受“赢者诅咒”：一个具有统计显著性的结果为了达到显著性阈值，必然会夸大真实的处理效应。Dipayan Biswas、Annika Abell和Roger Chacko在《消费者研究杂志》（2023）上发表的一项研究报告称，在一项A/B测试中，仅仅将方形按钮的边角变圆就能使在线点击率提高55%（p值0.037）——这一惊人发现对于寻求提升消费者参与度的数字产业可能具有广泛影响。基于我们处理数万次A/B测试（其中许多涉及类似的用户界面修改）的经验，我们发现这一戏剧性的主张大得令人难以置信。为了评估该主张并提供更准确的处理效应估计，我们进行了三项高功效的A/B测试，每项测试涉及的用户数量都是原始研究的2000倍以上。所有三项实验得出的效应量估计值均比最初报告的小约两个数量级，其95%置信区间包含零，即在0.05水平上不具有统计显著性。Evidoo进行的另外两项独立复制研究也发现了类似微小的效应。这些发现强调了功效分析和实验设计对于提高结果可信度与可重复性的至关重要性。