Underpowered studies (below 50% power) suffer from the winner's curse: A statistically significant positive estimate must exaggerate the true treatment effect to meet the significance threshold. A study by Dipayan Biswas, Annika Abell, and Roger Chacko published in the Journal of Consumer Research (2023) reported that in an A/B test, simply rounding the corners of square buttons increased the online click-through rate by 55% (p-value 0.037)$\unicode{x2014}$a striking finding with potentially wide-ranging implications for a digital industry that is seeking to enhance consumer engagement. Drawing on our experience with tens of thousands of A/B tests, many involving similar user interface modifications, we found this dramatic claim implausibly large. To evaluate the claim and provide a more accurate estimate of the treatment effect, we conducted three high-powered A/B tests, each involving over two thousand times more users than the original study. All three experiments yielded effect size estimates that were approximately two orders of magnitude smaller than initially reported, with 95% confidence intervals that include zero (i.e., not statistically significant at the 0.05 level). Two additional independent replications by Evidoo found similarly small effects. These findings underscore the critical importance of power analysis and experimental design in increasing trust and reproducibility of results.

翻译：统计检验力不足的研究受制于"赢家诅咒"现象：一项在统计上显著的正向估计值必须夸大真实处理效应才能达到显著性阈值。Dipayan Biswas、Annika Abell与Roger Chacko在《消费者研究期刊》（2023年）发表的研究报告称，在A/B测试中，仅将方形按钮的边角修改为圆角处理，即可使在线点击率提升55%（p值0.037），这一惊人发现可能对致力于提升用户参与度的数字产业产生广泛影响。基于我们数万次A/B测试（其中许多涉及相似的界面修改）的经验，我们认为这一戏剧性论断的效应量过于庞大而难以成立。为检验该论断并获取更精确的处理效应估计值，我们开展三项高检验力A/B测试，每项测试的样本量均为原研究的2000倍以上。三项实验所得的效应量估计值均比原始报告结果小约两个数量级，其95%置信区间包含零（即未达到0.05统计显著性水平）。Evidoo开展的两项独立重复实验同样发现效应量极小。这些发现凸显了检验力分析与实验设计在提升研究结果可信度与可重复性方面的关键作用。