Comparative simulation studies are workhorse tools for benchmarking statistical methods. As with other empirical studies, the success of simulation studies hinges on the quality of their design, execution and reporting. If not conducted carefully and transparently, their conclusions may be misleading. In this paper we discuss various questionable research practices which may impact the validity of simulation studies, some of which cannot be detected or prevented by the current publication process in statistics journals. To illustrate our point, we invent a novel prediction method with no expected performance gain and benchmark it in a pre-registered comparative simulation study. We show how easy it is to make the method appear superior over well-established competitor methods if questionable research practices are employed. Finally, we provide concrete suggestions for researchers, reviewers and other academic stakeholders for improving the methodological quality of comparative simulation studies, such as pre-registering simulation protocols, incentivizing neutral simulation studies and code and data sharing.

翻译：比较模拟研究是评估统计方法的常用工具。与其他实证研究一样，模拟研究的成功取决于其设计、执行和报告的质量。若未能严谨透明地开展，其结论可能具有误导性。本文讨论了可能影响模拟研究效度的各种可疑研究实践，其中部分实践在统计学期刊现行发表流程中无法被检测或预防。为阐明这一观点，我们发明了一种无预期性能提升的新型预测方法，并在预先注册的比较模拟研究中对其进行基准测试。我们展示了如果采用可疑研究实践，可以多么轻松地使该方法看起来优于公认的竞争者方法。最后，我们为研究人员、审稿人及其他学术利益相关者提供具体建议，以提升比较模拟研究的方法论质量，例如预注册模拟方案、激励中立性模拟研究以及代码与数据共享。