Materials discovery driven by statistical property models is an iterative decision process, during which an initial data collection is extended with new data proposed by a model-informed acquisition function--with the goal to maximize a certain "reward" over time, such as the maximum property value discovered so far. While the materials science community achieved much progress in developing property models that predict well on average with respect to the training distribution, this form of in-distribution performance measurement is not directly coupled with the discovery reward. This is because an iterative discovery process has a shifting reward distribution that is over-proportionally determined by the model performance for exceptional materials. We demonstrate this problem using the example of bulk modulus maximization among double perovskite oxides. We find that the in-distribution predictive performance suggests random forests as superior to Gaussian process regression, while the results are inverse in terms of the discovery rewards. We argue that the lack of proper performance estimation methods from pre-computed data collections is a fundamental problem for improving data-driven materials discovery, and we propose a novel such estimator that, in contrast to na\"ive reward estimation, successfully predicts Gaussian processes with the "expected improvement" acquisition function as the best out of four options in our demonstrational study for double perovskites. Importantly, it does so without requiring the over thousand ab initio computations that were needed to confirm this prediction.

翻译：基于统计属性模型的材料发现是一个迭代决策过程，在此过程中，初始数据集通过模型驱动的采集函数所推荐的新数据得以扩展——其目标是随时间最大化某种"回报"，例如当前已发现的最大属性值。尽管材料科学界在开发能对训练分布进行平均良好预测的属性模型方面取得了显著进展，但这种分布内性能评估方式与发现回报并无直接关联。这是因为迭代发现过程具有动态变化的回报分布，其决定性权重过比例地取决于模型对异常材料的预测性能。我们通过双钙钛矿氧化物体模量最大化案例论证了该问题。研究发现，分布内预测性能指标显示随机森林优于高斯过程回归，但在发现回报方面结果恰恰相反。我们认为，缺乏基于预计算数据集的合理性能评估方法是制约数据驱动材料发现发展的根本问题，并据此提出一种新型评估方法。与朴素回报估计不同，该方法成功识别出采用"期望改进"采集函数的高斯过程在双钙钛矿示范研究的四种候选方案中表现最优。关键的是，该评估方法无需进行原本为验证该结论所需的逾千次第一性原理计算。