Industrial systems are increasingly threatened by cyberattacks with potentially disastrous consequences. To counter such attacks, industrial intrusion detection systems strive to timely uncover even the most sophisticated breaches. Due to its criticality for society, this fast-growing field attracts researchers from diverse backgrounds, resulting in 130 new detection approaches in 2021 alone. This huge momentum facilitates the exploration of diverse promising paths but likewise risks fragmenting the research landscape and burying promising progress. Consequently, it needs sound and comprehensible evaluations to mitigate this risk and catalyze efforts into sustainable scientific progress with real-world applicability. In this paper, we therefore systematically analyze the evaluation methodologies of this field to understand the current state of industrial intrusion detection research. Our analysis of 609 publications shows that the rapid growth of this research field has positive and negative consequences. While we observe an increased use of public datasets, publications still only evaluate 1.3 datasets on average, and frequently used benchmarking metrics are ambiguous. At the same time, the adoption of newly developed benchmarking metrics sees little advancement. Finally, our systematic analysis enables us to provide actionable recommendations for all actors involved and thus bring the entire research field forward.

翻译：工业系统日益受到网络攻击的威胁，这些攻击可能造成灾难性后果。为应对此类攻击，工业入侵检测系统致力于及时揭露甚至最复杂的入侵行为。由于其对社会的重要性，这一快速发展的领域吸引了来自不同背景的研究人员，仅2021年就涌现出130种新的检测方法。这种巨大动能促进了多样化有前途路径的探索，但同时也面临研究格局碎片化、有价值进展被埋没的风险。因此，该领域亟需严谨且全面的评估来降低这一风险，并将各方努力转化为具有现实适用性的可持续科学进展。本文系统分析了该领域的评估方法论，以理解工业入侵检测研究的当前状态。我们对609篇论文的分析表明，该研究领域的快速增长兼具积极与消极影响。尽管观察到公共数据集的使用有所增加，但论文平均仅评估1.3个数据集，且常用的基准测试指标存在歧义。同时，新开发的基准测试指标采用进展甚微。最后，我们的系统性分析能够为所有相关参与者提供可操作的建议，从而推动整个研究领域向前发展。