Provenance-based Intrusion Detection Systems (PIDSes) have been widely used to detect Advanced Persistent Threats (APTs). Although many studies achieve high performance in the evaluations of their original papers, their performance in industrial scenarios remains unclear. To fill this gap, we conduct the first systematic evaluation and analysis of PIDSes in industrial scenarios. We first analyze the differences between the data from DARPA datasets and that collected in industrial scenarios, identifying three main new characteristics in industry: heterogeneous multi-source inputs, more powerful attackers, and increasing benign activity complexity. We then build several datasets to evaluate five state-of-the-art PIDSes. The evaluation results reveal challenges for existing PIDSes, including poor portability across different hosts and platforms, low detection performance against real-world attacks, and high false positive rates with ever-changing benign activities. Based on the evaluation results and our industrial practices, we provide several insights to solve or explain the above problems. For example, we propose a method to mitigate the high false positives, which reduces manual effort by 2/3. Finally, we propose several research suggestions to improve PIDSes.

翻译：基于溯源的入侵检测系统（PIDSes）已被广泛用于检测高级持续性威胁（APTs）。尽管许多研究在其原始论文的评估中取得了高性能，但这些系统在工业场景中的表现仍不明确。为填补这一空白，我们首次对工业场景下的PIDSes进行了系统性评估与分析。我们首先分析了DARPA数据集与工业场景采集数据之间的差异，识别出工业环境中的三大新特征：异构多源输入、更强的攻击者以及日益增长的正常活动复杂性。随后，我们构建了多个数据集以评估五种最先进的PIDSes。评估结果揭示了现有PIDSes面临的挑战，包括跨不同主机与平台的可移植性差、针对真实攻击的检测性能低下以及在不断变化的正常活动中误报率高企的问题。基于评估结果与工业实践经验，我们提出了若干解决或解释上述问题的见解。例如，我们提出了一种缓解高误报率的方法，将人工工作量减少了2/3。最后，我们提出了若干改进PIDSes的研究建议。