Predictive Process Monitoring aims to forecast the future progress of process instances using historical event data. As predictive process monitoring is increasingly applied in online settings to enable timely interventions, evaluating the performance of the underlying models becomes crucial for ensuring their consistency and reliability over time. This is especially important in high risk business scenarios where incorrect predictions may have severe consequences. However, predictive models are currently usually evaluated using a single, aggregated value or a time-series visualization, which makes it challenging to assess their performance and, specifically, their stability over time. This paper proposes an evaluation framework for assessing the stability of models for online predictive process monitoring. The framework introduces four performance meta-measures: the frequency of significant performance drops, the magnitude of such drops, the recovery rate, and the volatility of performance. To validate this framework, we applied it to two artificial and two real-world event logs. The results demonstrate that these meta-measures facilitate the comparison and selection of predictive models for different risk-taking scenarios. Such insights are of particular value to enhance decision-making in dynamic business environments.

翻译：预测性流程监控旨在利用历史事件数据预测流程实例的未来进展。随着预测性流程监控越来越多地应用于在线场景以实现及时干预，评估底层模型的性能对于确保其随时间的一致性和可靠性至关重要。这在高风险业务场景中尤为重要——错误的预测可能引发严重后果。然而，当前预测模型通常仅通过单一聚合值或时间序列可视化进行评估，这使得评估其性能尤其是随时间变化的稳定性变得极具挑战性。本文提出了一种用于评估在线预测性流程监控模型稳定性的评估框架。该框架引入了四个性能元度量指标：显著性能下降的频率、下降幅度、恢复率以及性能波动性。为验证该框架，我们将其应用于两个人工事件日志和两个真实事件日志。实验结果表明，这些元度量能够有效促进不同风险承担场景下预测模型的比较与选择。此类洞察对于增强动态商业环境中的决策具有特殊价值。