Business Process Simulation (BPS) is an approach to analyze the performance of business processes under different scenarios. For example, BPS allows us to estimate what would be the cycle time of a process if one or more resources became unavailable. The starting point of BPS is a process model annotated with simulation parameters (a BPS model). BPS models may be manually designed, based on information collected from stakeholders and empirical observations, or automatically discovered from execution data. Regardless of its origin, a key question when using a BPS model is how to assess its quality. In this paper, we propose a collection of measures to evaluate the quality of a BPS model w.r.t. its ability to replicate the observed behavior of the process. We advocate an approach whereby different measures tackle different process perspectives. We evaluate the ability of the proposed measures to discern the impact of modifications to a BPS model, and their ability to uncover the relative strengths and weaknesses of two approaches for automated discovery of BPS models. The evaluation shows that the measures not only capture how close a BPS model is to the observed behavior, but they also help us to identify sources of discrepancies.

翻译：业务流程仿真（BPS）是一种在不同场景下分析业务流程性能的方法。例如，BPS可估算当一项或多项资源不可用时流程的周期时间。BPS的起点是标注了仿真参数的流程模型（即BPS模型）。这类模型可通过基于利益相关者信息与经验观察的手工设计方式构建，也可由执行数据自动发现。无论其来源如何，使用BPS模型时的一个关键问题是如何评估其质量。本文提出了一套衡量指标，用于评估BPS模型在复现流程观测行为方面的能力。我们主张采用多维度评估方法，使不同指标分别针对流程的不同视角。通过实验验证了所提指标对BPS模型修改影响的辨别能力，以及揭示两种BPS模型自动发现方法相对优劣的能力。评估结果表明，这些指标不仅能反映BPS模型与观测行为的接近程度，还能帮助识别差异产生的根源。