Disciplines such as business process management and process mining aid organizations by discovering insights about processes on the basis of recorded event data. However, an obstacle to process analysis is data multi-modality: for instance, data in video form are not directly interpretable as events. Existing approaches rely on a dictionary of activity label as input, cannot provide frame-by-frame labeling explanations, or rely on superseded computer vision techniques. In this work, we present SnapLog, an approach to extract event data from videos by converting frames to feature vectors using image embeddings and performing temporal segmentation through frame-wise similarity matrices. A generalized few-shot classification is then used to assign labels to the video segments, yielding labeled, timestamped sub-sequences of frames that are interpretable as events. Conventional process mining techniques can be used to analyze the resulting data. We show that our approach produces logs that accurately reflect the process in the videos.

翻译：业务流程管理和流程挖掘等学科通过分析记录的事件数据，帮助组织洞察流程。然而，流程分析面临的一大障碍是数据的多模态性：例如，视频形式的数据无法直接作为事件进行解读。现有方法要么依赖活动标签字典作为输入，要么无法提供逐帧的标注解释，要么依赖过时的计算机视觉技术。在本工作中，我们提出了SnapLog方法，通过使用图像嵌入将帧转换为特征向量，并利用帧间相似性矩阵进行时间分割，从视频中提取事件数据。随后，采用广义小样本分类为视频片段分配标签，生成带有时间戳的、可解释为事件的帧子序列。传统的流程挖掘技术可用于分析由此产生的数据。实验表明，我们的方法生成的日志能够准确反映视频中的流程。