Detecting complex patterns in large volumes of event logs has diverse applications in various domains, such as business processes and fraud detection. Existing systems like ELK are commonly used to tackle this challenge, but their performance deteriorates for large patterns, while they suffer from limitations in terms of expressiveness and explanatory capabilities for their responses. In this work, we propose a solution that integrates a Complex Event Processing (CEP) engine into a broader query processsor on top of a decoupled storage infrastructure containing inverted indices of log events. The results demonstrate that our system excels in scalability and robustness, particularly in handling complex queries. Notably, our proposed system delivers responses for large complex patterns within seconds, while ELK experiences timeouts after 10 minutes. It also significantly outperforms solutions relying on FlinkCEP and executing MATCH_RECOGNIZE SQL queries.

翻译：在大规模事件日志中检测复杂模式在业务流程、欺诈检测等众多领域具有广泛应用。现有系统如ELK虽常被用以应对这一挑战，但在处理大型模式时性能下降，且其在表达能力和响应解释能力方面存在局限。本研究提出一种解决方案，将复杂事件处理引擎集成到更广泛的查询处理器中，该处理器构建于包含日志事件倒排索引的解耦存储基础设施之上。结果表明，我们的系统在可扩展性和鲁棒性方面表现卓越，尤其擅长处理复杂查询。值得注意的是，所提系统可在数秒内响应大型复杂模式，而ELK在10分钟后即超时。该系统同时显著优于依赖FlinkCEP及执行MATCH_RECOGNIZE SQL查询的解决方案。