Logs are a first-hand source of information for software maintenance and failure diagnosis. Log parsing, which converts semi-structured log messages into structured templates, is a prerequisite for automated log analysis tasks such as anomaly detection, troubleshooting, and root cause analysis. However, existing log parsers fail in real-world systems for three main reasons. First, traditional heuristics-based parsers require handcrafted features and domain knowledge, which are difficult to generalize at scale. Second, existing large language model-based parsers rely on periodic offline processing, limiting their effectiveness in real-time use cases. Third, existing online parsing algorithms are susceptible to log drift, where slight log changes create false positives that drown out real anomalies. To address these challenges, we propose HELP, a Hierarchical Embeddings-based Log Parser. HELP is the first online semantic-based parser to leverage LLMs for performant and cost-effective log parsing. We achieve this through a novel hierarchical embeddings module, which fine-tunes a text embedding model to cluster logs before parsing, reducing querying costs by multiple orders of magnitude. To combat log drift, we also develop an iterative rebalancing module, which periodically updates existing log groupings. We evaluate HELP extensively on 14 public large-scale datasets, showing that HELP achieves significantly higher F1-weighted grouping and parsing accuracy than current state-of-the-art online log parsers. We also implement HELP into Iudex's production observability platform, confirming HELP's practicality in a production environment. Our results show that HELP is effective and efficient for high-throughput real-world log parsing.

翻译：日志是软件维护与故障诊断的一手信息来源。日志解析将半结构化日志消息转换为结构化模板，是异常检测、故障排查与根因分析等自动化日志分析任务的前提。然而，现有日志解析器在现实系统中常因三大原因失效。首先，传统的基于启发式规则的解析器需要手工设计特征与领域知识，难以大规模泛化。其次，现有基于大语言模型的解析器依赖周期性离线处理，限制了其在实时场景中的有效性。第三，现有的在线解析算法易受日志漂移影响，即细微的日志变化会产生大量误报，淹没真实异常。为应对这些挑战，我们提出了HELP，一种基于分层嵌入的日志解析器。HELP是首个利用大语言模型实现高性能、低成本日志解析的在线语义解析器。我们通过新颖的分层嵌入模块实现这一目标，该模块对文本嵌入模型进行微调，在解析前对日志进行聚类，将查询成本降低数个数量级。为应对日志漂移，我们还开发了迭代再平衡模块，定期更新现有日志分组。我们在14个公开大规模数据集上对HELP进行了广泛评估，结果表明HELP在加权F1分组与解析准确率上显著优于当前最先进的在线日志解析器。我们还将HELP集成至Iudex生产可观测性平台，验证了HELP在生产环境中的实用性。实验结果表明，HELP对高吞吐量的现实日志解析任务具有高效性与实用性。