Observability and alerting form the backbone of modern reliability engineering. Alerts help teams catch faults early before they turn into production outages and serve as first clues for troubleshooting. However, designing effective alerts is challenging. They need to strike a fine balance between catching issues early and minimizing false alarms. On top of this, alerts often cover uncommon faults, so the code is rarely executed and therefore rarely checked. To address these challenges, several industry practitioners advocate for testing alerting code with the same rigor as application code. Still, there's a lack of tools that support such systematic design and validation of alerts. This paper introduces a new alerting extension for the observability experimentation tool OXN. It lets engineers experiment with alerts early during development. With OXN, engineers can now tune rules at design time and routinely validate the firing behavior of their alerts, avoiding future problems at runtime.

翻译：可观测性与告警构成了现代可靠性工程的基石。告警能帮助团队在故障演变为生产中断前及早发现，并为故障排查提供首要线索。然而，设计有效的告警具有挑战性：它们需要在及早发现问题与最小化误报之间取得精细平衡。此外，告警通常涵盖罕见故障，导致相关代码极少执行，因而也极少被检查。为应对这些挑战，多位行业实践者主张以与应用代码同等的严格程度测试告警代码。但目前仍缺乏支持此类系统性告警设计与验证的工具。本文为可观测性实验工具OXN引入了一种新型告警扩展功能。该功能使工程师能在开发早期对告警进行实验。通过OXN，工程师可在设计阶段调整规则，并常规化验证告警的触发行为，从而避免未来在运行时出现问题。