The distributed nature of microservice architecture introduces significant resilience challenges. Traditional testing methods, limited by extensive manual effort and oversimplified test environments, fail to capture production system complexity. To address these limitations, we present Cast, an automated, end-to-end framework for microservice resilience testing in production. It achieves high test fidelity by replaying production traffic against a comprehensive library of application-level faults to exercise internal error-handling logic. To manage the combinatorial test space, Cast employs a complexity-driven strategy to systematically prune redundant tests and prioritize high-value tests targeting the most critical service execution paths. Cast automates the testing lifecycle through a three-phase pipeline (i.e., startup, fault injection, and recovery) and uses a multi-faceted oracle to automatically verify system resilience against nuanced criteria. Deployed in Huawei Cloud for over eight months, Cast has been adopted by many service teams to proactively address resilience vulnerabilities. Our analysis on four large-scale applications with millions of traces reveals 137 potential vulnerabilities, with 89 confirmed by developers. To further quantify its performance, Cast is evaluated on a benchmark set of 48 reproduced bugs, achieving a high coverage of 90%. The results show that Cast is a practical and effective solution for systematically improving the reliability of industrial microservice systems.

翻译：微服务架构的分布式特性带来了显著的韧性挑战。传统测试方法受限于大量人工操作和过度简化的测试环境，难以捕捉生产系统的复杂性。为应对这些局限性，本文提出Cast——一个面向生产环境微服务韧性测试的自动化端到端框架。该框架通过在生产流量回放中注入应用级故障综合库，以激活内部错误处理逻辑，从而实现高测试保真度。为管理组合测试空间，Cast采用复杂度驱动策略，系统性地剪枝冗余测试，并优先针对最关键服务执行路径的高价值测试。Cast通过三阶段流水线（即启动、故障注入与恢复）实现测试生命周期自动化，并采用多维度预言机，依据精细标准自动验证系统韧性。在华为云部署八个月以来，Cast已被众多服务团队采用，用于主动发现韧性漏洞。基于对四个百万级调用链的大型应用的分析，Cast共发现137个潜在漏洞，其中89个获开发者确认。为进一步量化其性能，我们在包含48个复现缺陷的基准集上评估Cast，实现了90%的高覆盖率。结果表明，Cast是系统性提升工业级微服务系统可靠性的实用高效解决方案。