This is the first work to report on inferential testing at scale in industry. Specifically, it reports the experience of automated testing of integrity systems at Meta. We built an internal tool called ALPACAS for automated inference of end-to-end integrity tests. Integrity tests are designed to keep users safe online by checking that interventions take place when harmful behaviour occurs on a platform. ALPACAS infers not only the test input, but also the oracle, by observing production interventions to prevent harmful behaviour. This approach allows Meta to automate the process of generating integrity tests for its platforms, such as Facebook and Instagram, which consist of hundreds of millions of lines of production code. We outline the design and deployment of ALPACAS, and report results for its coverage, number of tests produced at each stage of the test inference process, and their pass rates. Specifically, we demonstrate that using ALPACAS significantly improves coverage from a manual test design for the particular aspect of integrity end-to-end testing it was applied to. Further, from a pool of 3 million data points, ALPACAS automatically yields 39 production-ready end-to-end integrity tests. We also report that the ALPACAS-inferred test suite enjoys exceptionally low flakiness for end-to-end testing with its average in-production pass rate of 99.84%.

翻译：本文首次报道了工业界大规模推断性测试的实践。具体而言，文章阐述了Meta公司完整性系统自动化测试的经验。我们构建了名为ALPACAS的内部工具，用于自动化推断端到端完整性测试。完整性测试通过检测平台上有害行为发生时是否触发干预措施，旨在保障用户在线安全。ALPACAS不仅推断测试输入，还通过观察生产环境中防止有害行为的干预措施来推断测试预言。该方法使Meta能够为旗下Facebook和Instagram等平台（拥有数亿行生产代码）自动化生成完整性测试流程。我们概述了ALPACAS的设计与部署，并报告了其覆盖率、测试推断各阶段生成的测试数量及通过率。具体而言，我们证明在特定完整性端到端测试环节中，使用ALPACAS显著提升了相较于手工测试设计的覆盖率。此外，从300万个数据点中，ALPACAS自动生成了39个可直接应用于生产环境的端到端完整性测试。我们同时报告称，经ALPACAS推断的测试套件展现出极低的端到端测试波动性，其生产环境平均通过率达99.84%。