Visual language model (VLM) agents show great promise in automating end-to-end (E2E) web testing against requirements in natural language. However, the probabilistic nature of language models can have inherent hallucinations. Therefore, given a detected inconsistency between the requirement and the web application, it is hard to distinguish whether it stems from the hallucination or a real application bug. Addressing this issue presents two core technical challenges: the implicit oracle inference challenge, where the agent must act as its own oracle to implicitly decide if the application's behavior is correct without guidance, and the probabilistic inference challenge, where an LLM's inconsistent reasoning undermines its trustworthiness as an oracle. Existing LLM-based approaches fail to capture such implicit oracles, either by treating any page navigation that doesn't crash as a success, or by checking each state in isolation, thus missing bugs dependent on context from prior steps. We introduce WebTestPilot, an LLM-based agent designed to address these challenges. WebTestPilot uses (1) a symbolization layer which detects and symbolizes critical GUI elements on the web application into symbols (i.e., variables) and (2) translates natural language specification into a sequence of steps, each of which is equipped with inferred pre- and post-conditions over the symbols as an oracle. This oracle captures data, temporal, and causal dependencies, enabling the validation of implicit requirements. To advance research in this area, we build a benchmark of bug-injected web apps for evaluating NL-to-E2E testing. The results show that WebTestPilot achieves a task completion rate of 99%, with 96% precision and 96% recall in bug detection, outperforming the best baseline (+70 precision, +27 recall). The agent generalizes across diverse natural language inputs and model scales.

翻译：视觉语言模型（VLM）智能体在根据自然语言需求自动化端到端（E2E）Web测试方面展现出巨大潜力。然而，语言模型的概率性本质可能导致固有的幻觉问题。因此，当检测到需求与Web应用之间存在不一致时，很难区分这种不一致是源于模型幻觉还是实际的应用缺陷。解决此问题面临两个核心技术挑战：一是隐式预言机推断挑战，即智能体必须充当自身的预言机，在没有明确指导的情况下隐式判断应用行为是否正确；二是概率推断挑战，即大语言模型（LLM）的不一致推理会削弱其作为预言机的可信度。现有基于LLM的方法未能捕获此类隐式预言机，它们要么将任何未崩溃的页面导航视为成功，要么孤立地检查每个状态，从而遗漏了依赖于先前步骤上下文的缺陷。我们提出了WebTestPilot，一种旨在应对这些挑战的基于LLM的智能体。WebTestPilot采用（1）符号化层，用于检测并将Web应用上的关键GUI元素符号化为符号（即变量），以及（2）将自然语言规约转化为一系列步骤，每个步骤都配备了基于这些符号推断的前置条件与后置条件作为预言机。该预言机捕获了数据、时间和因果依赖关系，从而能够验证隐式需求。为了推动该领域的研究，我们构建了一个包含缺陷注入的Web应用基准，用于评估自然语言到端到端测试的性能。结果表明，WebTestPilot实现了99%的任务完成率，在缺陷检测方面达到96%的精确率和96%的召回率，优于最佳基线方法（精确率提升70，召回率提升27）。该智能体能够泛化到多样化的自然语言输入和不同规模的模型。