Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amongst other development milestones. Recently, Large Language Models (LLMs) have been used for a variety of tasks including automation of UIs, however to our knowledge no one has yet explored their use in controlling assistive technologies for the purposes of supporting accessibility testing. In this paper, we explore the requirements of a natural language based accessibility testing workflow, starting with a formative study. From this we build a system that takes as input a manual accessibility test (e.g., ``Search for a show in VoiceOver'') and uses an LLM combined with pixel-based UI Understanding models to execute the test and produce a chaptered, navigable video. In each video, to help QA testers we apply heuristics to detect and flag accessibility issues (e.g., Text size not increasing with Large Text enabled, VoiceOver navigation loops). We evaluate this system through a 10 participant user study with accessibility QA professionals who indicated that the tool would be very useful in their current work and performed tests similarly to how they would manually test the features. The study also reveals insights for future work on using LLMs for accessibility testing.

翻译：开发人员和质量保证测试人员通常在整个产品生命周期中依赖手动测试来验证可访问性功能。然而，手动测试过程繁琐、覆盖范围过大，且难以与其他开发里程碑协调安排。近年来，大语言模型已被用于包括用户界面自动化在内的多种任务，但据我们所知，尚无研究探索将其用于控制辅助技术以支持可访问性测试。本文首先通过形成性研究探索了基于自然语言的可访问性测试工作流需求，据此构建了一个系统：该系统以手动可访问性测试为输入（例如“在VoiceOver中搜索节目”），结合大语言模型与基于像素的用户界面理解模型执行测试，并生成带章节标记、可导航的视频。在每个视频中，我们应用启发式规则辅助质量保证测试人员检测并标记可访问性问题（例如：启用大文本后文本尺寸未增大、VoiceOver导航循环）。我们通过10名可访问性质量保证专业人员参与的实验评估了该系统，结果表明该工具对其当前工作具有显著实用价值，且测试执行方式与手动测试相似。该研究也为未来利用大语言模型进行可访问性测试提供了启示。