Vision-language models (VLMs) are increasingly being explored for video game quality assurance, especially gameplay glitch detection. Most existing evaluations, however, treat glitches as static visual anomalies, asking models to detect failures from a single frame. We argue that this framing misses a key distinction: some glitches are spatial and visible in an isolated frame, whereas others are temporal and become evident only through changes across ordered frames. A preliminary study confirms this gap, showing that temporal glitches are substantially harder for VLMs to detect than spatial ones. To enable systematic evaluation of this underexplored setting, we introduce TempGlitch, a controlled gameplay video benchmark for temporal glitch detection. TempGlitch covers five temporal glitch types with balanced per-category samples, together with paired glitch-free videos that enable reliable binary evaluation. We evaluate 12 proprietary and open-weight VLMs across multiple frame-sampling settings. Our results show that current VLMs remain near chance on TempGlitch, often collapsing into either overly conservative behavior that misses most glitches or overly sensitive behavior that flags clean videos as glitchy. Moreover, denser frame sampling and larger model size do not reliably resolve these failures. TempGlitch provides a focused testbed for temporal reasoning, robust gameplay understanding, and automated glitch detection with VLMs. Code and data are available at the project website.

翻译：摘要：视觉-语言模型（VLM）正被日益探索应用于视频游戏质量保证，尤其是游戏玩法故障检测。然而，现有评估大多将故障视为静态视觉异常，要求模型从单帧中检测错误。本文认为，这种设定忽略了关键区分：部分故障属于空间性故障，在孤立帧中即可显现；另一些则属于时序性故障，仅通过有序帧间的变化才能暴露。初步研究证实了这一鸿沟，表明VLM对时序性故障的检测难度远高于空间性故障。为系统评估这一未被充分探索的场景，我们提出了TempGlitch——一个面向时序性故障检测的受控游戏视频基准。TempGlitch涵盖五种时序性故障类型，各类别样本均衡，并配有成对的无故障视频，从而支持可靠的二元评估。我们评估了12个开源与闭源VLM在多种帧采样设置下的表现。结果表明，当前VLM在TempGlitch上的性能接近随机水平，往往陷入两种极端：要么过度保守而遗漏多数故障，要么过度敏感而将干净视频误判为故障。此外，更密集的帧采样和更大的模型规模并未可靠解决这些失败。TempGlitch为时序推理、稳健的游戏理解以及基于VLM的自动化故障检测提供了聚焦性测试平台。代码与数据见项目网站。