Sign Language Translation (SLT) is typically evaluated with surface-form metrics such as BLEU and ROUGE, which reward lexical overlap but do not directly measure whether a translation preserves the meaning of the source sign sequence. This is in contrast with the final objective of integrating SLT in assistive technology. In this work, we shift the focus from Sign Language Translation (SLT) to Sign Language Understanding (SLU), with particular emphasis on semantic understanding. Specifically, we evaluate systems based on their ability to correctly recover, from the input video, key semantic aspects of the original sentence, such as actions taking place and facts about people and objects. To enable this evaluation systematically, we propose SLU-2K, a dataset of 2,350 closed-ended video question-answer pairs based on the popular PHOENIX-2014T and CSL-Daily datasets. To obtain SLU-2K, we propose and extensively evaluate an automated data generation pipeline which produces questions across 7 categories, namely actions, locations, numbers, objects, people, time, and weather conditions. We show the potential of SLU-2K by evaluating popular Multimodal Large Language Models (MLLMs) and two representative state-of-the-art systems, MMSTL and SpaMo. Our results show that MLLMs reach near-random performance, highlighting the need for a more systematic integration of SLU in current AI systems. Furthermore, state-of-the-art translation systems carefully fine-tuned on in-domain data still exhibit a substantial semantic gap, with results ranging from 56.7% to 75.2%. These findings suggest that current SLT evaluation protocols overestimate true understanding and that future progress should be measured not only by fluency and n-gram overlap, but also by semantic correctness. Code, prompts, and benchmark files are available at https://github.com/ZenoTsT/SLU-2K

翻译：手语翻译通常采用BLEU和ROUGE等表层形式指标进行评估，这些指标奖励词汇重叠，但并未直接衡量翻译是否保留了源手语序列的语义。这与将手语翻译集成到辅助技术中的最终目标相悖。本研究将焦点从手语翻译转向手语理解，特别强调语义理解。具体而言，我们根据系统从输入视频中正确恢复原句关键语义方面（如正在发生的动作以及关于人物和物体的事实）的能力来评估系统。为实现系统性评估，我们提出SLU-2K数据集，该数据集基于流行的PHOENIX-2014T和CSL-Daily数据集，包含2,350个封闭式视频问答对。为构建SLU-2K，我们提出并广泛评估了一个自动化数据生成流程，该流程生成涵盖动作、位置、数字、物体、人物、时间和天气状况7个类别的问题。我们通过评估流行的多模态大语言模型和两种代表性最先进系统（MMSTL和SpaMo）展示了SLU-2K的潜力。结果显示，多模态大语言模型的表现接近随机水平，突显了当前人工智能系统中更系统地整合手语理解的需求。此外，在域内数据上精心微调的最先进翻译系统仍存在显著的语义鸿沟，结果介于56.7%至75.2%之间。这些发现表明，当前手语翻译评估协议高估了真实理解能力，未来进展不仅应通过流畅度和n-gram重叠来衡量，还应通过语义正确性来评估。代码、提示词和基准文件可在https://github.com/ZenoTsT/SLU-2K获取。