Automatic evaluation metrics are crucial for advancing sign language translation (SLT). Current SLT evaluation metrics, such as BLEU and ROUGE, are only text-based, and it remains unclear to what extent text-based metrics can reliably capture the quality of SLT outputs. To address this gap, we investigate the limitations of text-based SLT evaluation metrics by analyzing six metrics, including BLEU, chrF, and ROUGE, as well as BLEURT on the one hand, and large language model (LLM)-based evaluators such as G-Eval and GEMBA zero-shot direct assessment on the other hand. Specifically, we assess the consistency and robustness of these metrics under three controlled conditions: paraphrasing, hallucinations in model outputs, and variations in sentence length. Our analysis highlights the limitations of lexical overlap metrics and demonstrates that while LLM-based evaluators better capture semantic equivalence often missed by conventional metrics, they can also exhibit bias toward LLM-paraphrased translations. Moreover, although all metrics are able to detect hallucinations, BLEU tends to be overly sensitive, whereas BLEURT and LLM-based evaluators are comparatively lenient toward subtle cases. This motivates the need for multimodal evaluation frameworks that extend beyond text-based metrics to enable a more holistic assessment of SLT outputs.

翻译：自动评估指标对于推进手语翻译（SLT）的发展至关重要。当前的手语翻译评估指标，如BLEU和ROUGE，仅基于文本，而文本指标能在多大程度上可靠地捕捉手语翻译输出的质量仍不明确。为填补这一空白，我们通过分析六种指标来研究基于文本的手语翻译评估指标的局限性，这些指标包括BLEU、chrF和ROUGE，以及BLEURT，另一方面还包括基于大型语言模型（LLM）的评估器，如G-Eval和GEMBA零样本直接评估。具体而言，我们在三种受控条件下评估这些指标的一致性和鲁棒性：释义、模型输出中的幻觉以及句子长度的变化。我们的分析突显了词汇重叠指标的局限性，并表明虽然基于LLM的评估器能更好地捕捉传统指标常忽略的语义等价性，但它们也可能对LLM释义的翻译表现出偏见。此外，尽管所有指标都能检测幻觉，但BLEU往往过于敏感，而BLEURT和基于LLM的评估器对细微情况则相对宽容。这促使我们需要超越文本指标的多模态评估框架，以实现对手语翻译输出更全面的评估。