Traditional evaluation metrics for textual and visual question answering, like ROUGE, METEOR, and Exact Match (EM), focus heavily on n-gram based lexical similarity, often missing the deeper semantic understanding needed for accurate assessment. While measures like BERTScore and MoverScore leverage contextual embeddings to address this limitation, they lack flexibility in balancing sentence-level and keyword-level semantics and ignore lexical similarity, which remains important. Large Language Model (LLM) based evaluators, though powerful, come with drawbacks like high costs, bias, inconsistency, and hallucinations. To address these issues, we introduce SMILE: Semantic Metric Integrating Lexical Exactness, a novel approach that combines sentence-level semantic understanding with keyword-level semantic understanding and easy keyword matching. This composite method balances lexical precision and semantic relevance, offering a comprehensive evaluation. Extensive benchmarks across text, image, and video QA tasks show SMILE is highly correlated with human judgments and computationally lightweight, bridging the gap between lexical and semantic evaluation.

翻译：传统的文本与视觉问答评估指标，如ROUGE、METEOR和精确匹配（EM），主要侧重于基于n-gram的词汇相似度，往往忽略了准确评估所需的深层语义理解。尽管BERTScore和MoverScore等度量方法利用上下文嵌入来弥补这一局限，但它们在平衡句子级与关键词级语义方面缺乏灵活性，且忽略了仍具重要性的词汇相似度。基于大语言模型（LLM）的评估器虽然强大，却存在高成本、偏见、不一致性及幻觉等缺陷。为解决这些问题，我们提出了SMILE：语义度量整合词汇精确性，这是一种将句子级语义理解、关键词级语义理解及简易关键词匹配相结合的新方法。该复合方法平衡了词汇精确性与语义相关性，提供了全面的评估。在文本、图像及视频问答任务上的广泛基准测试表明，SMILE与人类判断高度相关，且计算轻量，有效弥合了词汇评估与语义评估之间的鸿沟。