Semantic consistency of a language model is broadly defined as the model's ability to produce semantically-equivalent outputs, given semantically-equivalent inputs. We address the task of assessing question-answering (QA) semantic consistency of contemporary large language models (LLMs) by manually creating a benchmark dataset with high-quality paraphrases for factual questions, and release the dataset to the community. We further combine the semantic consistency metric with additional measurements suggested in prior work as correlating with LLM QA accuracy, for building and evaluating a framework for factual QA reference-less performance prediction -- predicting the likelihood of a language model to accurately answer a question. Evaluating the framework on five contemporary LLMs, we demonstrate encouraging, significantly outperforming baselines, results.

翻译：语言模型的语义一致性广义上定义为模型在给定语义等价输入时，产生语义等价输出的能力。我们通过手动构建一个包含高质量事实性问题释义的基准数据集，研究当代大语言模型（LLM）在问答（QA）方面的语义一致性评估任务，并将该数据集公开发布给社区。我们进一步将语义一致性度量与先前研究提出的与LLM问答准确性相关的其他指标相结合，构建并评估了一个面向事实性问答的无参考性能预测框架——即预测语言模型准确回答问题的可能性。在五个当代LLM上评估该框架的结果表明，该方法显著优于基线模型，展现了令人鼓舞的表现。