Large Language Models (LLMs) are revolutionizing various domains, yet verifying their answers remains a significant challenge, especially for intricate open-ended tasks such as consolidation, summarization, and extraction of knowledge. In this work, we propose CheckEmbed: an accurate, scalable, and simple LLM verification approach. CheckEmbed is driven by a straightforward yet powerful idea: in order to compare LLM solutions to one another or to the ground-truth, compare their corresponding answer-level embeddings obtained with a model such as GPT Text Embedding Large. This reduces a complex textual answer to a single embedding, facilitating straightforward, fast, and meaningful verification. We develop a comprehensive verification pipeline implementing the CheckEmbed methodology. The CheckEmbed pipeline also comes with metrics for assessing the truthfulness of the LLM answers, such as embedding heatmaps and their summaries. We show how to use these metrics for deploying practical engines that decide whether an LLM answer is satisfactory or not. We apply the pipeline to real-world document analysis tasks, including term extraction and document summarization, showcasing significant improvements in accuracy, cost-effectiveness, and runtime performance compared to existing token-, sentence-, and fact-level schemes such as BERTScore or SelfCheckGPT.

翻译：大语言模型（LLM）正在深刻变革众多领域，然而对其生成答案的验证仍面临重大挑战，尤其对于知识整合、摘要生成和信息抽取等复杂的开放式任务。本研究提出CheckEmbed：一种精确、可扩展且简洁的LLM验证方法。该方法基于一个简洁而强大的核心思想：为比较LLM生成的解决方案之间或与标准答案的差异，可通过GPT Text Embedding Large等模型获取答案级别的嵌入表示进行比对。该策略将复杂的文本答案简化为单一嵌入向量，从而实现直接、快速且具有语义意义的验证。我们开发了完整实现CheckEmbed方法的验证流程体系。该流程同时配备了评估LLM答案真实性的量化指标，例如嵌入热力图及其统计摘要。我们进一步演示如何运用这些指标构建实际决策引擎，以判定LLM答案是否满足要求。通过将该流程应用于真实场景的文档分析任务（包括术语抽取与文档摘要），本方法相较于现有的基于词元、句子和事实层面的验证方案（如BERTScore或SelfCheckGPT），在准确率、成本效益和运行效率方面均展现出显著优势。