With recent semantic query processing engines, semantic aggregation has become a primitive operator, enabling the reduction of a relation into a natural language aggregate using an LLM. However, the resulting semantic aggregate may contain claims that are not grounded in the underlying relation. Verifying such claims is challenging: they often involve quantifiers, groupings, and comparisons over relations that far exceed LLM context windows and require a costly combination of semantic and symbolic processing. We present Evergreen, a system that recasts claim verification as a semantic query processing task with tailored optimizations and provenance capture. Evergreen compiles each claim into a declarative semantic verification query and executes it on the same engine that produced the aggregate. To reduce cost and latency, Evergreen avoids unnecessary LLM calls through verification-aware optimizations (early stopping, relevance sorting, and estimation with confidence sequences) and general-purpose optimizations for semantic queries (operator fusion, similarity filtering, and prompt caching). Each verdict is accompanied by citations that identify a minimal set of tuples justifying the result, with semantics based on semiring provenance for first-order logic. On a benchmark of real-world restaurant review datasets reflecting production-inspired workloads, Evergreen achieves excellent verification quality (F1 = 1.00) with a strong LLM while reducing cost by 3.2x and latency by 4.0x compared to unoptimized verification. Even with a significantly weaker LLM, Evergreen outperforms a strong LLM-as-a-judge baseline in F1 at 48x lower cost and 2.3x lower latency. Relative to a retrieval-augmented agent, Evergreen compares favorably in F1 and latency with similar cost when both use a strong LLM; yet, with a much weaker LLM, it achieves the same F1 at 63x lower cost and 4.2x lower latency.
翻译:随着近期语义查询处理引擎的发展,语义聚合已成为一种基础算子,能够利用大语言模型将关系数据缩减为自然语言聚合结果。然而,生成的语义聚合可能包含与底层关系数据不符的声明。验证此类声明极具挑战性:它们往往涉及量词、分组以及远超大语言模型上下文窗口的关系比较,需要结合语义处理与符号处理的高昂成本。本文提出长青系统,将声明验证重构为语义查询处理任务,并融入定制化优化与溯源捕获机制。长青将每条声明编译为声明式语义验证查询,在生成聚合结果的同一引擎上执行。为降低开销与延迟,长青通过验证感知优化(早停策略、相关性排序、置信序列估计)和语义查询通用优化(算子融合、相似性过滤、提示缓存)避免不必要的语言模型调用。每个验证结论均附带引文,标识证明结果的最小元组集合,其语义基于一阶逻辑的半环溯源。在反映生产场景的真实餐厅评论数据集基准测试中,长青使用强语言模型时验证质量优异(F1=1.00),与未优化验证相比成本降低3.2倍、延迟降低4.0倍。即使使用显著较弱的语言模型,长青在F1指标上仍优于强语言模型评判基线,且成本降低48倍、延迟降低2.3倍。相较于检索增强代理,当两者均使用强语言模型时,长青在F1与延迟方面表现相当且成本相近;但若使用极弱语言模型,长青在F1保持不变的情况下,成本降低63倍、延迟降低4.2倍。