Federal agencies are deploying large language models (LLMs) to categorize public comment corpora, where the model's organization of the record shapes what policymakers see and which arguments register. Standard evaluation, anchored on stance accuracy against a small validated set, cannot detect when different models produce materially different categorizations of the same public input. We propose an Interpretive Audit Pipeline that treats multi-model disagreement as diagnostic of interpretive complexity and directs human review toward genuinely ambiguous public input. Analyzing 1,260 public comments on a federal USDA docket across four LLMs, we find that inter-model thematic divergence exceeds within-model prompt variation, and that an expert rubric suppresses deep interpretive disagreement without resolving it. In a two-stage labeling study on a stratified 40-comment subsample, four LLMs and a human annotator labeled independently and then revised after seeing the others' labels. Revision behavior varied across labelers, and the human annotator's revisions frequently introduced framings absent from the ensemble's collective output. We argue disagreement-based evaluation is a necessary complement to accuracy metrics for LLM-assisted interpretive coding.

翻译：联邦机构正部署大型语言模型（LLMs）对公众评论语料库进行分类，模型的记录组织方式决定了政策制定者能看到什么内容以及哪些论点会被纳入考量。标准评估方法——基于对小型验证集的立场准确率检验——无法检测出不同模型对同一公众输入产生实质性不同分类的情况。我们提出一种解释性审计框架，将多模型间的分歧视为解释复杂性的诊断指标，并引导人工审查聚焦于真正存在歧义的公众意见。通过对美国农业部联邦案卷中1260条公众评论在四个LLMs上的分析，我们发现：模型间的主题分歧超过了模型内提示词的变异程度，而专家评判标准虽能压制深层解释性分歧却无法真正化解分歧。在一项针对分层抽样40条评论子样本的两阶段标注研究中，四位LLM与一位人类标注员先独立标注，再观察他人标注结果后修正。不同标注者的修正行为存在差异，且人类标注员引入的修正版本常包含模型集成输出中未出现的框架。我们认为，基于分歧的评估方法是LLM辅助解释性编码中准确率指标的必要补充。