As large language models (LLMs) become default tools for online information verification, an implicit assumption follows them: that scale and general capability are sufficient for nuanced classification of misinformation discourse. We test this assumption directly on 900 Reddit comments spanning three PolitiFact-verified misinformation claims (environment, health, immigration), labelled as belief (propagates the claim), fact-check (corrects it), or other. We compare nine models across three paradigms -- BART-MNLI, three Llama variants, three commercial frontier LLMs (Claude Haiku 4.5, Gemini Flash Lite 2.5, Claude Sonnet 4.6), and fine-tuned DistilBERT and RoBERTa -- under universal and topic-specific label schemas. The assumption does not hold. Fine-tuned RoBERTa reaches 0.62 macro-$F_1$ against a best zero-shot result of 0.50 (Claude Haiku 4.5), at a fraction of the per-query cost; the supervised advantage is concentrated on the belief class, the implicit, affective category every zero-shot model under-detects. Scaling does not help: Llama-3-8B matches Llama-3-70B, and Claude Sonnet 4.6 underperforms the smaller Haiku under generic labels, collapsing belief detection to 0.17 and refusing outright on a subset of comments flagged as sensitive. This is a safety-alignment artefact, not a capacity limit. Label schema and topic jointly shape zero-shot performance, with the same model varying by more than 0.13 macro-$F_1$ across topics under matched labels. In a verification context, where missing belief is the costlier error, task-specific fine-tuning remains the more reliable choice despite the proliferation of large generative models.
翻译:随着大语言模型成为在线信息验证的默认工具,一个隐含假设随之而来:模型的规模与通用能力足以对错误信息讨论进行细粒度分类。我们直接检验了这一假设,基于900条Reddit评论(涵盖三个经PolitiFact验证的错误信息主张,涉及环境、健康、移民领域),并将评论标注为“相信”(传播该主张)、“事实核查”(纠正该主张)或“其他”。我们比较了三种范式下的九种模型——BART-MNLI、三种Llama变体、三种商用前沿大语言模型(Claude Haiku 4.5、Gemini Flash Lite 2.5、Claude Sonnet 4.6)以及微调后的DistilBERT和RoBERTa——在通用标签体系和主题特定标签体系下进行实验。该假设不成立。微调后的RoBERTa在宏观$F_1$值上达到0.62,而最佳零样本结果为0.50(来自Claude Haiku 4.5),且其每次查询成本仅为后者的零头。监督学习的优势集中体现在“相信”这一类别——所有零样本模型均难以检测这种隐含情感类别的错误信息。模型规模扩大并无助益:Llama-3-8B与Llama-3-70B表现相当;Claude Sonnet 4.6在通用标签体系下表现不及较小的Haiku,其“相信”检测率骤降至0.17,并对部分被标记为敏感的评论完全拒绝作答。这是安全对齐机制的人为产物,而非模型能力局限。标签体系与主题共同影响零样本性能:在匹配标签条件下,同一模型在不同主题上的宏观$F_1$值差异超过0.13。在验证场景中(漏检“相信”类错误的代价更高),尽管大规模生成模型激增,面向任务的微调仍是更可靠的选择。