Large language models (LLMs) are trained on broad corpora and then used in communities with specialized norms. Is providing LLMs with community rules enough for models to follow these norms? We evaluate LLMs' capacity to detect (Task 1) and correct (Task 2) biased Wikipedia edits according to Wikipedia's Neutral Point of View (NPOV) policy. LLMs struggled with bias detection, achieving only 64% accuracy on a balanced dataset. Models exhibited contrasting biases (some under- and others over-predicted bias), suggesting distinct priors about neutrality. LLMs performed better at generation, removing 79% of words removed by Wikipedia editors. However, LLMs made additional changes beyond Wikipedia editors' simpler neutralizations, resulting in high-recall but low-precision editing. Interestingly, crowdworkers rated AI rewrites as more neutral (70%) and fluent (61%) than Wikipedia-editor rewrites. Qualitative analysis found LLMs sometimes applied NPOV more comprehensively than Wikipedia editors but often made extraneous non-NPOV-related changes (such as grammar). LLMs may apply rules in ways that resonate with the public but diverge from community experts. While potentially effective for generation, LLMs may reduce editor agency and increase moderation workload (e.g., verifying additions). Even when rules are easy to articulate, having LLMs apply them like community members may still be difficult.

翻译：大语言模型（LLMs）在广泛语料上训练后，被应用于具有特定规范的社群。向LLMs提供社群规则是否足以使模型遵循这些规范？我们评估了LLMs根据维基百科中立观点（NPOV）政策检测（任务1）与纠正（任务2）偏见性编辑的能力。在偏见检测任务中，LLMs表现不佳，在平衡数据集上仅达到64%的准确率。模型展现出对比性偏差（部分模型低估偏差，另一部分则过度预测偏差），表明其对“中立性”存在不同的先验认知。LLMs在生成任务中表现更佳，能移除维基百科编辑删除的79%词汇。然而，LLMs在维基百科编辑更简洁的中立化修改之外进行了额外改动，导致高召回率但低精确度的编辑结果。值得注意的是，众包工作者认为AI改写比维基百科编辑的改写更具中立性（70%）和流畅性（61%）。定性分析发现，LLMs有时比维基百科编辑更全面地应用NPOV，但常进行与NPOV无关的额外修改（如语法修正）。LLMs可能以符合公众认知但偏离社区专家的方式应用规则。尽管LLMs在生成任务中可能有效，但可能削弱编辑自主权并增加审核工作量（例如验证新增内容）。即使规则易于表述，让LLMs像社区成员一样应用这些规则仍具挑战性。