The increased deployment of LMs for real-world tasks involving knowledge and facts makes it important to understand model epistemology: what LMs think they know, and how their attitudes toward that knowledge are affected by language use in their inputs. Here, we study an aspect of model epistemology: how epistemic markers of certainty, uncertainty, or evidentiality like "I'm sure it's", "I think it's", or "Wikipedia says it's" affect models, and whether they contribute to model failures. We develop a typology of epistemic markers and inject 50 markers into prompts for question answering. We find that LMs are highly sensitive to epistemic markers in prompts, with accuracies varying more than 80%. Surprisingly, we find that expressions of high certainty result in a 7% decrease in accuracy as compared to low certainty expressions; similarly, factive verbs hurt performance, while evidentials benefit performance. Our analysis of a popular pretraining dataset shows that these markers of uncertainty are associated with answers on question-answering websites, while markers of certainty are associated with questions. These associations may suggest that the behavior of LMs is based on mimicking observed language use, rather than truly reflecting epistemic uncertainty.

翻译：摘要： 随着语言模型在处理涉及知识与事实的现实任务中的应用日益增多，理解模型的认识论（即模型自认为所知的内容，以及输入中的语言使用如何影响其对所知内容的态度）变得至关重要。本文研究了模型认识论的一个方面：诸如“我确定它是”“我认为它是”或“维基百科说它是”等确定性、不确定性或证据性的认识标记如何影响模型，以及它们是否导致模型失效。我们开发了一套认识标记的类型学，并将50个标记注入问答提示中。研究发现，语言模型对提示中的认识标记高度敏感，准确率变化幅度超过80%。令人意外的是，与低确定性表达相比，高确定性表达会导致准确率下降7%；类似地，事实性动词会损害性能，而证据性标记则有利于性能。对热门预训练数据集的分析表明，这些不确定性标记与问答网站上的答案相关，而确定性标记则与问题相关。这些关联可能表明，语言模型的行为是基于模仿观察到的语言使用模式，而非真正反映认识论上的不确定性。