To evaluate knowledge in large language models (LLMs), current methods query the model and then evaluate its generated responses. In this work, we ask whether evaluation can be done $\textit{before}$ the model has generated any text. Concretely, is it possible to estimate how knowledgeable a model is about a certain entity, only from its internal computation? We study this question with two tasks: given a subject entity, the goal is to predict (a) the ability of the model to answer common questions about the entity, and (b) the factuality of responses generated by the model about the entity. Experiments with a variety of LLMs show that KEEN, a simple probe trained over internal subject representations, succeeds at both tasks - strongly correlating with both the QA accuracy of the model per-subject and FActScore, a recent factuality metric in open-ended generation. Moreover, KEEN naturally aligns with the model's hedging behavior and faithfully reflects changes in the model's knowledge after fine-tuning. Lastly, we show a more interpretable yet equally performant variant of KEEN, which highlights a small set of tokens that correlates with the model's lack of knowledge. Being simple and lightweight, KEEN can be leveraged to identify gaps and clusters of entity knowledge in LLMs, and guide decisions such as augmenting queries with retrieval.

翻译：当前评估大语言模型（LLMs）知识的方法通常需要查询模型并评估其生成的回答。在本研究中，我们探讨评估是否能在模型生成任何文本$\textit{之前}$进行。具体而言，是否可能仅通过模型的内部计算来估计其对特定实体的知识掌握程度？我们通过两个任务来研究这个问题：给定一个主体实体，目标是预测（a）模型回答关于该实体常见问题的能力，以及（b）模型生成的关于该实体回答的事实性。在多种LLMs上的实验表明，KEEN——一种基于内部主体表示训练的简单探针——在两项任务上均取得成功：其预测结果与模型在每个实体上的问答准确率以及FActScore（一种用于开放式生成的最新事实性度量指标）均呈现强相关性。此外，KEEN自然地与模型的模糊表达行为保持一致，并能忠实反映微调后模型知识的变化。最后，我们展示了一个更具可解释性且性能相当的KEEN变体，该变体能突显出一小部分与模型知识缺失相关的标记。KEEN方法简单轻量，可用于识别LLMs中实体知识的缺口与聚类，并指导诸如通过检索增强查询等决策。