While large language models (LLMs) excel at factual recall, the real challenge lies in knowledge application. A gap persists between their ability to answer complex questions and their effectiveness in performing tasks that require that knowledge. We investigate this gap using a patent classification problem that requires deep conceptual understanding to distinguish semantically similar but objectively different patents written in dense, strategic technical language. We find that LLMs often struggle with this distinction. To diagnose the source of these failures, we introduce a framework that decomposes model errors into two categories: missing knowledge and unused knowledge. Our method prompts models to generate clarifying questions and compares three settings -- raw performance, self-answered questions that activate internal knowledge, and externally provided answers that supply missing knowledge (if any). We show that most errors stem from failures to deploy existing knowledge rather than from true knowledge gaps. We also examine how models differ in constructing task-specific question-answer databases. Smaller models tend to generate simpler questions that they, and other models, can retrieve and use effectively, whereas larger models produce more complex questions that are less effective, suggesting complementary strengths across model scales. Together, our findings highlight that shifting evaluation from static fact recall to dynamic knowledge application offers a more informative view of model capabilities.

翻译：尽管大型语言模型（LLMs）在事实回忆方面表现出色，但真正的挑战在于知识的应用。它们回答复杂问题的能力与执行需要该知识的任务的有效性之间仍存在差距。我们通过一个专利分类问题来研究这一差距，该问题需要深入的概念理解以区分语义相似但客观不同的专利，这些专利以密集、策略性的技术语言撰写。我们发现LLMs经常难以进行这种区分。为了诊断这些失败的根源，我们引入了一个框架，将模型错误分解为两类：缺失知识和未用知识。我们的方法提示模型生成澄清性问题，并比较三种设置——原始性能、激活内部知识的自答问题，以及提供缺失知识（如有）的外部答案。我们表明，大多数错误源于未能部署现有知识，而非真正的知识缺口。我们还研究了模型在构建任务特定问答数据库方面的差异。较小的模型倾向于生成更简单的问题，它们自身及其他模型能够有效检索和使用这些问题；而较大的模型则产生更复杂但效果较差的问题，这表明不同规模模型之间存在互补优势。总之，我们的研究结果强调，将评估从静态事实回忆转向动态知识应用，能够为模型能力提供更具信息量的视角。