While large language models (LLMs) have become the de facto framework for literature-related tasks, they still struggle to function as domain-specific literature agents due to their inability to connect pieces of knowledge and reason across domain-specific contexts, terminologies, and nomenclatures. This challenge underscores the need for a tool that facilitates such domain-specific adaptation and enables rigorous benchmarking across literature tasks. To that end, we introduce LitBench, a benchmarking tool designed to enable the development and evaluation of domain-specific LLMs tailored to literature-related tasks. At its core, LitBench uses a data curation process that generates domain-specific literature sub-graphs and constructs training and evaluation datasets based on the textual attributes of the resulting nodes and edges. The tool is designed for flexibility, supporting the curation of literature graphs across any domain chosen by the user, whether high-level fields or specialized interdisciplinary areas. In addition to dataset curation, LitBench defines a comprehensive suite of literature tasks, ranging from node and edge level analyses to advanced applications such as related work generation. These tasks enable LLMs to internalize domain-specific knowledge and relationships embedded in the curated graph during training, while also supporting rigorous evaluation of model performance. Our results show that small domain-specific LLMs trained and evaluated on LitBench datasets achieve competitive performance compared to state-of-the-art models like GPT-4o and DeepSeek-R1. To enhance accessibility and ease of use, we open-source the tool along with an AI agent tool that streamlines data curation, model training, and evaluation.

翻译：尽管大语言模型已成为文献相关任务的事实标准框架，但由于其无法跨领域特定上下文、术语和命名体系进行知识关联与推理，在作为专业领域文献代理时仍存在局限。这一挑战凸显了开发可促进领域特定适配并实现文献任务严格基准测试的工具的必要性。为此，我们提出LitBench——专为文学任务设计的基准测试工具，支持面向文献任务的领域特定大语言模型开发与评估。该工具核心采用数据策展流程，通过生成领域特定文献子图，并基于所得节点与边的文本属性构建训练与评估数据集。LitBench设计具有灵活性，支持用户自主选择任意领域（从宏观学科到交叉细分领域）进行文献图谱策展。除数据集构建外，该工具还定义了涵盖节点级与边级分析、相关文献生成等高级应用的综合性文献任务体系。这些任务使大语言模型在训练过程中能够内化图谱中嵌入的领域知识与关联，同时支持严格的模型性能评估。实验表明，基于LitBench数据集训练与评估的小型领域特定大语言模型，在性能上可媲美GPT-4o和DeepSeek-R1等前沿模型。为提升可访问性与易用性，我们开源该工具及配套的人工智能代理工具，可简化数据策展、模型训练与评估流程。