Large language models (LLMs) trained on general domain corpora showed remarkable results on natural language processing (NLP) tasks. However, previous research demonstrated LLMs trained using domain-focused corpora perform better on specialized tasks. Inspired by this pivotal insight, we developed INDUS, a comprehensive suite of LLMs tailored for the Earth science, biology, physics, heliophysics, planetary sciences and astrophysics domains and trained using curated scientific corpora drawn from diverse data sources. The suite of models include: (1) an encoder model trained using domain-specific vocabulary and corpora to address natural language understanding tasks, (2) a contrastive-learning-based general text embedding model trained using a diverse set of datasets drawn from multiple sources to address information retrieval tasks and (3) smaller versions of these models created using knowledge distillation techniques to address applications which have latency or resource constraints. We also created three new scientific benchmark datasets namely, CLIMATE-CHANGE-NER (entity-recognition), NASA-QA (extractive QA) and NASA-IR (IR) to accelerate research in these multi-disciplinary fields. Finally, we show that our models outperform both general-purpose encoders (RoBERTa) and existing domain-specific encoders (SciBERT) on these new tasks as well as existing benchmark tasks in the domains of interest.

翻译：在通用领域语料上训练的大型语言模型在自然语言处理任务中展现出显著成果。然而，已有研究表明，使用领域专注语料训练的语言模型在专业任务上表现更佳。受这一关键洞察启发，我们开发了INDUS——一套专为地球科学、生物学、物理学、太阳物理学、行星科学及天体物理学领域量身定制的综合语言模型套件，其训练数据来自经过精心筛选的多源科学语料。该模型套件包括：(1) 采用领域专用词汇与语料训练的编码器模型，用于解决自然语言理解任务；(2) 基于多源数据集训练的对比学习通用文本嵌入模型，旨在处理信息检索任务；(3) 通过知识蒸馏技术生成的轻量化版本模型，适用于存在延迟或资源约束的应用场景。此外，我们创建了三个新型科学基准数据集：CLIMATE-CHANGE-NER（实体识别）、NASA-QA（抽取式问答）与NASA-IR（信息检索），以加速跨学科领域研究。实验证明，我们的模型在新任务及目标领域现有基准任务上，均显著优于通用型编码器（RoBERTa）和现有领域专用编码器（SciBERT）。