Large language models (LLMs) trained on general domain corpora showed remarkable results on natural language processing (NLP) tasks. However, previous research demonstrated LLMs trained using domain-focused corpora perform better on specialized tasks. Inspired by this insight, we developed INDUS, a comprehensive suite of LLMs tailored for the closely-related domains of Earth science, biology, physics, heliophysics, planetary sciences and astrophysics, and trained using curated scientific corpora drawn from diverse data sources. The suite of models include: (1) an encoder model trained using domain-specific vocabulary and corpora to address NLP tasks, (2) a contrastive-learning based text embedding model trained using a diverse set of datasets to address information retrieval tasks and (3) smaller versions of these models created using knowledge distillation for applications which have latency or resource constraints. We also created three new scientific benchmark datasets, CLIMATE-CHANGE NER (entity-recognition), NASA-QA (extractive QA) and NASA-IR (IR) to accelerate research in these multi-disciplinary fields. We show that our models outperform both general-purpose (RoBERTa) and domain-specific (SCIBERT) encoders on these new tasks as well as existing tasks in the domains of interest. Furthermore, we demonstrate the use of these models in two industrial settings -- as a retrieval model for large-scale vector search applications and in automatic content tagging systems.

翻译：基于通用领域语料库训练的大语言模型（LLM）在自然语言处理（NLP）任务中展现出卓越性能。然而，先前研究表明，使用领域聚焦语料训练的LLM在专业任务上表现更优。受此启发，我们开发了INDUS——一套专为地球科学、生物学、物理学、太阳物理学、行星科学与天体物理学等紧密关联领域定制的综合LLM套件，其训练数据源于多源精选科学语料库。该套件包含：（1）采用领域专用词汇与语料训练的编码器模型，用于处理NLP任务；（2）基于对比学习训练的文本嵌入模型，通过多样化数据集训练以应对信息检索任务；（3）通过知识蒸馏构建的轻量化模型版本，适用于存在延迟或资源限制的应用场景。我们还创建了三个新的科学基准数据集：CLIMATE-CHANGE NER（命名实体识别）、NASA-QA（抽取式问答）和NASA-IR（信息检索），以加速这些跨学科领域的研究。实验表明，我们的模型在新增任务及目标领域现有任务上均优于通用模型（RoBERTa）与领域专用模型（SCIBERT）。此外，我们展示了这些模型在两种工业场景中的应用——作为大规模向量搜索的检索模型，以及用于自动内容标注系统。