Recently, there has been a significant upsurge of interest in leveraging large language models (LLMs) to assist scientific discovery. However, most LLMs only focus on general science, while they lack domain-specific knowledge, such as chemical molecules and amino acid sequences. To bridge these gaps, we introduce SciDFM, a mixture-of-experts LLM, which is trained from scratch and is able to conduct college-level scientific reasoning and understand molecules and amino acid sequences. We collect a large-scale training corpus containing numerous scientific papers and books from different disciplines as well as data from domain-specific databases. We further fine-tune the pre-trained model on lots of instruction data to improve performances on downstream benchmarks. From experiment results, we show that SciDFM achieves strong performance on general scientific benchmarks such as SciEval and SciQ, and it reaches a SOTA performance on domain-specific benchmarks among models of similar size. We further analyze the expert layers and show that the results of expert selection vary with data from different disciplines. To benefit the broader research community, we open-source SciDFM at https://huggingface.co/OpenDFM/SciDFM-MoE-A5.6B-v1.0.

翻译：近年来，利用大语言模型辅助科学发现的研究兴趣显著高涨。然而，大多数大语言模型仅关注通用科学领域，缺乏对化学分子、氨基酸序列等特定领域知识的深入理解。为弥补这些不足，我们提出了SciDFM，一种从头开始训练的混合专家大语言模型，该模型能够进行大学水平的科学推理，并理解分子与氨基酸序列。我们收集了一个包含来自不同学科的大量科学论文、书籍以及特定领域数据库数据的大规模训练语料库。我们进一步在大量指令数据上对预训练模型进行微调，以提升其在下游基准测试上的性能。实验结果表明，SciDFM在SciEval和SciQ等通用科学基准测试中表现出色，并在同等规模模型中，于特定领域基准测试上达到了最先进的性能。我们进一步分析了专家层，结果表明专家选择的结果随不同学科的数据而变化。为使更广泛的研究社区受益，我们在 https://huggingface.co/OpenDFM/SciDFM-MoE-A5.6B-v1.0 开源了SciDFM。