Scientific literature understanding tasks have gained significant attention due to their potential to accelerate scientific discovery. Pre-trained language models (LMs) have shown effectiveness in these tasks, especially when tuned via contrastive learning. However, jointly utilizing pre-training data across multiple heterogeneous tasks (e.g., extreme multi-label paper classification, citation prediction, and literature search) remains largely unexplored. To bridge this gap, we propose a multi-task contrastive learning framework, SciMult, with a focus on facilitating common knowledge sharing across different scientific literature understanding tasks while preventing task-specific skills from interfering with each other. To be specific, we explore two techniques -- task-aware specialization and instruction tuning. The former adopts a Mixture-of-Experts Transformer architecture with task-aware sub-layers; the latter prepends task-specific instructions to the input text so as to produce task-aware outputs. Extensive experiments on a comprehensive collection of benchmark datasets verify the effectiveness of our task-aware specialization strategy, where we outperform state-of-the-art scientific pre-trained LMs. Code, datasets, and pre-trained models can be found at https://scimult.github.io/.

翻译：科学文献理解任务因能加速科学发现而备受关注。预训练语言模型（LMs）在这些任务中展现出有效性，尤其在通过对比学习进行调优后。然而，如何联合利用跨多个异构任务（例如极端多标签论文分类、引文预测及文献检索）的预训练数据，在很大程度上仍是未探索领域。为填补这一空白，我们提出多任务对比学习框架SciMult，旨在促进不同科学文献理解任务间的通用知识共享，同时防止任务特定技能相互干扰。具体而言，我们探索了两种技术——任务感知专业化与指令微调。前者采用带任务感知子层的混合专家Transformer架构；后者则在输入文本前添加任务特定指令，以生成任务感知输出。在综合基准数据集上的大量实验验证了我们的任务感知专业化策略的有效性，该策略超越了现有最优的科学预训练语言模型。代码、数据集及预训练模型可在https://scimult.github.io/获取。