Scientific literature understanding tasks have gained significant attention due to their potential to accelerate scientific discovery. Pre-trained language models (LMs) have shown effectiveness in these tasks, especially when tuned via contrastive learning. However, jointly utilizing pre-training data across multiple heterogeneous tasks (e.g., extreme classification, citation prediction, and literature search) remains largely unexplored. To bridge this gap, we propose a multi-task contrastive learning framework, SciMult, with a focus on facilitating common knowledge sharing across different scientific literature understanding tasks while preventing task-specific skills from interfering with each other. To be specific, we explore two techniques -- task-aware specialization and instruction tuning. The former adopts a Mixture-of-Experts Transformer architecture with task-aware sub-layers; the latter prepends task-specific instructions to the input text so as to produce task-aware outputs. Extensive experiments on a comprehensive collection of benchmark datasets verify the effectiveness of our task-aware specialization strategy in various tasks, where we outperform state-of-the-art scientific LMs.

翻译：科学文献理解任务因有望加速科学发现而受到广泛关注。预训练语言模型在这些任务中表现出有效性，尤其是在通过对比学习进行调优后。然而，如何联合利用来自多个异构任务（例如极端分类、引文预测和文献检索）的预训练数据仍鲜有探索。为弥合这一差距，我们提出了多任务对比学习框架SciMult，其核心在于促进不同科学文献理解任务间的通用知识共享，同时防止任务特定技能相互干扰。具体而言，我们探索了两种技术——任务感知专业化与指令微调。前者采用基于混合专家Transformer架构的任务感知子层；后者则通过在输入文本前置任务特定指令以生成任务感知输出。在一系列基准数据集上的广泛实验验证了任务感知专业化策略在多种任务中的有效性，其表现超越了当前最先进的科学语言模型。