Many useful tasks on scientific documents, such as topic classification and citation prediction, involve corpora that span multiple scientific domains. Typically, such tasks are accomplished by representing the text with a vector embedding obtained from a Transformer's single CLS token. In this paper, we argue that using multiple CLS tokens could make a Transformer better specialize to multiple scientific domains. We present Multi2SPE: it encourages each of multiple CLS tokens to learn diverse ways of aggregating token embeddings, then sums them up together to create a single vector representation. We also propose our new multi-domain benchmark, Multi-SciDocs, to test scientific paper vector encoders under multi-domain settings. We show that Multi2SPE reduces error by up to 25 percent in multi-domain citation prediction, while requiring only a negligible amount of computation in addition to one BERT forward pass.

翻译：许多针对科学文档的有用任务，例如主题分类和引文预测，涉及跨越多个科学领域的语料库。通常，这些任务通过从Transformer的单个CLS令牌获得文本的向量嵌入来完成。本文认为，使用多个CLS令牌可以使Transformer更好地适应多个科学领域。我们提出了Multi2SPE：该方法鼓励多个CLS令牌中的每一个学习聚合令牌嵌入的不同方式，然后将它们汇总起来以创建单个向量表示。我们还提出了新的多领域基准——Multi-SciDocs，用于在多领域设置下测试科学论文向量编码器。我们证明，Multi2SPE在多领域引文预测中最多可将错误减少25%，同时仅需在单次BERT前向传播的基础上增加微不足道的计算量。