Large Language Models (LLMs) pre-trained on massive corpora have exhibited remarkable performance on various NLP tasks. However, applying these models to specific domains still poses significant challenges, such as lack of domain knowledge, limited capacity to leverage domain knowledge and inadequate adaptation to domain-specific data formats. Considering the exorbitant cost of training LLMs from scratch and the scarcity of annotated data within particular domains, in this work, we focus on domain-specific continual pre-training of LLMs using E-commerce domain as an exemplar. Specifically, we explore the impact of continual pre-training on LLMs employing unlabeled general and E-commercial corpora. Furthermore, we design a mixing strategy among different data sources to better leverage E-commercial semi-structured data. We construct multiple tasks to assess LLMs' few-shot In-context Learning ability and their zero-shot performance after instruction tuning in E-commerce domain. Experimental results demonstrate the effectiveness of continual pre-training of E-commerce LLMs and the efficacy of our devised data mixing strategy.

翻译：在大规模语料库上预训练的大型语言模型（LLMs）已在各类自然语言处理任务中展现出卓越性能。然而，将这些模型应用于特定领域仍面临重大挑战，例如领域知识匮乏、利用领域知识的能力有限以及对特定数据格式的适应性不足。考虑到从头训练LLMs的高昂成本及特定领域标注数据的稀缺性，本文以电商领域为例，聚焦于LLMs的领域特定持续预训练。具体而言，我们探究了使用未标注通用与电商语料库进行持续预训练对LLMs的影响，并设计了一种多源数据混合策略以更优地利用电商半结构化数据。我们构建了多项任务以评估LLMs在电商领域的少样本上下文学习能力及其经指令微调后的零样本性能。实验结果表明，对电商LLMs进行持续预训练的有效性及所提出的数据混合策略的优越性。