By focusing the pre-training process on domain-specific corpora, some domain-specific pre-trained language models (PLMs) have achieved state-of-the-art results. However, it is under-investigated to design a unified paradigm to inject domain knowledge in the PLM fine-tuning stage. We propose KnowledgeDA, a unified domain language model development service to enhance the task-specific training procedure with domain knowledge graphs. Given domain-specific task texts input, KnowledgeDA can automatically generate a domain-specific language model following three steps: (i) localize domain knowledge entities in texts via an embedding-similarity approach; (ii) generate augmented samples by retrieving replaceable domain entity pairs from two views of both knowledge graph and training data; (iii) select high-quality augmented samples for fine-tuning via confidence-based assessment. We implement a prototype of KnowledgeDA to learn language models for two domains, healthcare and software development. Experiments on domain-specific text classification and QA tasks verify the effectiveness and generalizability of KnowledgeDA.

翻译：通过将预训练过程聚焦于领域特定语料，部分领域预训练语言模型（PLMs）已取得最优结果。然而，目前尚缺乏针对在PLM微调阶段注入领域知识的统一范式研究。我们提出KnowledgeDA——一种统一的领域语言模型开发服务，旨在通过领域知识图谱增强任务特定训练过程。给定领域任务文本输入，KnowledgeDA可自动生成领域语言模型，其执行三步流程：（i）基于嵌入相似度方法定位文本中的领域知识实体；（ii）从知识图谱与训练数据双视角检索可替换领域实体对，生成增强样本；（iii）通过置信度评估筛选高质量增强样本用于模型微调。我们实现了KnowledgeDA原型系统，用于学习医疗健康与软件开发两个领域的语言模型。在领域文本分类与问答任务上的实验验证了KnowledgeDA的有效性与泛化能力。