The rapid advancements in Large Language Models (LLMs) have enabled their adoption in real-world industrial scenarios for various natural language processing tasks. However, the high inference cost of large-scale LLMs makes their deployment impractical, necessitating the use of smaller models. Despite their efficiency, smaller LLMs lack robust zero-shot instruction-following capabilities across diverse domains, limiting their adaptability to dynamic user requirements. Traditional fine-tuning approaches exacerbate this issue by inducing catastrophic forgetting, reducing the model's generalization ability for unseen tasks. In this paper, we propose Domain Adaptive Continual Instruction Pre-Training via Reading Comprehension (DACIP-RC), a continual pre-training technique that enhances smaller LLMs' domain adaptability for business conversational tasks. Unlike conventional pre-training approaches that rely on next-token prediction, DACIP-RC generates diverse task instructions and responses via reading comprehension on conversation transcripts, enabling better instruction generalization. Our empirical evaluations demonstrate that DACIP-RC significantly improves zero-shot generalization across a wide range of business conversational tasks, including meeting summarization, action item generation, and call purpose identification. To the best of our knowledge, this is the first work to apply instruction pre-training on business conversational data, providing insights into how industries can leverage proprietary datasets for domain adaptation.

翻译：大型语言模型（LLM）的快速发展使其能够应用于现实工业场景中的各种自然语言处理任务。然而，大规模LLM的高推理成本使其部署不切实际，因此需要使用更小的模型。尽管小型LLM效率较高，但其缺乏跨不同领域的强大零样本指令遵循能力，限制了其对动态用户需求的适应性。传统的微调方法会引发灾难性遗忘，进一步加剧了这一问题，从而降低了模型对未见任务的泛化能力。本文提出了一种基于阅读理解的领域自适应持续指令预训练方法（DACIP-RC），这是一种持续预训练技术，旨在增强小型LLM在企业对话任务中的领域适应性。与依赖下一词预测的传统预训练方法不同，DACIP-RC通过对对话文本进行阅读理解来生成多样化的任务指令和响应，从而实现更好的指令泛化。我们的实证评估表明，DACIP-RC显著提高了在广泛的企业对话任务（包括会议摘要、行动项生成和通话目的识别）中的零样本泛化能力。据我们所知，这是首次将指令预训练应用于企业对话数据的研究，为行业如何利用专有数据集进行领域适应提供了见解。