Large language models (LLMs) have significantly advanced the field of natural language processing, with GPT models at the forefront. While their remarkable performance spans a range of tasks, adapting LLMs for real-world business scenarios still poses challenges warranting further investigation. This paper presents an empirical analysis aimed at bridging the gap in adapting LLMs to practical use cases. To do that, we select the question answering (QA) task of insurance as a case study due to its challenge of reasoning. Based on the task we design a new model relied on LLMs which are empowered by domain-specific knowledge extracted from insurance policy rulebooks. The domain-specific knowledge helps LLMs to understand new concepts of insurance for domain adaptation. Preliminary results on real QA pairs show that knowledge enhancement from policy rulebooks significantly improves the reasoning ability of GPT-3.5 of 50.4% in terms of accuracy. The analysis also indicates that existing public knowledge bases, e.g., DBPedia is beneficial for knowledge enhancement. Our findings reveal that the inherent complexity of business scenarios often necessitates the incorporation of domain-specific knowledge and external resources for effective problem-solving.

翻译：大型语言模型（LLMs）显著推动了自然语言处理领域的发展，其中GPT模型处于领先地位。尽管这些模型在一系列任务中展现出卓越性能，但将其适配至实际业务场景仍面临挑战，有待进一步研究。本文通过实证分析，旨在弥合LLMs适配实际应用场景的差距。为此，我们选取保险领域的问答（QA）任务作为案例，因其涉及推理难题。基于该任务，我们设计了一种依托LLMs的新模型，该模型通过从保险政策规则手册中提取的领域特定知识增强能力。领域特定知识帮助LLMs理解保险领域的新概念以实现领域适配。基于真实问答对的初步结果表明，政策规则手册的知识增强使GPT-3.5的推理准确率显著提升50.4%。分析还显示，现有公共知识库（如DBPedia）对知识增强具有增益效果。研究发现揭示，业务场景的内在复杂性往往需要融合领域特定知识与外部资源才能有效解决问题。