Large language models (LLMs) have significantly advanced the field of natural language processing, with GPT models at the forefront. While their remarkable performance spans a range of tasks, adapting LLMs for real-world business scenarios still poses challenges warranting further investigation. This paper presents an empirical analysis aimed at bridging the gap in adapting LLMs to practical use cases. To do that, we select the question answering (QA) task of insurance as a case study due to its challenge of reasoning. Based on the task we design a new model relied on LLMs which are empowered by additional knowledge extracted from insurance policy rulebooks and DBpedia. The additional knowledge helps LLMs to understand new concepts of insurance for domain adaptation. Preliminary results on two QA datasets show that knowledge enhancement significantly improves the reasoning ability of GPT-3.5 (55.80% and 57.83% in terms of accuracy). The analysis also indicates that existing public knowledge bases, e.g., DBPedia is beneficial for knowledge enhancement. Our findings reveal that the inherent complexity of business scenarios often necessitates the incorporation of domain-specific knowledge and external resources for effective problem-solving.

翻译：大型语言模型（LLM）显著推动了自然语言处理领域的发展，其中GPT模型处于前沿地位。尽管这些模型在诸多任务中展现出卓越性能，但将其适配到实际商业场景仍面临挑战，亟需进一步研究。本文提出一项实证分析，旨在弥合LLM适配实际应用场景的差距。为此，我们选取保险领域的问答任务作为案例研究，因其推理难度具有代表性。基于该任务，我们设计了一种新型模型——该模型以LLM为核心，并通过从保险政策规则手册和DBpedia中提取的额外知识进行增强。额外知识帮助LLM理解保险领域的新概念，从而实现领域适配。两个问答数据集的初步结果显示，知识增强显著提升了GPT-3.5的推理能力（准确率分别达到55.80%和57.83%）。分析还表明，现有公共知识库（如DBpedia）对知识增强具有积极作用。我们的研究揭示：商业场景的内在复杂性往往需要融合领域特定知识与外部资源，方能实现有效问题求解。