Extensive studies have been devoted to privatizing general-domain Large Language Models (LLMs) as Domain-Specific LLMs via feeding specific-domain data. However, these privatization efforts often ignored a critical aspect: Dual Logic Ability, which is a core reasoning ability for LLMs. The dual logic ability of LLMs ensures that they can maintain a consistent stance when confronted with both positive and negative statements about the same fact. Our study focuses on how the dual logic ability of LLMs is affected during the privatization process in the medical domain. We conduct several experiments to analyze the dual logic ability of LLMs by examining the consistency of the stance in responses to paired questions about the same fact. In our experiments, interestingly, we observed a significant decrease in the dual logic ability of existing LLMs after privatization. Besides, our results indicate that incorporating general domain dual logic data into LLMs not only enhances LLMs' dual logic ability but also further improves their accuracy. These findings underscore the importance of prioritizing LLMs' dual logic ability during the privatization process. Our study establishes a benchmark for future research aimed at exploring LLMs' dual logic ability during the privatization process and offers valuable guidance for privatization efforts in real-world applications.
翻译:已有大量研究致力于将通用大语言模型(LLMs)通过喂养特定领域数据,私有化为领域专用LLMs。然而,这些私有化工作往往忽视了一个关键方面:双重逻辑能力,这是LLMs的核心推理能力。LLMs的双重逻辑能力确保其在面对同一事实的正反陈述时能保持一致立场。本研究聚焦于医疗领域私有化过程中LLMs双重逻辑能力如何受到影响。我们通过分析LLMs对同一事实配对问题回复中的立场一致性,开展多项实验以评估其双重逻辑能力。有趣的是,实验发现现有LLMs在私有化后双重逻辑能力显著下降。此外,结果表明将通用领域双重逻辑数据融入LLMs不仅能增强其双重逻辑能力,还能进一步提高其准确性。这些发现强调了私有化过程中优先考虑LLMs双重逻辑能力的重要性。我们的研究为未来探索私有化过程中LLMs双重逻辑能力的研究建立了基准,并为现实应用中的私有化工作提供了有价值的指导。