In recent years, large language models (LLMs), such as GPTs, have attained great impact worldwide. However, how to adapt these LLMs to better suit the vertical domain-specific tasks by utilizing external knowledge remains not completely solved. Indeed, there have emerged a few works on this line where most of them rely on an alignment heuristic that is built to inject the corresponding knowledge tuple into the associated text sample. However, despite the promise, we identify a pivotal problem in this work ubiquitously. Simply put, we find that injecting unaligned (i.e., random) knowledge tuple into the LLMs achieves comparable (and sometimes better) results than the aligned knowledge being injected. We therefore take a thorough investigation of this frustrating finding on a variety of related prior work and further provide a chain of potential interpretations for the phenomenon. Based on all that, we offer a simple remediated technique. Briefly, the core of this technique is rooted in an ideological emphasis on the pruning and purification of the external knowledge base to be injected into LLMs. At last, we show that by integrating this technique into most (if not all) knowledge injection frameworks and recent LLMs, it manages to overcome the aforementioned sanity problem and further pushes the boundary of the performance of the domain-adaptive LLMs.

翻译：近年来，诸如GPTs等大型语言模型（LLMs）在全球产生了巨大影响。然而，如何利用外部知识使这些LLMs更好地适应垂直领域特定任务的问题尚未完全解决。事实上，这一方向已有少量研究工作涌现，其中大多数依赖于构建的对齐启发式方法，将对应的知识元组注入关联的文本样本中。然而，尽管这些方法前景可期，我们识别出一个普遍存在的关键问题。简言之，我们发现将未对齐（即随机）知识元组注入LLMs时，其效果与注入对齐知识相当（有时甚至更优）。因此，我们对这一令人沮丧的发现进行了深入探究，涉及多种相关前期工作，并进一步为这一现象提供了一系列潜在解释。基于此，我们提出了一种简单补救技术。简而言之，该技术的核心在于一种理念强调：对注入LLMs的外部知识库进行修剪与净化。最后，我们证明，将该技术整合到大多数（若非全部）知识注入框架及最新LLMs中，能够克服上述合理性问题，并进一步推升领域自适应LLMs的性能边界。