Large language models (LLMs) have shown incredible performance in completing various real-world tasks. The current knowledge learning paradigm of LLMs is mainly based on learning from examples, in which LLMs learn the internal rule implicitly from a certain number of supervised examples. However, this learning paradigm may not well learn those complicated rules, especially when the training examples are limited. We are inspired that humans can learn the new tasks or knowledge in another way by learning from rules. That is, humans can learn new tasks or grasps new knowledge quickly and generalize well given only a detailed rule and a few optional examples. Therefore, in this paper, we aim to explore the feasibility of this new learning paradigm, which targets on encoding rule-based knowledge into LLMs. We further propose rule distillation, which first uses the strong in-context abilities of LLMs to extract the knowledge from the textual rules, and then explicitly encode the knowledge into the parameters of LLMs by learning from the above in-context signals produced inside the model. Our experiments show that making LLMs learn from rules by our method is much more efficient than example-based learning in both the sample size and generalization ability. Warning: This paper may contain examples with offensive content.

翻译：大型语言模型在完成各种现实世界任务中展现出惊人性能。当前LLMs的知识学习范式主要基于从示例中学习，即模型通过一定数量的监督示例隐式学习内在规则。然而，当训练示例有限时，这种学习范式可能无法很好地掌握那些复杂规则。受人类可通过从规则中学习这一新途径掌握新任务或知识的启发——即仅凭详细规则和少量可选示例，人类就能快速掌握新任务或知识并实现良好泛化——本文旨在探索这种以规则知识编码为核心的新学习范式的可行性。我们进一步提出规则蒸馏方法：首先利用LLMs强大的上下文学习能力从文本规则中提取知识，然后通过学习模型内部产生的上述上下文信号，将知识显式编码到LLMs参数中。实验表明，与基于示例的学习相比，我们的方法在样本规模和泛化能力两方面均使LLMs从规则中学习具有更高效率。警告：本文可能包含含有冒犯性内容的示例。