Large language models (LLMs) have shown incredible performance in completing various real-world tasks. The current knowledge learning paradigm of LLMs is mainly based on learning from examples, in which LLMs learn the internal rule implicitly from a certain number of supervised examples. However, the learning paradigm may not well learn those complicated rules, especially when the training examples are limited. We are inspired that humans can learn the new tasks or knowledge in another way by learning from rules. That is, humans can grasp the new tasks or knowledge quickly and generalize well given only a detailed rule and a few optional examples. Therefore, in this paper, we aim to explore the feasibility of this new learning paradigm, which encodes the rule-based knowledge into LLMs. We propose rule distillation, which first uses the strong in-context abilities of LLMs to extract the knowledge from the textual rules and then explicitly encode the knowledge into LLMs' parameters by learning from the above in-context signals produced inside the model. Our experiments show that making LLMs learn from rules by our method is much more efficient than example-based learning in both the sample size and generalization ability.

翻译：大语言模型（LLMs）在完成各类实际任务中展现出卓越性能。当前LLMs的知识学习范式主要基于从示例中学习，即模型通过一定数量的监督示例隐式习得内部规则。然而，这种学习范式可能难以充分习得复杂规则，尤其是在训练示例有限的情况下。我们受到人类能够通过另一种方式——从规则中学习新任务或知识的启发。即人类只需详细的规则和少量可选示例，就能快速掌握新任务或知识并实现良好的泛化。因此，本文旨在探索这一新学习范式的可行性，将基于规则的知识编码到LLMs中。我们提出了规则蒸馏方法，首先利用LLMs强大的上下文学习能力从文本规则中提取知识，然后通过学习模型内部产生的上述上下文信号，将知识显式编码到LLMs的参数中。实验表明，我们的方法使LLMs从规则中学习，在样本规模和泛化能力两方面均显著优于基于示例的学习。