We investigate an approach for extracting knowledge from trained neural networks based on Angluin's exact learning model with membership and equivalence queries to an oracle. In this approach, the oracle is a trained neural network. We consider Angluin's classical algorithm for learning Horn theories and study the necessary changes to make it applicable to learn from neural networks. In particular, we have to consider that trained neural networks may not behave as Horn oracles, meaning that their underlying target theory may not be Horn. We propose a new algorithm that aims at extracting the ``tightest Horn approximation'' of the target theory and that is guaranteed to terminate in exponential time (in the worst case) and in polynomial time if the target has polynomially many non-Horn examples. To showcase the applicability of the approach, we perform experiments on pre-trained language models and extract rules that expose occupation-based gender biases.

翻译：我们研究了一种基于Angluin精确学习模型的方法，通过向神谕进行成员查询和等价查询，从训练的神经网络中提取知识。在该方法中，神谕即为训练后的神经网络。我们考虑使用Angluin经典算法学习Horn理论，并研究为使其适用于神经网络学习所需的关键改进。特别地，我们必须认识到训练后的神经网络可能不会表现出Horn神谕行为，即其底层目标理论可能并非Horn理论。为此我们提出一种新算法，旨在提取目标理论的"最紧凑Horn近似"，该算法最坏情况下保证在指数时间内终止，若目标包含多项式数量级非Horn样例时则可在多项式时间内终止。为展示该方法的实用性，我们对预训练语言模型进行实验，提取出揭示职业性别偏见的规则。