Despite their limited interpretability, weights and biases are still the most popular encoding of the functions learned by ReLU Neural Networks (ReLU NNs). That is why we introduce SkelEx, an algorithm to extract a skeleton of the membership functions learned by ReLU NNs, making those functions easier to interpret and analyze. To the best of our knowledge, this is the first work that considers linear regions from the perspective of critical points. As a natural follow-up, we also introduce BoundEx, which is the first analytical method known to us to extract the decision boundary from the realization of a ReLU NN. Both of those methods introduce very natural visualization tool for ReLU NNs trained on low-dimensional data.

翻译：尽管可解释性有限，权重和偏置仍然是ReLU神经网络（ReLU NN）所学函数的最主流编码方式。为此，我们提出SkelEx算法，通过提取ReLU神经网络所学习隶属函数的骨架结构，使这些函数更易于解释和分析。据我们所知，这是首个从临界点视角研究线性区域的工作。作为自然延伸，我们还提出BoundEx方法——这是目前已知首个从ReLU神经网络实现中提取决策边界的分析方法。这两种方法为低维数据训练的ReLU神经网络提供了非常直观的可视化工具。