Nonparametric learning is a fundamental concept in machine learning that aims to capture complex patterns and relationships in data without making strong assumptions about the underlying data distribution. Owing to simplicity and familiarity, one of the most well-known algorithms under this paradigm is the $k$-nearest neighbors ($k$-NN) algorithm. Driven by the usage of machine learning in safety-critical applications, in this work, we shed new light on the traditional nearest neighbors algorithm from the perspective of information theory and propose a robust and interpretable framework for tasks such as classification, regression, and anomaly detection using a single model. Instead of using a traditional distance measure which needs to be scaled and contextualized, we use a novel formulation of \textit{surprisal} (amount of information required to explain the difference between the observed and expected result). Finally, we demonstrate this architecture's capability to perform at-par or above the state-of-the-art on classification, regression, and anomaly detection tasks using a single model with enhanced interpretability by providing novel concepts for characterizing data and predictions.

翻译：非参数学习是机器学习中的一个基本概念，旨在捕捉数据中的复杂模式和关系，而无需对底层数据分布做出强烈假设。由于简单且易于理解，该范式下最著名的算法之一是 $k$ 近邻（$k$-NN）算法。受机器学习在安全关键型应用中使用的推动，本研究从信息论的角度重新审视传统的近邻算法，并提出了一个鲁棒且可解释的框架，用于分类、回归和异常检测等任务，且仅使用单一模型。我们并未采用需要缩放和上下文化的传统距离度量，而是使用了一种新颖的“惊异度”公式（即解释观测结果与预期结果之间差异所需的信息量）。最后，我们通过提供描述数据和预测的新概念，展示了该架构在分类、回归和异常检测任务中仅使用单一模型即可达到或超越现有技术水平的性能，并增强了可解释性。