Pre-trained Language Models (PLMs), as parametric-based eager learners, have become the de-facto choice for current paradigms of Natural Language Processing (NLP). In contrast, k-Nearest-Neighbor (kNN) classifiers, as the lazy learning paradigm, tend to mitigate over-fitting and isolated noise. In this paper, we revisit kNN classifiers for augmenting the PLMs-based classifiers. From the methodological level, we propose to adopt kNN with textual representations of PLMs in two steps: (1) Utilize kNN as prior knowledge to calibrate the training process. (2) Linearly interpolate the probability distribution predicted by kNN with that of the PLMs' classifier. At the heart of our approach is the implementation of kNN-calibrated training, which treats predicted results as indicators for easy versus hard examples during the training process. From the perspective of the diversity of application scenarios, we conduct extensive experiments on fine-tuning, prompt-tuning paradigms and zero-shot, few-shot and fully-supervised settings, respectively, across eight diverse end-tasks. We hope our exploration will encourage the community to revisit the power of classical methods for efficient NLP. Code and datasets are available in https://github.com/zjunlp/Revisit-KNN.

翻译：预训练语言模型（PLMs）作为基于参数的急切学习器，已成为当前自然语言处理（NLP）范式的默认选择。相比之下，k-最近邻（kNN）分类器作为一种惰性学习范式，倾向于缓解过拟合和孤立噪声。本文重新审视kNN分类器，以增强基于PLMs的分类器。在方法论层面，我们提出通过两个步骤采用基于PLMs文本表示的kNN：（1）利用kNN作为先验知识来校准训练过程；（2）将kNN预测的概率分布与PLMs分类器的概率分布进行线性插值。该方法的核心是实现kNN校准训练，即将预测结果作为训练过程中简单样本与困难样本的指示器。从应用场景多样性的角度出发，我们在八个不同下游任务上分别对微调、提示调优范式和零样本、少样本及全监督设置进行了广泛实验。希望我们的探索能鼓励社区重新审视经典方法在高效NLP中的潜力。代码与数据集见https://github.com/zjunlp/Revisit-KNN。