The estimation of the generalization error of classifiers often relies on a validation set. Such a set is hardly available in few-shot learning scenarios, a highly disregarded shortcoming in the field. In these scenarios, it is common to rely on features extracted from pre-trained neural networks combined with distance-based classifiers such as nearest class mean. In this work, we introduce a Gaussian model of the feature distribution. By estimating the parameters of this model, we are able to predict the generalization error on new classification tasks with few samples. We observe that accurate distance estimates between class-conditional densities are the key to accurate estimates of the generalization performance. Therefore, we propose an unbiased estimator for these distances and integrate it in our numerical analysis. We empirically show that our approach outperforms alternatives such as the leave-one-out cross-validation strategy.

翻译：分类器的泛化误差估计通常依赖于验证集。然而，在小样本学习场景中，验证集几乎不可用，这是该领域一个被严重忽视的缺陷。在此类场景中，通常依赖从预训练神经网络中提取的特征，并结合基于距离的分类器（如最近类均值法）。本研究引入了一个特征分布的高斯模型。通过估计该模型的参数，我们能够预测新分类任务（样本量极少）的泛化误差。我们观察到，类别条件密度之间距离的精确估计是实现泛化性能精确估计的关键。因此，我们提出了这些距离的无偏估计量，并将其整合到数值分析中。实验表明，我们的方法优于留一交叉验证策略等替代方案。