Numerous benchmarks for Few-Shot Learning have been proposed in the last decade. However all of these benchmarks focus on performance averaged over many tasks, and the question of how to reliably evaluate and tune models trained for individual tasks in this regime has not been addressed. This paper presents the first investigation into task-level evaluation -- a fundamental step when deploying a model. We measure the accuracy of performance estimators in the few-shot setting, consider strategies for model selection, and examine the reasons for the failure of evaluators usually thought of as being robust. We conclude that cross-validation with a low number of folds is the best choice for directly estimating the performance of a model, whereas using bootstrapping or cross validation with a large number of folds is better for model selection purposes. Overall, we find that existing benchmarks for few-shot learning are not designed in such a way that one can get a reliable picture of how effectively methods can be used on individual tasks.

翻译：在过去的十年中，针对小样本学习提出了大量基准测试。然而，所有这些基准测试都侧重于多个任务的平均性能，而对于该领域内如何可靠地评估和调整为单个任务训练的模型这一问题，尚未得到解决。本文首次研究了任务级评估——这是部署模型时的一个基本步骤。我们测量了小样本设置下性能估计器的准确性，考虑了模型选择的策略，并探讨了那些通常被认为鲁棒的评估器失败的原因。我们得出结论：对于直接估计模型性能，折数较少的交叉验证是最佳选择；而对于模型选择目的，使用自助法或折数较大的交叉验证更为合适。总体而言，我们发现现有的小样本学习基准测试在设计上无法可靠地反映方法在单个任务上的有效使用情况。