基于神经常微分方程的原型优化用于少样本学习 (Prototype Optimization with Neural ODE for Few-Shot Learning)

from arxiv, An extended version of metanode: prototype optimization as a neural ode for few-shot learning. arXiv admin note: text overlap with arXiv:2103.14341

Few-Shot Learning (FSL) is a challenging task, which aims to recognize novel classes with few examples. Pre-training based methods effectively tackle the problem by pre-training a feature extractor and then performing class prediction via a cosine classifier with mean-based prototypes. Nevertheless, due to the data scarcity, the mean-based prototypes are usually biased. In this paper, we attempt to diminish the prototype bias by regarding it as a prototype optimization problem. To this end, we propose a novel prototype optimization framework to rectify prototypes, i.e., introducing a meta-optimizer to optimize prototypes. Although the existing meta-optimizers can also be adapted to our framework, they all overlook a crucial gradient bias issue, i.e., the mean-based gradient estimation is also biased on sparse data. To address this issue, in this paper, we regard the gradient and its flow as meta-knowledge and then propose a novel Neural Ordinary Differential Equation (ODE)-based meta-optimizer to optimize prototypes, called MetaNODE. Although MetaNODE has shown superior performance, it suffers from a huge computational burden. To further improve its computation efficiency, we conduct a detailed analysis on MetaNODE and then design an effective and efficient MetaNODE extension version (called E2MetaNODE). It consists of two novel modules: E2GradNet and E2Solver, which aim to estimate accurate gradient flows and solve optimal prototypes in an effective and efficient manner, respectively. Extensive experiments show that 1) our methods achieve superior performance over previous FSL methods and 2) our E2MetaNODE significantly improves computation efficiency meanwhile without performance degradation.

翻译：少样本学习（FSL）是一项具有挑战性的任务，其目标是在仅有少量样本的情况下识别新类别。基于预训练的方法通过预训练一个特征提取器，然后使用基于均值的原型通过余弦分类器进行类别预测，从而有效解决了该问题。然而，由于数据稀缺，基于均值的原型通常存在偏差。本文试图通过将其视为一个原型优化问题来减少原型偏差。为此，我们提出了一种新颖的原型优化框架来修正原型，即引入一个元优化器来优化原型。尽管现有的元优化器也可以适配到我们的框架中，但它们都忽视了一个关键的梯度偏差问题，即基于均值的梯度估计在稀疏数据上同样存在偏差。为了解决这个问题，本文将梯度及其流视为元知识，进而提出了一种新颖的基于神经常微分方程（ODE）的元优化器来优化原型，称为MetaNODE。尽管MetaNODE表现出优越的性能，但其计算负担巨大。为了进一步提升其计算效率，我们对MetaNODE进行了详细分析，并设计了一个高效且有效的MetaNODE扩展版本（称为E2MetaNODE）。它包含两个新颖的模块：E2GradNet和E2Solver，分别旨在以高效且有效的方式估计准确的梯度流和求解最优原型。大量实验表明：1）我们的方法在性能上优于以往的FSL方法；2）我们的E2MetaNODE在显著提升计算效率的同时，没有造成性能下降。