Machine learning problems rely heavily on stochastic gradient descent (SGD) for optimization. The effectiveness of SGD is contingent upon accurately estimating gradients from a mini-batch of data samples. Instead of the commonly used uniform sampling, adaptive or importance sampling reduces noise in gradient estimation by forming mini-batches that prioritize crucial data points. Previous research has suggested that data points should be selected with probabilities proportional to their gradient norm. Nevertheless, existing algorithms have struggled to efficiently integrate importance sampling into machine learning frameworks. In this work, we make two contributions. First, we present an algorithm that can incorporate existing importance functions into our framework. Second, we propose a simplified importance function that relies solely on the loss gradient of the output layer. By leveraging our proposed gradient estimation techniques, we observe improved convergence in classification and regression tasks with minimal computational overhead. We validate the effectiveness of our adaptive and importance-sampling approach on image and point-cloud datasets.

翻译：机器学习问题严重依赖随机梯度下降法（SGD）进行优化。SGD的有效性取决于能否通过小批量数据样本精确估计梯度。与常用的均匀采样不同，自适应采样或重要性采样通过构建优先关注关键数据点的小批量，降低了梯度估计中的噪声。已有研究表明，数据点应按照与其梯度范数成比例的概率进行选择。然而，现有算法难以将重要性采样高效整合到机器学习框架中。本文做出两项贡献：第一，提出一种能够将现有重要性函数纳入我们框架的算法；第二，提出一种仅依赖输出层损失梯度的简化重要性函数。通过采用我们提出的梯度估计技术，在分类和回归任务中观察到收敛性能的提升，且计算开销极小。我们在图像和点云数据集上验证了自适应采样与重要性采样方法的有效性。