We study here a fixed mini-batch gradient decent (FMGD) algorithm to solve optimization problems with massive datasets. In FMGD, the whole sample is split into multiple non-overlapping partitions. Once the partitions are formed, they are then fixed throughout the rest of the algorithm. For convenience, we refer to the fixed partitions as fixed mini-batches. Then for each computation iteration, the gradients are sequentially calculated on each fixed mini-batch. Because the size of fixed mini-batches is typically much smaller than the whole sample size, it can be easily computed. This leads to much reduced computation cost for each computational iteration. It makes FMGD computationally efficient and practically more feasible. To demonstrate the theoretical properties of FMGD, we start with a linear regression model with a constant learning rate. We study its numerical convergence and statistical efficiency properties. We find that sufficiently small learning rates are necessarily required for both numerical convergence and statistical efficiency. Nevertheless, an extremely small learning rate might lead to painfully slow numerical convergence. To solve the problem, a diminishing learning rate scheduling strategy can be used. This leads to the FMGD estimator with faster numerical convergence and better statistical efficiency. Finally, the FMGD algorithms with random shuffling and a general loss function are also studied.

翻译：本文研究了一种固定小批量梯度下降算法，用于解决大规模数据集下的优化问题。在该算法中，整个样本被划分为多个互不重叠的分区，这些分区一旦形成，在后续算法过程中保持不变。为方便起见，我们将这些固定分区称为固定小批量。在每个计算迭代中，梯度依次在每个固定小批量上计算。由于固定小批量的大小通常远小于整个样本量，计算过程相对容易，从而大幅降低了每次迭代的计算成本，使得固定小批量梯度下降算法在计算上高效且在实际应用中更具可行性。为证明该算法的理论性质，我们首先以恒定学习率的线性回归模型为起点，研究其数值收敛性和统计效率特性。研究发现，充分小的学习率对于数值收敛和统计效率两者均必不可少。然而，极小的学习率可能导致数值收敛速度过慢。为解决这一问题，可采用递减学习率调度策略，这使得固定小批量梯度下降估计量具有更快的数值收敛速度和更好的统计效率。最后，本文还研究了采用随机排序和一般损失函数的固定小批量梯度下降算法。