In this paper, we present a novel training approach called the Homotopy Relaxation Training Algorithm (HRTA), aimed at accelerating the training process in contrast to traditional methods. Our algorithm incorporates two key mechanisms: one involves building a homotopy activation function that seamlessly connects the linear activation function with the ReLU activation function; the other technique entails relaxing the homotopy parameter to enhance the training refinement process. We have conducted an in-depth analysis of this novel method within the context of the neural tangent kernel (NTK), revealing significantly improved convergence rates. Our experimental results, especially when considering networks with larger widths, validate the theoretical conclusions. This proposed HRTA exhibits the potential for other activation functions and deep neural networks.

翻译：本文提出了一种名为同伦松弛训练算法（HRTA）的新型训练方法，旨在加快训练过程，相较于传统方法具有显著优势。该算法包含两个关键机制：一是构建同伦激活函数，该函数无缝连接线性激活函数与ReLU激活函数；二是通过松弛同伦参数来增强训练优化过程。我们在神经正切核（NTK）框架下对该新方法进行了深入分析，揭示了其收敛速度的显著提升。实验结果，特别是在考虑更宽的网络时，验证了理论结论。本文提出的HRTA方法具有推广至其他激活函数与深度神经网络的潜力。