Activation function is a pivotal component of deep learning, facilitating the extraction of intricate data patterns. While classical activation functions like ReLU and its variants are extensively utilized, their static nature and simplicity, despite being advantageous, often limit their effectiveness in specialized tasks. The trainable activation functions also struggle sometimes to adapt to the unique characteristics of the data. Addressing these limitations, we introduce a novel trainable activation function, adaptive piecewise approximated activation linear unit (APALU), to enhance the learning performance of deep learning across a broad range of tasks. It presents a unique set of features that enable it to maintain stability and efficiency in the learning process while adapting to complex data representations. Experiments reveal significant improvements over widely used activation functions for different tasks. In image classification, APALU increases MobileNet and GoogleNet accuracy by 0.37% and 0.04%, respectively, on the CIFAR10 dataset. In anomaly detection, it improves the average area under the curve of One-CLASS Deep SVDD by 0.8% on the MNIST dataset, 1.81% and 1.11% improvements with DifferNet, and knowledge distillation, respectively, on the MVTech dataset. Notably, APALU achieves 100% accuracy on a sign language recognition task with a limited dataset. For regression tasks, APALU enhances the performance of deep neural networks and recurrent neural networks on different datasets. These improvements highlight the robustness and adaptability of APALU across diverse deep-learning applications.

翻译：激活函数是深度学习的关键组成部分，有助于提取复杂的数据模式。虽然ReLU及其变体等经典激活函数被广泛使用，但其静态特性和简单性虽然有利，却常常限制了它们在特定任务中的有效性。可训练激活函数有时也难以适应数据的独特特征。针对这些局限性，我们提出了一种新颖的可训练激活函数——自适应分段近似激活线性单元（APALU），以提升深度学习在广泛任务中的学习性能。它具备一系列独特特性，能够在适应复杂数据表示的同时，保持学习过程中的稳定性和效率。实验表明，在不同任务中，APALU相比广泛使用的激活函数有显著改进。在图像分类方面，APALU在CIFAR10数据集上将MobileNet和GoogleNet的准确率分别提升了0.37%和0.04%。在异常检测中，它在MNIST数据集上将One-Class Deep SVDD的平均曲线下面积提升了0.8%，在MVTech数据集上通过DifferNet和知识蒸馏分别提升了1.81%和1.11%。值得注意的是，在有限数据集的手语识别任务上，APALU实现了100%的准确率。对于回归任务，APALU在不同数据集上提升了深度神经网络和循环神经网络的性能。这些改进凸显了APALU在多样深度学习应用中的鲁棒性和适应性。