The recent discovery of special human neocortical pyramidal neurons that can individually learn the XOR function highlights the significant performance gap between biological and artificial neurons. The output of these pyramidal neurons first increases to a maximum with input and then decreases. Artificial neurons with similar characteristics can be designed with oscillating activation functions. Oscillating activation functions have multiple zeros allowing single neurons to have multiple hyper-planes in their decision boundary. This enables even single neurons to learn the XOR function. This paper proposes four new oscillating activation functions inspired by human pyramidal neurons that can also individually learn the XOR function. Oscillating activation functions are non-saturating for all inputs unlike popular activation functions, leading to improved gradient flow and faster convergence. Using oscillating activation functions instead of popular monotonic or non-monotonic single-zero activation functions enables neural networks to train faster and solve classification problems with fewer layers. An extensive comparison of 23 activation functions on CIFAR 10, CIFAR 100, and Imagentte benchmarks is presented and the oscillating activation functions proposed in this paper are shown to outperform all known popular activation functions.

翻译：近期发现的人类新皮层特殊锥体神经元能够单独学习XOR函数，这凸显了生物神经元与人工神经元之间的显著性能差距。这类锥体神经元的输出随输入先增至最大值后递减。具有类似特征的人工神经元可通过振荡激活函数实现设计。振荡激活函数具有多个零点，使单个神经元在其决策边界上形成多个超平面，这使单个神经元也能学习XOR函数。本文提出四种受人类锥体神经元启发的新型振荡激活函数，它们同样能独立学习XOR函数。与主流激活函数不同，振荡激活函数对所有输入均不饱和，从而改善梯度流动并加快收敛速度。使用振荡激活函数替代主流的单调或非单调单零点激活函数，可使神经网络训练更快，并以更少的层级解决分类问题。本文在CIFAR 10、CIFAR 100和ImageNet基准上对23种激活函数进行了广泛对比，并证明所提出的振荡激活函数优于所有已知的主流激活函数。