Activation functions play a critical role in the performance and behaviour of neural networks, significantly impacting their ability to learn and generalise. Traditional activation functions, such as ReLU, sigmoid, and tanh, have been widely used with considerable success. However, these functions may not always provide optimal performance for all tasks and datasets. In this paper, we introduce Neuvo GEAF - an innovative approach leveraging grammatical evolution (GE) to automatically evolve novel activation functions tailored to specific neural network architectures and datasets. Experiments conducted on well-known binary classification datasets show statistically significant improvements in F1-score (between 2.4% and 9.4%) over ReLU using identical network architectures. Notably, these performance gains were achieved without increasing the network's parameter count, supporting the trend toward more efficient neural networks that can operate effectively on resource-constrained edge devices. This paper's findings suggest that evolved activation functions can provide significant performance improvements for compact networks while maintaining energy efficiency during both training and inference phases.

翻译：激活函数在神经网络的性能与行为中起着关键作用，显著影响其学习与泛化能力。传统激活函数（如ReLU、sigmoid和tanh）已被广泛使用并取得显著成功。然而，这些函数未必能为所有任务和数据集提供最优性能。本文提出Nuevo GEAF——一种创新方法，利用文法进化（GE）自动演化出针对特定神经网络架构和数据集定制的新型激活函数。在经典二分类数据集上进行的实验表明，在相同网络架构下，相较于ReLU，该方法在F1分数上取得了统计学意义的显著提升（提升幅度介于2.4%至9.4%之间）。值得注意的是，这些性能提升并未增加网络参数量，顺应了神经网络向高效化发展的趋势，使其能在资源受限的边缘设备上有效运行。本文的研究结果表明，演化得到的激活函数能够为紧凑型网络带来显著的性能提升，同时在训练和推理阶段保持优异的能效表现。