In the literature on deep neural networks, there is considerable interest in developing activation functions that can enhance neural network performance. In recent years, there has been renewed scientific interest in proposing activation functions that can be trained throughout the learning process, as they appear to improve network performance, especially by reducing overfitting. In this paper, we propose a trainable activation function whose parameters need to be estimated. A fully Bayesian model is developed to automatically estimate from the learning data both the model weights and activation function parameters. An MCMC-based optimization scheme is developed to build the inference. The proposed method aims to solve the aforementioned problems and improve convergence time by using an efficient sampling scheme that guarantees convergence to the global maximum. The proposed scheme is tested on three datasets with three different CNNs. Promising results demonstrate the usefulness of our proposed approach in improving model accuracy due to the proposed activation function and Bayesian estimation of the parameters.

翻译：在深度神经网络的文献中，开发能提升网络性能的激活函数一直备受关注。近年来，可训练的激活函数再次成为研究热点，因其能提升网络性能，尤其能有效减少过拟合。本文提出一种参数需估计的可训练激活函数，并构建了全贝叶斯模型，从学习数据中自动估计模型权重及激活函数参数。我们开发了基于MCMC的优化方案进行推断，该方法通过能保证收敛至全局最优的高效采样方案，旨在解决前述问题并缩短收敛时间。本方案在三个数据集上使用三种不同卷积神经网络进行测试，结果表明，由于所提出的激活函数及参数的贝叶斯估计方法，我们的方法在提升模型准确率方面具有显著价值。