Neural networks have shown tremendous growth in recent years to solve numerous problems. Various types of neural networks have been introduced to deal with different types of problems. However, the main goal of any neural network is to transform the non-linearly separable input data into more linearly separable abstract features using a hierarchy of layers. These layers are combinations of linear and nonlinear functions. The most popular and common non-linearity layers are activation functions (AFs), such as Logistic Sigmoid, Tanh, ReLU, ELU, Swish and Mish. In this paper, a comprehensive overview and survey is presented for AFs in neural networks for deep learning. Different classes of AFs such as Logistic Sigmoid and Tanh based, ReLU based, ELU based, and Learning based are covered. Several characteristics of AFs such as output range, monotonicity, and smoothness are also pointed out. A performance comparison is also performed among 18 state-of-the-art AFs with different networks on different types of data. The insights of AFs are presented to benefit the researchers for doing further research and practitioners to select among different choices. The code used for experimental comparison is released at: \url{https://github.com/shivram1987/ActivationFunctions}.

翻译：近些年来,神经网络表现出了巨大的增长,以解决许多问题。各种类型的神经网络已经引入了各种类型的神经网络,以处理不同类型的问题。然而,任何神经网络的主要目标是利用层次层次的层次将非线性分离的输入数据转换成更线性分离的抽象特征。这些层次是线性和非线性功能的组合。最流行和常见的非线性层是激活功能(AFs),如物流Sigmoid、Tanh、ReLU、ELU、Swish和Mish。本文为神经网络中的AFs提供了全面的概览和调查。各种AFs的洞见为深层次学习提供了参考,例如物流Sigmoid和Tanh基础的、RELU基础的、ELU基础的和学习基础的不同类别的AFs 类别。一些AF的特性,如产出范围、单调和光滑等。在18个州-艺术AFs和不同类型数据网络之间也进行了业绩比较。AFsur-theart AFs之间对不同类型数据的比较。AFs的洞察介绍了AFs的见解,以利研究人员在进行进一步的实验性比较。用于进行进一步的实验性研究。使用。