The activation function in neural network introduces the non-linearity required to deal with the complex tasks. Several activation/non-linearity functions are developed for deep learning models. However, most of the existing activation functions suffer due to the dying gradient problem and non-utilization of the large negative input values. In this paper, we propose a Linearly Scaled Hyperbolic Tangent (LiSHT) for Neural Networks (NNs) by scaling the Tanh linearly. The proposed LiSHT is non-parametric and tackles the dying gradient problem. We perform the experiments on benchmark datasets of different type, such as vector data, image data and natural language data. We observe the superior performance using Multi-layer Perceptron (MLP), Residual Network (ResNet) and Long-short term memory (LSTM) for data classification, image classification and tweets classification tasks, respectively. The accuracy on CIFAR100 dataset using ResNet model with LiSHT is improved by 9.48, 3.40, 3.16, 4.26, and 1.17\% as compared to Tanh, ReLU, PReLU, LReLU, and Swish, respectively. We also show the qualitative results using loss landscape, weight distribution and activations maps in support of the proposed activation function.

翻译：激活函数为神经网络引入了处理复杂任务所需的非线性特性。针对深度学习模型，研究者已开发出多种激活/非线性函数。然而，现有激活函数大多存在梯度消失问题以及负大输入值利用率低等缺陷。本文提出一种面向神经网络的线性缩放双曲正切函数（LiSHT），通过对双曲正切（Tanh）函数进行线性缩放实现。该LiSHT函数具有非参数特性，可有效解决梯度消失问题。我们在向量数据、图像数据和自然语言数据等多类型基准数据集上进行实验，观察到采用多层感知机（MLP）、残差网络（ResNet）和长短期记忆网络（LSTM）时，LiSHT在数据分类、图像分类及推文分类任务中均展现优越性能。在CIFAR100数据集上，使用LiSHT的ResNet模型相较Tanh、ReLU、PReLU、LReLU和Swish，准确率分别提升9.48%、3.40%、3.16%、4.26%和1.17%。我们还通过损失景观、权重分布和激活图等定性结果，进一步验证所提激活函数的有效性。