Deep learning training training algorithms are a huge success in recent years in many fields including speech, text,image video etc. Deeper and deeper layers are proposed with huge success with resnet structures having around 152 layers. Shallow convolution neural networks(CNN's) are still an active research, where some phenomena are still unexplained. Activation functions used in the network are of utmost importance, as they provide non linearity to the networks. Relu's are the most commonly used activation function.We show a complex piece-wise linear(PWL) activation in the hidden layer. We show that these PWL activations work much better than relu activations in our networks for convolution neural networks and multilayer perceptrons. Result comparison in PyTorch for shallow and deep CNNs are given to further strengthen our case.

翻译：深度学习训练算法近年来在语音、文本、图像视频等多个领域取得了巨大成功。随着残差网络结构成功采用约152层，网络层数不断加深。浅层卷积神经网络（CNN）仍是一个活跃的研究领域，其中一些现象尚未得到解释。网络中使用的激活函数至关重要，因为它们为网络提供非线性特性。ReLU是使用最广泛的激活函数。我们在隐藏层中引入了一种复杂的分段线性（PWL）激活函数，并证明这些PWL激活函数在卷积神经网络和多层感知机中比ReLU激活函数表现更优。通过PyTorch中浅层和深层CNN的结果对比，进一步验证了我们的观点。