Deep learning training training algorithms are a huge success in recent years in many fields including speech, text,image video etc. Deeper and deeper layers are proposed with huge success with resnet structures having around 152 layers. Shallow convolution neural networks(CNN's) are still an active research, where some phenomena are still unexplained. Activation functions used in the network are of utmost importance, as they provide non linearity to the networks. Relu's are the most commonly used activation function.We show a complex piece-wise linear(PWL) activation in the hidden layer. We show that these PWL activations work much better than relu activations in our networks for convolution neural networks and multilayer perceptrons. Result comparison in PyTorch for shallow and deep CNNs are given to further strengthen our case.

翻译：深度学习训练算法近年来在语音、文本、图像及视频等多个领域取得了巨大成功。随着网络层数不断加深，诸如拥有约152层的残差网络结构（ResNet）也取得了显著成效。然而，浅层卷积神经网络（CNN）中仍存在一些未解现象，成为当前活跃的研究方向。网络中所使用的激活函数至关重要，它们为网络提供了非线性特性。ReLU是目前最常用的激活函数。本研究在隐藏层中引入了一种复杂的分段线性（PWL）激活函数。实验结果表明，在卷积神经网络和多层感知机中，这种PWL激活函数的性能显著优于ReLU激活函数。我们还在PyTorch框架下给出了浅层与深层CNN的对比结果，进一步验证了上述结论。