Activation functions introduce nonlinearity into deep neural networks. Most popular activation functions allow positive values to pass through while blocking or suppressing negative values. From the idea that positive values and negative values are equally important, and they must compete for activation, we proposed a new Competition-based Adaptive ReLU (CAReLU). CAReLU scales the input values based on the competition results between positive values and negative values. It defines two parameters to adjust the scaling strategy and can be trained uniformly with other network parameters. We verify the effectiveness of CAReLU on image classification, super-resolution, and natural language processing tasks. In the experiment, our method performs better than other widely used activation functions. In the case of replacing ReLU in ResNet-18 with our proposed activation function, it improves the classification accuracy on the CIFAR-100 dataset. The effectiveness and the new perspective on the utilization of competition results between positive values and negative values make CAReLU a promising activation function.

翻译：激活函数为深度神经网络引入了非线性。最流行的激活函数允许正值通过，同时阻断或抑制负值。基于正负值同等重要且必须竞争激活的理念，我们提出了一种新的基于竞争的自适应ReLU（CAReLU）。CAReLU根据正值与负值之间的竞争结果对输入值进行缩放。它定义了两个参数来调整缩放策略，并能与其他网络参数统一训练。我们在图像分类、超分辨率和自然语言处理任务上验证了CAReLU的有效性。实验中，我们的方法优于其他广泛使用的激活函数。在将ResNet-18中的ReLU替换为我们提出的激活函数的情况下，该方法提高了在CIFAR-100数据集上的分类准确率。CAReLU的有效性以及对正负值竞争结果利用的新视角，使其成为一种具有前景的激活函数。