Nonlinear activation functions are pivotal to the success of deep neural nets, and choosing the appropriate activation function can significantly affect their performance. Most networks use fixed activation functions (e.g., ReLU, GELU, etc.), and this choice might limit their expressiveness. Furthermore, different layers may benefit from diverse activation functions. Consequently, there has been a growing interest in trainable activation functions. In this paper, we introduce DiTAC, a trainable highly-expressive activation function based on an efficient diffeomorphic transformation (called CPAB). Despite introducing only a negligible number of trainable parameters, DiTAC enhances model expressiveness and performance, often yielding substantial improvements. It also outperforms existing activation functions (regardless whether the latter are fixed or trainable) in tasks such as semantic segmentation, image generation, regression problems, and image classification. Our code is available at https://github.com/BGU-CS-VIL/DiTAC.

翻译：非线性激活函数对深度神经网络的成功至关重要，选择合适的激活函数能显著影响其性能。大多数网络使用固定的激活函数（如ReLU、GELU等），这种选择可能限制其表达能力。此外，不同层可能受益于多样化的激活函数。因此，可训练激活函数日益受到关注。本文提出DiTAC——一种基于高效微分同胚变换（称为CPAB）的可训练高表达能力激活函数。尽管仅引入可忽略数量的可训练参数，DiTAC仍能增强模型表达能力和性能，通常带来显著提升。在语义分割、图像生成、回归问题和图像分类等任务中，其表现均优于现有激活函数（无论后者是固定还是可训练的）。我们的代码发布于https://github.com/BGU-CS-VIL/DiTAC。