Activation functions play a central role in neural networks by shaping internal representations. Recently, learning binary activation representations has attracted significant attention due to their advantages in computational and memory efficiency, as well as interpretability. However, training neural networks with Heaviside activations remains challenging, as their non-differentiability obstructs standard gradient-based optimization. In this paper, we propose Heavy Tailed Activation Function (HTAF), a smooth approximation to the Heaviside function that enables stable training with gradient-based optimization. We construct HTAF as a sigmoid hyperbolic tangent composite function and theoretically show that it maintains a large gradient mass around zero inputs while exhibiting slower gradient decay in the tail regions. We show that Spiking Neural Networks, Binary Neural Networks and Deep Heaviside neural Networks can be trained stably using HTAF with gradient-based optimization. Finally, we introduce Implicit Concept Bottleneck Models (ICBMs), an interpretable image model that leverages HTAF to induce discrete feature representations. Extensive experiments across various architectures and image datasets demonstrate that ICBM enables stable discretization while achieving prediction performance comparable to or better than standard models.

翻译：激活函数通过塑造内部表示在神经网络中发挥着核心作用。近年来，学习二值激活表示因其在计算和存储效率以及可解释性方面的优势而备受关注。然而，使用赫维赛德激活函数训练神经网络仍然具有挑战性，因为其不可微性阻碍了标准的基于梯度的优化。本文提出了重尾激活函数（HTAF），这是一种对赫维赛德函数的平滑近似，能够实现基于梯度优化的稳定训练。我们将HTAF构建为Sigmoid双曲正切复合函数，并从理论上证明它在零输入附近保持较大的梯度质量，同时在尾部区域表现出较慢的梯度衰减。研究表明，使用HTAF结合基于梯度的优化，可以稳定地训练脉冲神经网络、二值神经网络和深度赫维赛德神经网络。最后，我们引入了隐式概念瓶颈模型（ICBM），这是一种可解释的图像模型，利用HTAF来诱导离散特征表示。跨多种架构和图像数据集的大量实验表明，ICBM能够实现稳定的离散化，同时达到与标准模型相当或更优的预测性能。