Activation functions are fundamental to deep neural networks, governing gradient flow, optimization stability, and representational capacity. Within historic deep architectures, while ReLU has been the dominant choice for the activation function, modern transformer-based models increasingly are adopting smoother alternatives such as GELU and other self-gated alternatives. Despite their empirical success, the mathematical relationships among these functions and the principles underlying their effectiveness remains only partially understood. We introduce IGLU, a parametric activation function derived as a scale mixture of GELU gates under a half-normal mixing distribution. This derivation yields a closed-form expression whose gating component is exactly the Cauchy CDF, providing a principled one-parameter family that continuously interpolates between identity-like and ReLU-like behavior via a single sharpness parameter $σ$. Unlike GELU's Gaussian gate, IGLU's heavy-tailed Cauchy gate decays polynomially in the negative tail, guaranteeing non-zero gradients for all finite inputs and offering greater robustness to vanishing gradients. We further introduce IGLU-Approx, a computationally efficient rational approximation of IGLU expressed entirely in terms of ReLU operations that eliminates transcendental function evaluation. Through evaluations on CIFAR-10, CIFAR-100, and WikiText-103 across ResNet-20, ViT-Tiny, and GPT-2 Small, IGLU achieves competitive or superior performance on both vision and language datasets against ReLU and GELU baselines, with IGLU-Approx recovering this performance at substantially reduced computational cost. In particular, we show that employing a heavy-tailed gate leads to considerable performance gains in heavily imbalanced classification datasets.

翻译：激活函数是深度神经网络的核心，控制着梯度流动、优化稳定性与表示能力。在经典深度架构中，ReLU一直是激活函数的主流选择，而现代基于Transformer的模型日益倾向于采用更平滑的替代方案，如GELU及其他自门控函数。尽管这些函数在实证中取得了成功，但它们在数学上的相互关系及其效能背后的原理仍仅得到部分理解。本文提出IGLU，一种以半正态混合分布下的GELU门控尺度混合导出的参数化激活函数。该推导过程得出一个闭合形式的表达式，其门控分量恰好为柯西累积分布函数（CDF），从而构建了一个通过单一尖锐度参数σ即可在恒等映射行为与ReLU行为之间连续插值的单参数系列。与GELU的高斯门控不同，IGLU的重尾柯西门控在负半轴呈现多项式衰减，确保所有有限输入均获得非零梯度，从而对梯度消失具有更强的鲁棒性。我们进一步提出IGLU-Approx，一种完全基于ReLU运算表示的计算高效的有理近似形式，消除了超越函数求值。通过在CIFAR-10、CIFAR-100和WikiText-103数据集上使用ResNet-20、ViT-Tiny和GPT-2 Small进行评估，IGLU在视觉和语言任务中均取得了与ReLU和GELU基线相当或更优的性能，而IGLU-Approx在显著降低计算成本的同时恢复了原有性能。特别地，我们证明在高度不平衡的分类数据集中，采用重尾门控能够带来显著的性能提升。