Nowadays artificial neural network models achieve remarkable results in many disciplines. Functions mapping the representation provided by the model to the probability distribution are the inseparable aspect of deep learning solutions. Although softmax is a commonly accepted probability mapping function in the machine learning community, it cannot return sparse outputs and always spreads the positive probability to all positions. In this paper, we propose r-softmax, a modification of the softmax, outputting sparse probability distribution with controllable sparsity rate. In contrast to the existing sparse probability mapping functions, we provide an intuitive mechanism for controlling the output sparsity level. We show on several multi-label datasets that r-softmax outperforms other sparse alternatives to softmax and is highly competitive with the original softmax. We also apply r-softmax to the self-attention module of a pre-trained transformer language model and demonstrate that it leads to improved performance when fine-tuning the model on different natural language processing tasks.

翻译：现今，人工神经网络模型在许多学科领域内取得了显著成果。将模型提供的表示映射到概率分布的函数是深度学习解决方案中不可或缺的一环。尽管softmax在机器学习社区中被广泛接受为概率映射函数，但它无法返回稀疏输出，并总是将正概率分布到所有位置。本文提出r-softmax——一种softmax的改进版本，能够输出具有可控稀疏率的稀疏概率分布。与现有的稀疏概率映射函数不同，我们提供了一种直观的机制来控制输出稀疏程度。我们在多个多标签数据集上证明，r-softmax优于其他稀疏替代softmax函数，并与原始softmax相比具有很强的竞争力。我们还将r-softmax应用于预训练Transformer语言模型的自注意力模块，并证明在针对不同自然语言处理任务微调模型时，它能带来性能提升。