Large pre-trained transformers are show-stealer in modern-day deep learning, and it becomes crucial to comprehend the parsimonious patterns that exist within them as they grow in scale. With exploding parameter counts, Lottery Ticket Hypothesis (LTH) and its variants, have lost their pragmatism in sparsifying them due to high computation and memory bottleneck of the repetitive train-prune-retrain routine of iterative magnitude pruning (IMP) which worsens with increasing model size. In this paper, we comprehensively study induced sparse patterns across multiple large pre-trained vision and language transformers. We propose the existence of -- essential sparsity defined with a sharp dropping point beyond which the performance declines much faster w.r.t the rise of sparsity level, when we directly remove weights with the smallest magnitudes in one-shot. In the sparsity-performance curve We also present an intriguing emerging phenomenon of abrupt sparsification during the pre-training of BERT, i.e., BERT suddenly becomes heavily sparse in pre-training after certain iterations. Moreover, our observations also indicate a counter-intuitive finding that BERT trained with a larger amount of pre-training data tends to have a better ability to condense knowledge in comparatively relatively fewer parameters. Lastly, we investigate the effect of the pre-training loss on essential sparsity and discover that self-supervised learning (SSL) objectives trigger stronger emergent sparsification properties than supervised learning (SL). Our codes are available at \url{https://github.com/VITA-Group/essential\_sparsity}.

翻译：大型预训练变换器是现代深度学习的焦点，随着其规模扩大，理解其中存在的简约模式变得至关重要。由于参数数量激增，彩票假说（LTH）及其变体在稀疏化这些模型时失去了实用性，因为迭代幅度剪枝（IMP）重复的“训练-剪枝-再训练”过程导致计算和内存瓶颈加剧，且随着模型尺寸增加而恶化。在本文中，我们全面研究了多个大型预训练视觉和语言变换器中的诱导稀疏模式。我们提出了一种“本质稀疏性”的存在，定义为在一次性移除最小幅度的权重时，稀疏度水平上升导致性能下降速度急剧加快的临界点。在稀疏-性能曲线中，我们还发现了一个有趣的涌现现象：BERT在预训练过程中会出现突然的稀疏化，即经过一定迭代次数后，BERT突然变得高度稀疏。此外，我们的观察还揭示了一个反直觉的发现：使用更多预训练数据训练的BERT，往往能更好地将知识浓缩在相对较少的参数中。最后，我们研究了预训练损失对本质稀疏性的影响，并发现自监督学习（SSL）目标比监督学习（SL）能触发更强的涌现稀疏化特性。我们的代码可在\url{https://github.com/VITA-Group/essential_sparsity}获取。