Effectively scaling large Transformer models is a main driver of recent advances in natural language processing. Dynamic neural networks, as an emerging research direction, are capable of scaling up neural networks with sub-linear increases in computation and time by dynamically adjusting their computational path based on the input. Dynamic neural networks could be a promising solution to the growing parameter numbers of pretrained language models, allowing both model pretraining with trillions of parameters and faster inference on mobile devices. In this survey, we summarize progress of three types of dynamic neural networks in NLP: skimming, mixture of experts, and early exit. We also highlight current challenges in dynamic neural networks and directions for future research.

翻译：有效扩展大型Transformer模型是近期自然语言处理领域取得进展的主要驱动力。动态神经网络作为新兴研究方向，能够根据输入动态调整计算路径，以亚线性计算和时间开销实现神经网络规模的扩展。动态神经网络有望解决预训练语言模型参数持续增长的问题，既能支持具有万亿参数规模的模型预训练，也能在移动设备上实现更快的推理速度。本综述系统梳理了自然语言处理中三类动态神经网络的研究进展：跳跃式处理、混合专家模型和早期退出机制，同时重点阐述了当前动态神经网络面临的挑战及未来研究方向。