Parameter-shared pre-trained language models (PLMs) have emerged as a successful approach in resource-constrained environments, enabling substantial reductions in model storage and memory costs without significant performance compromise. However, it is important to note that parameter sharing does not alleviate computational burdens associated with inference, thus impeding its practicality in situations characterized by limited stringent latency requirements or computational resources. Building upon neural ordinary differential equations (ODEs), we introduce a straightforward technique to enhance the inference efficiency of parameter-shared PLMs. Additionally, we propose a simple pre-training technique that leads to fully or partially shared models capable of achieving even greater inference acceleration. The experimental results demonstrate the effectiveness of our methods on both autoregressive and autoencoding PLMs, providing novel insights into more efficient utilization of parameter-shared models in resource-constrained settings.

翻译：参数共享预训练语言模型（PLMs）已成为资源受限环境中一种成功的方法，能够在不大幅降低性能的前提下显著减少模型存储和内存成本。然而，需注意参数共享并未减轻与推理相关的计算负担，从而阻碍了其在延迟要求严格或计算资源有限场景中的实用性。基于神经常微分方程（ODEs），我们引入了一种直接技术来提升参数共享PLMs的推理效率。此外，我们提出了一种简单的预训练技术，可生成完全或部分共享的模型，从而实现更大的推理加速。实验结果表明，我们的方法在自回归和自编码PLMs上均有效，为在资源受限环境中更高效利用参数共享模型提供了新见解。