Large Language Models pre-trained with self-supervised learning have demonstrated impressive zero-shot generalization capabilities on a wide spectrum of tasks. In this work, we present WeLM: a well-read pre-trained language model for Chinese that is able to seamlessly perform different types of tasks with zero or few-shot demonstrations. WeLM is trained with 10B parameters by "reading" a curated high-quality corpus covering a wide range of topics. We show that WeLM is equipped with broad knowledge on various domains and languages. On 18 monolingual (Chinese) tasks, WeLM can significantly outperform existing pre-trained models with similar sizes and match the performance of models up to 25 times larger. WeLM also exhibits strong capabilities in multi-lingual and code-switching understanding, outperforming existing multilingual language models pre-trained on 30 languages. Furthermore, We collected human-written prompts for a large set of supervised datasets in Chinese and fine-tuned WeLM with multi-prompted training. The resulting model can attain strong generalization on unseen types of tasks and outperform the unsupervised WeLM in zero-shot learning. Finally, we demonstrate that WeLM has basic skills at explaining and calibrating the decisions from itself, which can be promising directions for future research. Our models can be applied from https://welm.weixin.qq.com/docs/api/.

翻译：通过自监督学习预训练的大型语言模型在广泛任务上展现出惊人的零样本泛化能力。本文提出WeLM：一款博闻广识的中文预训练语言模型，能够以零样本或少样本示例无缝执行不同类型的任务。WeLM通过“阅读”涵盖广泛主题的高质量精选语料库，以100亿参数进行训练。我们证明WeLM具备跨领域和跨语言的广博知识。在18项中文单语任务中，WeLM显著优于同等规模的现有预训练模型，并达到体积大25倍模型的性能水平。WeLM还展现出强大的多语言及语码转换理解能力，优于基于30种语言预训练的现有多语言模型。此外，我们收集了大规模中文监督数据集的人工编写提示，并通过多提示训练对WeLM进行微调。所得模型能在未见任务类型上实现强泛化，并在零样本学习中优于无监督WeLM。最后，我们证明WeLM具备解释并校准自身决策的基本能力，这为未来研究提供了有前景的方向。我们的模型可通过https://welm.weixin.qq.com/docs/api/ 获取。