Pre-trained language models (PLMs) have achieved remarkable success in NLP tasks. Despite the great success, mainstream solutions largely follow the pre-training then finetuning paradigm, which brings in both high deployment costs and low training efficiency. Nevertheless, fine-tuning on a specific task is essential because PLMs are only pre-trained with language signal from large raw data. In this paper, we propose a novel fine-tuning-free strategy for language models, to consider both language signal and teacher signal. Teacher signal is an abstraction of a battery of downstream tasks, provided in a unified proposition format. Trained with both language and strong task-aware teacher signals in an interactive manner, our FreeLM model demonstrates strong generalization and robustness. FreeLM outperforms large models e.g., GPT-3 and InstructGPT, on a range of language understanding tasks in experiments. FreeLM is much smaller with 0.3B parameters, compared to 175B in these models.

翻译：预训练语言模型（PLMs）已在自然语言处理任务中取得显著成功。然而，尽管成果卓著，主流方案仍主要遵循"预训练-微调"范式，这既带来高昂的部署成本，又导致训练效率低下。由于PLMs仅通过大规模原始数据中的语言信号进行预训练，针对特定任务的微调显得不可或缺。本文提出一种新颖的免微调语言模型策略，同时兼顾语言信号与教师信号。教师信号是对一系列下游任务的抽象概括，以统一的命题格式呈现。通过以交互方式同时利用语言信号与强任务感知教师信号进行训练，我们的FreeLM模型展现出强大的泛化能力与鲁棒性。实验结果表明，FreeLM在一系列语言理解任务上优于GPT-3、InstructGPT等大规模模型。相较于这些模型的1750亿参数规模，FreeLM仅需3亿参数即可实现更优性能。