GPT series models, such as GPT-3, CodeX, InstructGPT, ChatGPT, and so on, have gained considerable attention due to their exceptional natural language processing capabilities. However, despite the abundance of research on the difference in capabilities between GPT series models and fine-tuned models, there has been limited attention given to the evolution of GPT series models' capabilities over time. To conduct a comprehensive analysis of the capabilities of GPT series models, we select six representative models, comprising two GPT-3 series models (i.e., davinci and text-davinci-001) and four GPT-3.5 series models (i.e., code-davinci-002, text-davinci-002, text-davinci-003, and gpt-3.5-turbo). We evaluate their performance on nine natural language understanding (NLU) tasks using 21 datasets. In particular, we compare the performance and robustness of different models for each task under zero-shot and few-shot scenarios. Our extensive experiments reveal that the overall ability of GPT series models on NLU tasks does not increase gradually as the models evolve, especially with the introduction of the RLHF training strategy. While this strategy enhances the models' ability to generate human-like responses, it also compromises their ability to solve some tasks. Furthermore, our findings indicate that there is still room for improvement in areas such as model robustness.

翻译：GPT系列模型，如GPT-3、CodeX、InstructGPT、ChatGPT等，因其卓越的自然语言处理能力而备受关注。然而，尽管已有大量研究对比GPT系列模型与微调模型之间的能力差异，但针对GPT系列模型自身能力随版本演化的规律却鲜有探讨。为全面剖析GPT系列模型的能力演化，本文选取六种代表性模型，包括两种GPT-3系列模型（davinci和text-davinci-001）及四种GPT-3.5系列模型（code-davinci-002、text-davinci-002、text-davinci-003和gpt-3.5-turbo），基于21个数据集在九项自然语言理解任务上进行评估。我们特别关注零样本与少样本场景下不同模型在各任务中的性能表现与鲁棒性。大量实验表明，GPT系列模型在自然语言理解任务上的整体能力并未随模型演化逐步提升，尤其在引入RLHF训练策略后更为显著——该策略虽增强了模型生成类人化回复的能力，却同时削弱了其在部分任务上的解决能力。此外，研究结果揭示，模型在鲁棒性等方面仍存在改进空间。