Transformer language models have received widespread public attention, yet their generated text is often surprising even to NLP researchers. In this survey, we discuss over 250 recent studies of English language model behavior before task-specific fine-tuning. Language models possess basic capabilities in syntax, semantics, pragmatics, world knowledge, and reasoning, but these capabilities are sensitive to specific inputs and surface features. Despite dramatic increases in generated text quality as models scale to hundreds of billions of parameters, the models are still prone to unfactual responses, commonsense errors, memorized text, and social biases. Many of these weaknesses can be framed as over-generalizations or under-generalizations of learned patterns in text. We synthesize recent results to highlight what is currently known about what large language models can and cannot do.

翻译：Transformer语言模型已获得公众广泛关注，但其生成的文本即便对自然语言处理研究人员而言也常令人惊讶。本综述梳理了超过250项关于英语语言模型在特定任务微调前行为的研究。语言模型具备语法、语义、语用、世界知识与推理方面的基础能力，但这些能力对特定输入和表层特征较为敏感。尽管模型参数规模扩展至数千亿量级后生成文本质量显著提升，其仍然容易出现事实性错误、常识性谬误、记忆性文本复制以及社会偏见等问题。这些缺陷大多可归因于模型对文本中学习模式的过度泛化或欠泛化。本文综合最新研究成果，系统阐释了当前关于大型语言模型能力边界的主要认知。