Recent advancements in large language models have demonstrated remarkable capabilities across various NLP tasks. But many questions remain, including whether open-source models match closed ones, why these models excel or struggle with certain tasks, and what types of practical procedures can improve performance. We address these questions in the context of classification by evaluating three classes of models using eight datasets across three distinct tasks: named entity recognition, political party prediction, and misinformation detection. While larger LLMs often lead to improved performance, open-source models can rival their closed-source counterparts by fine-tuning. Moreover, supervised smaller models, like RoBERTa, can achieve similar or even greater performance in many datasets compared to generative LLMs. On the other hand, closed models maintain an advantage in hard tasks that demand the most generalizability. This study underscores the importance of model selection based on task requirements

翻译：近期大型语言模型的进展在各种自然语言处理任务中展现出卓越能力。但仍存诸多疑问，包括开源模型能否媲美闭源模型、这些模型为何在某些任务中表现优异或面临挑战，以及何种实践方法能有效提升性能。本研究聚焦分类任务，通过三个不同任务（命名实体识别、政党倾向预测、虚假信息检测）中的八个数据集对三类模型进行评估。结果表明：虽然大规模LLM通常能带来性能提升，但开源模型可通过微调与闭源模型相匹敌；此外，受监督的小型模型（如RoBERTa）在多数数据集上可达到甚至超越生成式LLM的性能。另一方面，闭源模型在需要最强泛化能力的困难任务中仍保持优势。本研究强调了基于任务需求进行模型选择的重要性。