OpenAI has released the Chat Generative Pre-trained Transformer (ChatGPT) and revolutionized the approach in artificial intelligence to human-model interaction. Several publications on ChatGPT evaluation test its effectiveness on well-known natural language processing (NLP) tasks. However, the existing studies are mostly non-automated and tested on a very limited scale. In this work, we examined ChatGPT's capabilities on 25 diverse analytical NLP tasks, most of them subjective even to humans, such as sentiment analysis, emotion recognition, offensiveness, and stance detection. In contrast, the other tasks require more objective reasoning like word sense disambiguation, linguistic acceptability, and question answering. We also evaluated GPT-4 model on five selected subsets of NLP tasks. We automated ChatGPT and GPT-4 prompting process and analyzed more than 49k responses. Our comparison of its results with available State-of-the-Art (SOTA) solutions showed that the average loss in quality of the ChatGPT model was about 25% for zero-shot and few-shot evaluation. For GPT-4 model, a loss for semantic tasks is significantly lower than for ChatGPT. We showed that the more difficult the task (lower SOTA performance), the higher the ChatGPT loss. It especially refers to pragmatic NLP problems like emotion recognition. We also tested the ability to personalize ChatGPT responses for selected subjective tasks via Random Contextual Few-Shot Personalization, and we obtained significantly better user-based predictions. Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI. Our results provide the basis for a fundamental discussion of whether the high quality of recent predictive NLP models can indicate a tool's usefulness to society and how the learning and validation procedures for such systems should be established.

翻译：OpenAI发布了聊天生成预训练变换器（ChatGPT），彻底改变了人工智能与人类交互的方式。多篇关于ChatGPT评估的出版物测试了其在著名自然语言处理（NLP）任务上的有效性。然而，现有研究大多是非自动化的，且测试规模非常有限。在这项工作中，我们考察了ChatGPT在25种不同的分析型NLP任务上的能力，其中大部分任务即便对人类来说也带有主观性，例如情感分析、情绪识别、冒犯性检测和立场检测。相比之下，其他任务则需要更客观的推理，如词义消歧、语言可接受性和问答。我们还在五个选定的NLP任务子集上评估了GPT-4模型。我们自动化了ChatGPT和GPT-4的提示过程，并分析了超过4.9万条响应。将其结果与现有最先进（SOTA）解决方案进行比较后发现，在零样本和少样本评估中，ChatGPT模型的平均质量损失约为25%。对于GPT-4模型，语义任务的损失显著低于ChatGPT。我们表明，任务越困难（SOTA性能越低），ChatGPT的损失越高。这尤其涉及像情绪识别这类实用型NLP问题。我们还通过随机上下文少样本个性化方法测试了在选定主观任务上个性化ChatGPT响应的能力，并获得了显著更优的基于用户的预测。额外的定性分析揭示了ChatGPT的偏差，这很可能是由于OpenAI对人类训练员施加的规则所致。我们的结果为以下基础性讨论提供了依据：近期高预测性能的NLP模型是否预示着该工具对社会的实用性，以及应如何建立此类系统的学习和验证程序。