OpenAI has released the Chat Generative Pre-trained Transformer (ChatGPT) and revolutionized the approach in artificial intelligence to human-model interaction. Several publications on ChatGPT evaluation test its effectiveness on well-known natural language processing (NLP) tasks. However, the existing studies are mostly non-automated and tested on a very limited scale. In this work, we examined ChatGPT's capabilities on 25 diverse analytical NLP tasks, most of them subjective even to humans, such as sentiment analysis, emotion recognition, offensiveness, and stance detection. In contrast, the other tasks require more objective reasoning like word sense disambiguation, linguistic acceptability, and question answering. We also evaluated GPT-4 model on five selected subsets of NLP tasks. We automated ChatGPT and GPT-4 prompting process and analyzed more than 49k responses. Our comparison of its results with available State-of-the-Art (SOTA) solutions showed that the average loss in quality of the ChatGPT model was about 25% for zero-shot and few-shot evaluation. For GPT-4 model, a loss for semantic tasks is significantly lower than for ChatGPT. We showed that the more difficult the task (lower SOTA performance), the higher the ChatGPT loss. It especially refers to pragmatic NLP problems like emotion recognition. We also tested the ability to personalize ChatGPT responses for selected subjective tasks via Random Contextual Few-Shot Personalization, and we obtained significantly better user-based predictions. Additional qualitative analysis revealed a ChatGPT bias, most likely due to the rules imposed on human trainers by OpenAI. Our results provide the basis for a fundamental discussion of whether the high quality of recent predictive NLP models can indicate a tool's usefulness to society and how the learning and validation procedures for such systems should be established.

翻译：OpenAI发布了聊天生成预训练变换器（ChatGPT），彻底改变了人工智能中人类与模型交互的方式。多篇关于ChatGPT评估的出版物测试了其在知名自然语言处理（NLP）任务上的有效性。然而，现有研究大多为非自动化，且测试规模极为有限。在本工作中，我们考察了ChatGPT在25种多样化的分析性NLP任务上的能力，其中多数任务对人类而言也具有主观性，例如情感分析、情绪识别、冒犯性检测和立场检测。相比之下，其他任务则需要更客观的推理，如词义消歧、语言可接受性和问答。我们还评估了GPT-4模型在五个选定的NLP任务子集上的表现。我们自动化了ChatGPT和GPT-4的提示过程，并分析了超过49,000条响应。将其结果与现有最先进（SOTA）解决方案进行比较后，我们发现ChatGPT模型在零样本和少样本评估中，质量平均损失约25%。对于GPT-4模型，语义任务的损失显著低于ChatGPT。我们表明，任务越困难（SOTA性能越低），ChatGPT的损失越高。这尤其适用于诸如情绪识别等语用NLP问题。我们还通过随机上下文少样本个性化方法测试了在选定的主观任务上个性化ChatGPT响应的能力，并获得了显著更好的基于用户的预测。额外的定性分析揭示了ChatGPT的偏差，这很可能源于OpenAI对人类训练师施加的规则。我们的结果为以下根本性讨论提供了基础：近期高水平的预测性NLP模型是否能表明工具对社会的有用性，以及应如何建立此类系统的学习与验证流程。