ChatGPT, the first large language model (LLM) with mass adoption, has demonstrated remarkable performance in numerous natural language tasks. Despite its evident usefulness, evaluating ChatGPT's performance in diverse problem domains remains challenging due to the closed nature of the model and its continuous updates via Reinforcement Learning from Human Feedback (RLHF). We highlight the issue of data contamination in ChatGPT evaluations, with a case study of the task of stance detection. We discuss the challenge of preventing data contamination and ensuring fair model evaluation in the age of closed and continuously trained models.

翻译：ChatGPT作为首个被广泛采用的大语言模型（LLM），在众多自然语言处理任务中展现出卓越性能。尽管其实用性显而易见，但由于该模型的封闭性及其通过人类反馈强化学习（RLHF）持续更新的特性，评估ChatGPT在不同问题领域的表现仍具挑战性。我们通过立场检测任务的案例研究，揭示了ChatGPT评估中的数据污染问题，并探讨了在封闭且持续训练的模型时代，如何防止数据污染并确保模型评估的公平性。