The introduction of ChatGPT has put Artificial Intelligence (AI) Natural Language Processing (NLP) in the spotlight. ChatGPT adoption has been exponential with millions of users experimenting with it in a myriad of tasks and application domains with impressive results. However, ChatGPT has limitations and suffers hallucinations, for example producing answers that look plausible but they are completely wrong. Evaluating the performance of ChatGPT and similar AI tools is a complex issue that is being explored from different perspectives. In this work, we contribute to those efforts with ChatWords, an automated test system, to evaluate ChatGPT knowledge of an arbitrary set of words. ChatWords is designed to be extensible, easy to use, and adaptable to evaluate also other NLP AI tools. ChatWords is publicly available and its main goal is to facilitate research on the lexical knowledge of AI tools. The benefits of ChatWords are illustrated with two case studies: evaluating the knowledge that ChatGPT has of the Spanish lexicon (taken from the official dictionary of the "Real Academia Espa\~nola") and of the words that appear in the Quixote, the well-known novel written by Miguel de Cervantes. The results show that ChatGPT is only able to recognize approximately 80% of the words in the dictionary and 90% of the words in the Quixote, in some cases with an incorrect meaning. The implications of the lexical knowledge of NLP AI tools and potential applications of ChatWords are also discussed providing directions for further work on the study of the lexical knowledge of AI tools.

翻译：ChatGPT 的推出将人工智能自然语言处理（NLP）推向了聚光灯下。ChatGPT 的应用呈指数级增长，数百万用户在各种任务和应用领域中对其进行了实验，结果令人印象深刻。然而，ChatGPT 存在局限性并会出现幻觉，例如，它可能生成看似合理但完全错误的答案。评估 ChatGPT 及类似人工智能工具的性能是一个复杂问题，目前正从不同角度进行探索。在本研究中，我们通过 ChatWords 系统为此类努力做出贡献。ChatWords 是一个自动化测试系统，用于评估 ChatGPT 对任意单词集的词汇知识。ChatWords 设计为可扩展、易于使用，并适用于评估其他 NLP 人工智能工具。ChatWords 是公开可用的，其主要目标是促进对人工智能工具词汇知识的研究。通过两个案例研究说明了 ChatWords 的优势：一是评估 ChatGPT 对西班牙语词汇（取自“皇家西班牙语学院”的官方词典）的知识，二是评估 ChatGPT 对米格尔·德·塞万提斯所著著名小说《堂吉诃德》中出现的单词的知识。结果表明，ChatGPT 仅能识别约 80% 的词典词汇和约 90% 的《堂吉诃德》中的词汇，且在部分情况下含义不正确。此外，还探讨了 NLP 人工智能工具词汇知识的影响以及 ChatWords 的潜在应用，为后续研究人工智能工具词汇知识的工作提供了方向。