Recently, ChatGPT has emerged as a powerful NLP tool that can carry out several tasks. However, the range of languages ChatGPT can handle remains largely a mystery. In this work, we investigate ChatGPT's language identification abilities. For this purpose, we compile Babel-670, a benchmark comprising $670$ languages representing $23$ language families. Languages in Babel-670 run the gamut between the very high-resource to the very low-resource and are spoken in five continents. We then study ChatGPT's (both GPT-3.5 and GPT-4) ability to (i) identify both language names and language codes (ii) under both zero- and few-shot conditions (iii) with and without provision of label set. When compared to smaller finetuned language identification tools, we find that ChatGPT lags behind. Our empirical analysis shows the reality that ChatGPT still resides in a state of potential enhancement before it can sufficiently serve diverse communities.

翻译：近期，ChatGPT作为一种强大的自然语言处理工具，能够执行多项任务。然而，其可处理的语言范围在很大程度上仍是一个谜团。本研究系统探究了ChatGPT的语言识别能力。为此，我们构建了Babel-670基准测试集，涵盖来自23个语系的670种语言。该测试集中的语言覆盖五个大洲，资源丰度从极高资源到极低资源不等。随后，我们考察了ChatGPT（包括GPT-3.5和GPT-4）在以下方面的能力：（i）识别语言名称与语言代码；（ii）在零样本与少样本条件下；（iii）是否提供标签集。与经过微调的较小语言识别工具相比，ChatGPT的表现仍有差距。实证分析表明，在能够充分服务于多元语言社区之前，ChatGPT仍处于需要持续优化的现实状态。