We present a large-scale study of linguistic bias exhibited by ChatGPT covering ten dialects of English (Standard American English, Standard British English, and eight widely spoken non-"standard" varieties from around the world). We prompted GPT-3.5 Turbo and GPT-4 with text by native speakers of each variety and analyzed the responses via detailed linguistic feature annotation and native speaker evaluation. We find that the models default to "standard" varieties of English; based on evaluation by native speakers, we also find that model responses to non-"standard" varieties consistently exhibit a range of issues: lack of comprehension (10% worse compared to "standard" varieties), stereotyping (16% worse), demeaning content (22% worse), and condescending responses (12% worse). We also find that if these models are asked to imitate the writing style of prompts in non-"standard" varieties, they produce text that exhibits lower comprehension of the input and is especially prone to stereotyping. GPT-4 improves on GPT-3.5 in terms of comprehension, warmth, and friendliness, but it also results in a marked increase in stereotyping (+17%). The results suggest that GPT-3.5 Turbo and GPT-4 exhibit linguistic discrimination in ways that can exacerbate harms for speakers of non-"standard" varieties.

翻译：我们开展了一项大规模研究，考察ChatGPT在十种英语方言（标准美国英语、标准英国英语以及全球八种广泛使用的非“标准”变体）中表现出的语言偏见。我们使用各种方言母语者的文本提示GPT-3.5 Turbo和GPT-4，并通过详细的语言特征标注与母语者评估分析模型响应。研究发现，模型默认采用英语“标准”变体；根据母语者评估，模型对非“标准”变体的响应持续表现出多重问题：理解力不足（较“标准”变体低10%）、刻板印象（低16%）、贬损性内容（低22%）以及居高临下的回应（低12%）。研究还发现，若要求模型模仿非“标准”变体提示文本的写作风格，其生成文本会表现出对输入内容的理解力下降，且尤其容易产生刻板印象。GPT-4在理解力、亲和力与友好度方面较GPT-3.5有所改善，但同时也导致刻板印象显著增加（+17%）。结果表明，GPT-3.5 Turbo和GPT-4表现出的语言歧视可能加剧对非“标准”变体使用者的伤害。