English Natural Language Understanding (NLU) systems have achieved great performances and even outperformed humans on benchmarks like GLUE and SuperGLUE. However, these benchmarks contain only textbook Standard American English (SAE). Other dialects have been largely overlooked in the NLP community. This leads to biased and inequitable NLU systems that serve only a sub-population of speakers. To understand disparities in current models and to facilitate more dialect-competent NLU systems, we introduce the VernAcular Language Understanding Evaluation (VALUE) benchmark, a challenging variant of GLUE that we created with a set of lexical and morphosyntactic transformation rules. In this initial release (V.1), we construct rules for 11 features of African American Vernacular English (AAVE), and we recruit fluent AAVE speakers to validate each feature transformation via linguistic acceptability judgments in a participatory design manner. Experiments show that these new dialectal features can lead to a drop in model performance. To run the transformation code and download both synthetic and gold-standard dialectal GLUE benchmarks, see https://github.com/SALT-NLP/value

翻译：英国自然语言理解(NLUE)系统在GLUE和SuperGLUE等基准上取得了巨大的业绩,甚至比人类的成绩都高。然而,这些基准只包含教科书中的美国标准英语(SAE),其他方言在NLP社区中基本上被忽视。这导致了偏向和不公平的NLU系统,这些系统仅服务于一个亚人口。为了理解当前模式的差异,并促进更符合方言能力的NLU系统,我们引入了Vernaculal语言理解评价(VALUE)基准,这是我们用一套词汇和形态学转换规则创建的GLUE的具有挑战性的变体。在最初的版本(V.1)中,我们为非裔美国人通用英语(AAAAVEE)的11个特征制定了规则,我们招聘了流言者AVEAVE,通过参与性的语言可接受性判断来验证每个特性的转变。实验表明,这些新的方言词特征可以导致模型性能下降。运行转换代码并下载合成和金制方言方言的GLUE基准,见http://githubb.com/SALT-NLP/valual-de-d/valuew。