Language identification is an important first step in many IR and NLP applications. Most publicly available language identification datasets, however, are compiled under the assumption that the gold label of each instance is determined by where texts are retrieved from. Research has shown that this is a problematic assumption, particularly in the case of very similar languages (e.g., Croatian and Serbian) and national language varieties (e.g., Brazilian and European Portuguese), where texts may contain no distinctive marker of the particular language or variety. To overcome this important limitation, this paper presents DSL True Labels (DSL-TL), the first human-annotated multilingual dataset for language variety identification. DSL-TL contains a total of 12,900 instances in Portuguese, split between European Portuguese and Brazilian Portuguese; Spanish, split between Argentine Spanish and Castilian Spanish; and English, split between American English and British English. We trained multiple models to discriminate between these language varieties, and we present the results in detail. The data and models presented in this paper provide a reliable benchmark toward the development of robust and fairer language variety identification systems. We make DSL-TL freely available to the research community.

翻译：语言识别是许多信息检索和自然语言处理应用中的重要第一步。然而，大多数公开可用的语言识别数据集都是在假设每个实例的黄金标签由文本的检索来源决定的条件下编制的。研究表明，这一假设存在问题，尤其是在高度相似的语言（如克罗地亚语和塞尔维亚语）以及国家语言变体（如巴西葡萄牙语和欧洲葡萄牙语）的情况下，这些文本可能不包含特定语言或变体的独特标记。为克服这一重要局限性，本文提出了DSL True Labels（DSL-TL），这是首个用于语言变体识别的人工标注多语言数据集。DSL-TL共包含12,900个实例，涵盖葡萄牙语（分为欧洲葡萄牙语和巴西葡萄牙语）、西班牙语（分为阿根廷西班牙语和卡斯蒂利亚西班牙语）以及英语（分为美国英语和英国英语）。我们训练了多个模型来区分这些语言变体，并详细呈现了结果。本文提供的数据和模型为开发稳健且更公平的语言变体识别系统奠定了可靠的基准。我们将DSL-TL免费提供给研究社区。