Language development experts need tools that can automatically identify languages from fluent, conversational speech, and provide reliable estimates of usage rates at the level of an individual recording. However, language identification systems are typically evaluated on metrics such as equal error rate and balanced accuracy, applied at the level of an entire speech corpus. These overview metrics do not provide information about model performance at the level of individual speakers, recordings, or units of speech with different linguistic characteristics. Overview statistics may therefore mask systematic errors in model performance for some subsets of the data, and consequently, have worse performance on data derived from some subsets of human speakers, creating a kind of algorithmic bias. In the current paper, we investigate how well a number of language identification systems perform on individual recordings and speech units with different linguistic properties in the MERLIon CCS Challenge. The Challenge dataset features accented English-Mandarin code-switched child-directed speech.

翻译：语言发展专家需要能够自动从流利对话语音中识别语言，并提供个体录音层面上使用率的可靠估算工具。然而，语言识别系统的评估通常采用等错误率和均衡准确率等指标，且应用于整个语音语料库的层面。这些概览性指标无法提供关于模型在个体说话者、录音或具有不同语言特征的语音单元层面上的性能信息。因此，概览统计可能掩盖模型对某些数据子集的系统性错误，从而导致在某些人类说话者子集的数据上表现更差，形成一种算法偏见。本文旨在探究MERLIon CCS挑战赛中多个语言识别系统在具有不同语言属性的个体录音和语音单元上的表现。该挑战赛的数据集包含带有口音的英汉混合语儿童导向语音。