For many real-world applications, the user-generated inputs usually contain various noises due to speech recognition errors caused by linguistic variations1 or typographical errors (typos). Thus, it is crucial to test model performance on data with realistic input noises to ensure robustness and fairness. However, little study has been done to construct such benchmarks for Chinese, where various language-specific input noises happen in the real world. In order to fill this important gap, we construct READIN: a Chinese multi-task benchmark with REalistic And Diverse Input Noises. READIN contains four diverse tasks and requests annotators to re-enter the original test data with two commonly used Chinese input methods: Pinyin input and speech input. We designed our annotation pipeline to maximize diversity, for example by instructing the annotators to use diverse input method editors (IMEs) for keyboard noises and recruiting speakers from diverse dialectical groups for speech noises. We experiment with a series of strong pretrained language models as well as robust training methods, we find that these models often suffer significant performance drops on READIN even with robustness methods like data augmentation. As the first large-scale attempt in creating a benchmark with noises geared towards user-generated inputs, we believe that READIN serves as an important complement to existing Chinese NLP benchmarks. The source code and dataset can be obtained from https://github.com/thunlp/READIN.

翻译：在许多实际应用中，用户生成的输入通常包含由语言变异导致的语音识别错误或拼写错误（typos）等多种噪声。因此，在带有真实输入噪声的数据上测试模型性能，对于确保其鲁棒性和公平性至关重要。然而，目前针对中文构建此类基准的研究很少，而现实中中文特有的输入噪声类型多样。为填补这一重要空白，我们构建了READIN：一个包含真实且多样输入噪声的中文多任务基准。READIN涵盖四项不同任务，要求标注人员使用两种常见中文输入法（拼音输入和语音输入）重新输入原始测试数据。我们设计了最大化多样性的标注流程，例如指导标注人员使用不同的输入法编辑器（IME）生成键盘噪声，并招募来自不同方言群体的说话者生成语音噪声。我们在一系列强预训练语言模型及鲁棒训练方法上进行了实验，发现即使采用数据增强等鲁棒性方法，这些模型在READIN上的性能仍显著下降。作为首个大规模构建面向用户生成输入噪声基准的尝试，我们认为READIN是对现有中文NLP基准的重要补充。源代码与数据集可从https://github.com/thunlp/READIN获取。