Modern large language models demonstrate impressive capabilities in text generation and generalization. However, they often struggle with solving text editing tasks, particularly when it comes to correcting spelling errors and mistypings. In this paper, we present a methodology for generative spelling correction (SC), which was tested on English and Russian languages and potentially can be extended to any language with minor changes. Our research mainly focuses on exploring natural spelling errors and mistypings in texts and studying the ways those errors can be emulated in correct sentences to effectively enrich generative models' pre-train procedure. We investigate the impact of such emulations and the models' abilities across different text domains. In this work, we investigate two spelling corruption techniques: 1) first one mimics human behavior when making a mistake through leveraging statistics of errors from particular dataset and 2) second adds the most common spelling errors, keyboard miss clicks, and some heuristics within the texts. We conducted experiments employing various corruption strategies, models' architectures and sizes on the pre-training and fine-tuning stages and evaluated the models using single-domain and multi-domain test sets. As a practical outcome of our work, we introduce SAGE (Spell checking via Augmentation and Generative distribution Emulation) is a library for automatic generative SC that includes a family of pre-trained generative models and built-in augmentation algorithms.

翻译：现代大型语言模型在文本生成和泛化方面展现出令人印象深刻的能力。然而，它们在处理文本编辑任务时往往表现不佳，尤其是在纠正拼写错误和打字错误方面。在本文中，我们提出了一种用于生成式拼写校正（SC）的方法，该方法在英语和俄语上进行了测试，并有望通过微小改动扩展到任何语言。我们的研究主要聚焦于探索文本中的自然拼写错误和打字错误，以及研究如何在正确的句子中模拟这些错误，从而有效丰富生成模型的预训练过程。我们研究了此类模拟的影响以及模型在不同文本领域中的能力。在本工作中，我们探讨了两种拼写损坏技术：1）第一种通过利用特定数据集的错误统计模拟人类犯错行为；2）第二种在文本中引入最常见的拼写错误、键盘误击和一些启发式规则。我们采用多种损坏策略、模型架构和规模在预训练和微调阶段进行了实验，并使用单领域和多领域测试集评估了模型。作为本工作的实际成果，我们引入了SAGE（通过增强和生成式分布模拟进行拼写检查）——一个用于自动生成式拼写校正的库，该库包含一系列预训练生成模型和内建增强算法。