We often represent text using Unicode formats (UTF-8 and UTF-16). The UTF-8 format is increasingly popular, especially on the web (XML, HTML, JSON, Rust, Go, Swift, Ruby). The UTF-16 format is most common in Java, .NET, and inside operating systems such as Windows. Software systems frequently have to convert text from one Unicode format to the other. While recent disks have bandwidths of 5 GiB/s or more, conventional approaches transcode non-ASCII text at a fraction of a gigabyte per second. We show that we can validate and transcode Unicode text at gigabytes per second on current systems (x64 and ARM) without sacrificing safety. Our open-source library can be ten times faster than the popular ICU library on non-ASCII strings and even faster on ASCII strings.

翻译：我们通常使用Unicode格式（UTF-8和UTF-16）表示文本。UTF-8格式日益普及，尤其在Web领域（XML、HTML、JSON、Rust、Go、Swift、Ruby），而UTF-16格式在Java、.NET以及Windows等操作系统中最为常见。软件系统时常需要将文本从一种Unicode格式转换为另一种。尽管当前磁盘带宽已达5 GiB/s以上，但传统方法处理非ASCII文本时每秒仅能处理数分之一吉字节。我们证明，在现有系统（x64和ARM架构）上，无需牺牲安全性即可实现每秒吉字节级别的Unicode验证与转码。我们的开源库在处理非ASCII字符串时速度可达广受欢迎的ICU库的十倍，在处理ASCII字符串时甚至更快。