Multilingual speech foundation models such as Whisper are trained on web-scale data, where data for each language consists of a myriad of regional varieties. However, different regional varieties often employ different scripts to write the same language, rendering speech recognition output also subject to non-determinism in the output script. To mitigate this problem, we show that script is linearly encoded in the activation space of multilingual speech models, and that modifying activations at inference time enables direct control over output script. We find the addition of such script vectors to activations at test time can induce script change even in unconventional language-script pairings (e.g. Italian in Cyrillic and Japanese in Latin script). We apply this approach to inducing post-hoc control over the script of speech recognition output, where we observe competitive performance across all model sizes of Whisper.

翻译：诸如Whisper等多语言语音基础模型基于网络规模数据进行训练，其中每种语言的数据包含大量地域变体。然而，不同地域变体常使用不同文字系统书写同一语言，导致语音识别输出在文字形式上存在非确定性。为缓解此问题，我们证明文字信息在多语言语音模型的激活空间中呈线性编码，通过在推理时修正激活值可直接控制输出文字形式。研究发现，在测试阶段向激活值添加此类文字向量，即使在非常规的语言-文字配对中（例如西里尔字母书写的意大利语和拉丁字母书写的日语）也能诱发文字转换。我们将此方法应用于语音识别输出的后置文字控制，在Whisper所有模型规模上均观察到具有竞争力的性能表现。