Speech is the surface form of a finite set of phonetic units, which can be represented by discrete codes. We propose the Code BERT (CoBERT) approach for self-supervised speech representation learning. The idea is to convert an utterance to a sequence of discrete codes, and perform code representation learning, where we predict the code representations based on a masked view of the original speech input. Unlike the prior self-distillation approaches of which the teacher and the student are of the same modality, our target model predicts representations from a different modality. CoBERT outperforms the most recent state-of-the-art performance on the ASR task and brings significant improvements on the SUPERB speech translation (ST) task. Our code and models are released at https://github.com/mct10/CoBERT.

翻译：语音是有限音素集合的表层形式，可通过离散代码进行表示。我们提出Code BERT（CoBERT）方法用于自监督语音表示学习。其核心思想是将语音话语转换为离散代码序列，并执行代码表示学习——基于原始语音输入的掩码视图预测代码表示。不同于先前师生模型属于同模态的自蒸馏方法，我们的目标模型从不同模态预测表示。CoBERT在ASR任务上超越最新最优性能，并在SUPERB语音翻译（ST）任务中带来显著提升。我们的代码与模型已发布至https://github.com/mct10/CoBERT。