Self-supervised learning (SSL) models confront challenges of abrupt informational collapse or slow dimensional collapse. We propose TriNet, which introduces a novel triple-branch architecture for preventing collapse and stabilizing the pre-training. TriNet learns the SSL latent embedding space and incorporates it to a higher level space for predicting pseudo target vectors generated by a frozen teacher. Our experimental results show that the proposed method notably stabilizes and accelerates pre-training and achieves a relative word error rate reduction (WERR) of 6.06% compared to the state-of-the-art (SOTA) Data2vec for a downstream benchmark ASR task. We will release our code at https://github.com/tencent-ailab/.

翻译：自监督学习（SSL）模型面临信息突然坍缩或缓慢维度坍缩的挑战。我们提出TriNet，其引入新颖的三分支架构以防止坍缩并稳定预训练过程。TriNet学习SSL潜在嵌入空间，并将其融入更高层级空间以预测由冻结教师网络生成的伪目标向量。实验结果表明，所提方法显著稳定并加速了预训练，在下游基准自动语音识别（ASR）任务中，相较于当前最优的（SOTA）Data2vec实现了6.06%的相对词错误率降低（WERR）。我们将于https://github.com/tencent-ailab/发布代码。