Self-supervised learning has emerged as a key approach for learning generic representations from speech data. Despite promising results in downstream tasks such as speech recognition, speaker verification, and emotion recognition, a significant number of parameters is required, which makes fine-tuning for each task memory-inefficient. To address this limitation, we introduce ELP-adapter tuning, a novel method for parameter-efficient fine-tuning using three types of adapter, namely encoder adapters (E-adapters), layer adapters (L-adapters), and a prompt adapter (P-adapter). The E-adapters are integrated into transformer-based encoder layers and help to learn fine-grained speech representations that are effective for speech recognition. The L-adapters create paths from each encoder layer to the downstream head and help to extract non-linguistic features from lower encoder layers that are effective for speaker verification and emotion recognition. The P-adapter appends pseudo features to CNN features to further improve effectiveness and efficiency. With these adapters, models can be quickly adapted to various speech processing tasks. Our evaluation across four downstream tasks using five backbone models demonstrated the effectiveness of the proposed method. With the WavLM backbone, its performance was comparable to or better than that of full fine-tuning on all tasks while requiring 90% fewer learnable parameters.

翻译：自监督学习已成为从语音数据中学习通用表征的关键方法。尽管在语音识别、说话人验证和情感识别等下游任务中取得了有前景的结果，但该方法需要大量参数，导致针对每个任务进行微调时内存效率低下。为克服这一限制，我们提出了ELP-适配器调优方法——一种利用三种适配器实现参数高效微调的新颖方法，包括编码器适配器（E-adapters）、层级适配器（L-adapters）以及提示适配器（P-adapter）。E-adapters集成于基于Transformer的编码器层中，有助于学习对语音识别有效的细粒度语音表征；L-adapters构建从各编码器层至下游任务头的路径，有助于从底层编码器提取对说话人验证和情感识别有效的非语言特征；P-adapter通过向CNN特征附加伪特征进一步提升效能与效率。借助这些适配器，模型能够快速适应多种语音处理任务。我们在四项下游任务中使用五种骨干模型的评估结果验证了所提方法的有效性。以WavLM为骨干网络时，该方法在所有任务上的性能与全参数微调相当或更优，同时可学习参数减少了90%。