To perform automatic family audio analysis, past studies have collected recordings using phone, video, or audio-only recording devices like LENA, investigated supervised learning methods, and used or fine-tuned general-purpose embeddings learned from large pretrained models. In this study, we advance the audio component of a new infant wearable multi-modal device called LittleBeats (LB) by learning family audio representation via wav2vec 2.0 (W2V2) pertaining. We show given a limited number of labeled LB home recordings, W2V2 pretrained using 1k-hour of unlabeled home recordings outperforms oracle W2V2 pretrained on 52k-hour unlabeled audio in terms of parent/infant speaker diarization (SD) and vocalization classifications (VC) at home. Extra relevant external unlabeled and labeled data further benefit W2V2 pretraining and fine-tuning. With SpecAug and environmental speech corruptions, we obtain 12% relative gain on SD and moderate boost on VC. Code and model weights are available.

翻译：为进行自动家庭音频分析，以往研究通过手机、视频或纯音频录音设备（如LENA）采集数据，探索监督学习方法，并利用或微调从大型预训练模型中习得的通用嵌入。本研究通过wav2vec 2.0（W2V2）预训练学习家庭音频表征，推进了新型婴儿可穿戴多模态设备LittleBeats（LB）的音频模块。结果显示，在LB家庭录音标签数量有限的情况下，使用1000小时无标签家庭录音预训练的W2V2模型，在家中的父母/婴儿说话人日志（SD）和发声分类（VC）任务上优于使用52000小时无标签音频预训练的最优W2V2模型。额外引入的相关外部无标签与标签数据进一步提升了W2V2的预训练和微调效果。结合SpecAug和环境语音扰动，我们在SD任务上获得12%的相对增益，并在VC任务上实现适度提升。代码与模型权重均已开源。