To perform automatic family audio analysis, past studies have collected recordings using phone, video, or audio-only recording devices like LENA, investigated supervised learning methods, and used or fine-tuned general-purpose embeddings learned from large pretrained models. In this study, we advance the audio component of a new infant wearable multi-modal device called LittleBeats (LB) by learning family audio representation via wav2vec 2.0 (W2V2) pertaining. We show given a limited number of labeled LB home recordings, W2V2 pretrained using 1k-hour of unlabeled home recordings outperforms oracle W2V2 pretrained on 52k-hour unlabeled audio in terms of parent/infant speaker diarization (SD) and vocalization classifications (VC) at home. Extra relevant external unlabeled and labeled data further benefit W2V2 pretraining and fine-tuning. With SpecAug and environmental speech corruptions, we obtain 12% relative gain on SD and moderate boost on VC. Code and model weights are available.

翻译：为进行自动家庭音频分析，以往研究通过电话、视频或LENA等纯音频录制设备采集录音，采用监督学习方法，并利用或微调从大规模预训练模型中提取的通用特征嵌入。本研究通过wav2vec 2.0（W2V2）预训练学习家庭音频表征，推进了一款名为LittleBeats（LB）的新型婴儿可穿戴多模态设备的音频模块。研究表明，在标注LB家庭录音数量有限的情况下，使用1千小时无标签家庭录音预训练的W2V2模型，在家庭环境下的父母/婴儿说话人分离（SD）和发声分类（VC）任务上，优于基于5.2万小时无标签音频预训练的Oracle W2V2模型。额外引入相关外部无标签和标签数据可进一步优化W2V2预训练与微调。结合SpecAug和语音环境噪声扰动，我们在SD任务上获得12%的相对性能提升，并在VC任务上获得适度改进。代码与模型权重已公开提供。