Learning grounded word meaning from natural experience requires resolving two ambiguities in infant-view recordings: when the named referent appears and where it is in a cluttered frame. In SAYCam-style data, caregiver speech is sparse and weakly synchronized with egocentric video, so single-frame contrastive pairing yields noisy positives in which the intended object is absent or entangled with distractors. We propose BabyMind, an object-first bias for child-view contrastive learning under sparse, noisy supervision. BabyMind extracts candidate object embeddings using an offline mask-based region interface, links candidates across a short utterance-centered window into lightweight object files via tracking, and aligns utterances to bags of object files with a prototype-space multiple-instance contrastive objective. Track-coherence and global-object agreement regularizers stabilize learning and transfer object-file structure into the global frame embedding used at evaluation. On SAYCam-S, BabyMind improves Labeled-S 15 forced-choice accuracy by +2.6 points over CVCL and yields consistent gains on in-vocabulary out-of-distribution benchmarks. Code is available at https://github.com/sathiiii/BabyMind.
翻译:从自然经验中学习 grounded 词义需解决婴儿视角记录中的两种歧义:命名指代对象何时出现,以及其在杂乱画面中的位置。在 SAYCam 风格数据中,看护者语言稀疏且与自我中心视频弱同步,因此单帧对比配对会产生含有缺失或受干扰目标对象的噪声正例。我们提出 BabyMind——一种在稀疏噪声监督下适用于儿童视角对比学习的物体优先归纳偏好方法。BabyMind 通过离线掩码区域接口提取候选物体嵌入,在短句为中心的窗口内通过追踪将候选对象关联为轻量级物体档案,并利用原型空间多实例对比目标将语句与物体档案袋对齐。追踪连贯性与全局物体一致性正则化项可稳定学习过程,并将物体档案结构迁移至评估所用的全局帧嵌入中。在 SAYCam-S 数据集上,BabyMind 在 Labeled-S 15 项强制选择准确率上较 CVCL 提升 2.6 个百分点,并在词表内分布外基准测试中取得一致增益。代码开源地址:https://github.com/sathiiii/BabyMind