Quantification of behavior is critical in applications ranging from neuroscience, veterinary medicine and animal conservation efforts. A common key step for behavioral analysis is first extracting relevant keypoints on animals, known as pose estimation. However, reliable inference of poses currently requires domain knowledge and manual labeling effort to build supervised models. We present a series of technical innovations that enable a new method, collectively called SuperAnimal, to develop unified foundation models that can be used on over 45 species, without additional human labels. Concretely, we introduce a method to unify the keypoint space across differently labeled datasets (via our generalized data converter) and for training these diverse datasets in a manner such that they don't catastrophically forget keypoints given the unbalanced inputs (via our keypoint gradient masking and memory replay approaches). These models show excellent performance across six pose benchmarks. Then, to ensure maximal usability for end-users, we demonstrate how to fine-tune the models on differently labeled data and provide tooling for unsupervised video adaptation to boost performance and decrease jitter across frames. If the models are fine-tuned, we show SuperAnimal models are 10-100$\times$ more data efficient than prior transfer-learning-based approaches. We illustrate the utility of our models in behavioral classification in mice and gait analysis in horses. Collectively, this presents a data-efficient solution for animal pose estimation.

翻译：行为量化在神经科学、兽医医学及动物保护等应用中至关重要。行为分析的一个常见关键步骤是首先提取动物的相关关键点（即姿态估计）。然而，当前可靠地推断姿态需要领域知识和人工标注来构建监督模型。本文提出一系列技术创新，统称为SuperAnimal方法，可开发适用于超过45个物种的统一基础模型，无需额外人工标注。具体而言，我们引入一种方法（通过广义数据转换器）统一跨不同标注数据集的关键点空间，并采用关键点梯度掩蔽与记忆重放策略训练这些多样化数据集，使其在输入不平衡时不会灾难性遗忘关键点。这些模型在六个姿态基准测试中展现出卓越性能。为确保最终用户的最大可用性，我们进一步展示如何在不同标注数据上微调模型，并提供无监督视频适应性工具以提升性能并减少帧间抖动。经微调后，SuperAnimal模型的数据效率比以往基于迁移学习的方法高10-100倍。我们通过小鼠行为分类和马步态分析验证了模型的实用性。综上，本研究为动物姿态估计提供了一种数据高效的解决方案。