Human-centric perception tasks, e.g., pedestrian detection, skeleton-based action recognition, and pose estimation, have wide industrial applications, such as metaverse and sports analysis. There is a recent surge to develop human-centric foundation models that can benefit a broad range of human-centric perception tasks. While many human-centric foundation models have achieved success, they did not explore 3D and vision-language tasks for human-centric and required task-specific finetuning. These limitations restrict their application to more downstream tasks and situations. To tackle these problems, we present Hulk, the first multimodal human-centric generalist model, capable of addressing 2D vision, 3D vision, skeleton-based, and vision-language tasks without task-specific finetuning. The key to achieving this is condensing various task-specific heads into two general heads, one for discrete representations, e.g., languages, and the other for continuous representations, e.g., location coordinates. The outputs of two heads can be further stacked into four distinct input and output modalities. This uniform representation enables Hulk to treat diverse human-centric tasks as modality translation, integrating knowledge across a wide range of tasks. Comprehensive evaluations of Hulk on 12 benchmarks covering 8 human-centric tasks demonstrate the superiority of our proposed method, achieving state-of-the-art performance in 11 benchmarks. The code is available on https://github.com/OpenGVLab/Hulk.

翻译：人类中心感知任务（如行人检测、基于骨骼的动作识别和姿态估计）在元宇宙、体育分析等工业领域具有广泛应用。近年来，能够惠及广泛人类中心感知任务的基础模型开发掀起热潮。尽管众多人类中心基础模型已取得显著成效，但仍存在两方面局限：一是未探索3D与视觉语言任务，二是需要针对特定任务进行微调。这些限制阻碍了其在下游任务及场景中的推广应用。为解决上述问题，我们提出首个多模态人类中心通才模型——浩克，该模型无需任务特定微调即可处理2D视觉、3D视觉、骨骼分析及视觉语言四大类任务。实现这一突破的关键在于将各类任务专用解码器精简为两个通用解码器：一个负责离散表征（如语言），另一个负责连续表征（如位置坐标）。两个解码器的输出可进一步组合为四种不同的输入输出模态。这种统一表征使浩克能够将多样化的人类中心任务转化为模态翻译问题，从而整合跨任务知识。在覆盖8项人类中心任务的12个基准测试中，浩克展现出卓越性能，其中11项达到业界最优水平。相关代码已在https://github.com/OpenGVLab/Hulk 开源。