Numerous methods have been proposed to detect, estimate, and analyze properties of people in images, including the estimation of 3D pose, shape, contact, human-object interaction, emotion, and more. Each of these methods works in isolation instead of synergistically. Here we address this problem and build a language-driven human understanding system -- ChatHuman, which combines and integrates the skills of many different methods. To do so, we finetune a Large Language Model (LLM) to select and use a wide variety of existing tools in response to user inputs. In doing so, ChatHuman is able to combine information from multiple tools to solve problems more accurately than the individual tools themselves and to leverage tool output to improve its ability to reason about humans. The novel features of ChatHuman include leveraging academic publications to guide the application of 3D human-related tools, employing a retrieval-augmented generation model to generate in-context-learning examples for handling new tools, and discriminating and integrating tool results to enhance 3D human understanding. Our experiments show that ChatHuman outperforms existing models in both tool selection accuracy and performance across multiple 3D human-related tasks. ChatHuman is a step towards consolidating diverse methods for human analysis into a single, powerful, system for 3D human reasoning.

翻译：现有大量方法被提出用于检测、估计和分析图像中人物的属性，包括三维姿态、形状、接触、人-物交互、情绪等。但这些方法各自独立运行，缺乏协同性。针对这一问题，我们构建了语言驱动的人体理解系统——ChatHuman，它能够整合多种不同方法的技术能力。为此，我们微调了一个大型语言模型（LLM），使其能够根据用户输入选择和运用一系列现有工具。通过这种方式，ChatHuman能够融合多个工具的信息，从而比单一工具更准确地解决问题，并利用工具输出提升自身对人体推理的能力。ChatHuman的创新点包括：利用学术文献指导三维人体相关工具的应用，采用检索增强生成模型为处理新工具生成上下文学习样例，以及通过区分和整合工具结果来增强三维人体理解。实验表明，ChatHuman在工具选择准确性和多项三维人体相关任务性能上均优于现有模型。ChatHuman是将多样化人体分析方法整合为一个强大统一的三维人体推理系统的重要探索。