Embodied agents operating in smart homes must understand human behavior through diverse sensory inputs and communicate via natural language. While Vision-Language Models (VLMs) have enabled impressive language-grounded perception, their reliance on visual data limits robustness in real-world scenarios with occlusions, poor lighting, or privacy constraints. In this paper, we introduce HoloLLM, a Multimodal Large Language Model (MLLM) that integrates uncommon but powerful sensing modalities, such as LiDAR, infrared, mmWave radar, and WiFi, to enable seamless human perception and reasoning across heterogeneous environments. We address two key challenges: (1) the scarcity of aligned modality-text data for rare sensors, and (2) the heterogeneity of their physical signal representations. To overcome these, we design a Universal Modality-Injection Projector (UMIP) that enhances pre-aligned modality embeddings with fine-grained, text-aligned features from tailored encoders via coarse-to-fine cross-attention without introducing significant alignment overhead. We further introduce a human-VLM collaborative data curation pipeline to generate paired textual annotations for sensing datasets. Extensive experiments on two newly constructed benchmarks show that HoloLLM significantly outperforms existing MLLMs, improving language-grounded human sensing accuracy by up to 30%. This work establishes a new foundation for real-world, language-informed multisensory embodied intelligence.

翻译：在智能家居中运行的具身智能体必须通过多样化的感官输入理解人类行为，并通过自然语言进行交流。尽管视觉-语言模型（VLMs）已实现了令人瞩目的语言驱动感知能力，但其对视觉数据的依赖限制了其在遮挡、光照不足或隐私受限等现实场景中的鲁棒性。本文提出HoloLLM，一种多模态大语言模型（MLLM），它集成了LiDAR、红外、毫米波雷达和WiFi等不常见但强大的传感模态，以实现跨异构环境的无缝人类感知与推理。我们解决了两个关键挑战：（1）稀有传感器模态与文本对齐数据的稀缺性；（2）其物理信号表征的异构性。为此，我们设计了一种通用模态注入投影器（UMIP），通过从定制编码器中提取细粒度文本对齐特征，并利用由粗到精的交叉注意力机制增强预对齐的模态嵌入，而无需引入显著的对齐开销。我们进一步提出一种人机协作的数据标注流程，为传感数据集生成配对的文本标注。在两个新构建的基准测试上的大量实验表明，HoloLLM显著优于现有MLLMs，将语言驱动的人类感知准确率最高提升30%。这项工作为现实世界中基于语言的多感官具身智能奠定了新的基础。