Passive tracking methods, such as phone and wearable sensing, have become dominant in monitoring human behaviors in modern ubiquitous computing studies. While there have been significant advances in machine-learning approaches to translate periods of raw sensor data to model momentary behaviors, (e.g., physical activity recognition), there still remains a significant gap in the translation of these sensing streams into meaningful, high-level, context-aware insights that are required for various applications (e.g., summarizing an individual's daily routine). To bridge this gap, experts often need to employ a context-driven sensemaking process in real-world studies to derive insights. This process often requires manual effort and can be challenging even for experienced researchers due to the complexity of human behaviors. We conducted three rounds of user studies with 21 experts to explore solutions to address challenges with sensemaking. We follow a human-centered design process to identify needs and design, iterate, build, and evaluate Vital Insight (VI), a novel, LLM-assisted, prototype system to enable human-in-the-loop inference (sensemaking) and visualizations of multi-modal passive sensing data from smartphones and wearables. Using the prototype as a technology probe, we observe experts' interactions with it and develop an expert sensemaking model that explains how experts move between direct data representations and AI-supported inferences to explore, question, and validate insights. Through this iterative process, we also synthesize and discuss a list of design implications for the design of future AI-augmented visualization systems to better assist experts' sensemaking processes in multi-modal health sensing data.

翻译：被动追踪方法（如手机与可穿戴设备传感）已成为现代泛在计算研究中监测人类行为的主流手段。尽管机器学习方法在将原始传感器数据片段转化为瞬时行为建模（例如，身体活动识别）方面取得了显著进展，但将这些传感流转化为各类应用所需的有意义、高层次、情境感知的洞察（例如，总结个人日常作息）仍存在巨大差距。为弥合此差距，专家在实际研究中常需采用情境驱动的意义建构过程以获取洞察。该过程通常需人工介入，且因人类行为的复杂性，即使对经验丰富的研究者而言亦具挑战性。我们与21位专家进行了三轮用户研究，以探索应对意义建构挑战的解决方案。遵循以人为中心的设计流程，我们识别需求并设计、迭代、构建与评估了Vital Insight（VI）——一个新颖的、大语言模型辅助的原型系统，旨在实现对智能手机与可穿戴设备多模态被动传感数据的人机协同推理（意义建构）与可视化。通过将该原型作为技术探针，我们观察专家与其的交互，并构建了一个专家意义建构模型，阐释专家如何在直接数据表征与AI辅助推理之间切换，以探索、质疑与验证洞察。通过此迭代过程，我们进一步综合并讨论了一系列设计启示，为未来AI增强的可视化系统设计提供参考，以更好地辅助专家在多模态健康传感数据中的意义建构过程。