Despite advances in human activity recognition (HAR) with different modalities, a precise, robust, and accurate daily log system is not yet available. Current solutions primarily rely on controlled, lab-based data collection, which limits their real-world applicability. The challenges towards a fine-grained daily log are 1) contextual awareness, 2) spatial awareness, and 3) effective fusion of multi-modal sensor data. To solve them, we propose EgoLog, which integrates effective audio-IMU fusion for daily log with ubiquitous wearables. Our approach first fuses audio and IMU data from two perspectives: temporal understanding and spatial understanding. We extract scenario-level features and aggregate them in the time dimension, while using motion compensation to enhance the performance of sound source localization. The knowledge obtained from these steps is then integrated into a multi-modal HAR framework. Here, the scenario provides prior knowledge, and the spatial location helps differentiate the user from the background. Furthermore, we integrate a LLM to enhance scenario recognition through logical reasoning. The knowledge derived from the LLM is subsequently transferred back to the local device to enable efficient, on-device inference. Evaluated on both public and self-collected dataset, EgoLog achieves effective multimodal fusion for both activity and scenraio recognition, outperforms the baseline by 12% and 15%, respectively.

翻译：尽管多模态人类活动识别（HAR）技术已取得进展，但目前仍缺乏一个精确、鲁棒且准确的日常日志系统。现有解决方案主要依赖于受控的实验室数据采集，这限制了其在实际场景中的应用。实现细粒度日常日志面临的挑战包括：1）情境感知，2）空间感知，以及3）多模态传感器数据的有效融合。为解决这些问题，我们提出了EgoLog，该系统通过有效的音频-IMU融合技术，利用普适可穿戴设备实现日常日志记录。我们的方法首先从两个维度融合音频与IMU数据：时序理解与空间理解。我们提取场景级特征并在时间维度进行聚合，同时利用运动补偿增强声源定位性能。随后，将上述步骤获得的知识整合到多模态HAR框架中。在此框架中，场景信息提供先验知识，空间定位则有助于区分用户与背景。此外，我们集成大型语言模型（LLM），通过逻辑推理增强场景识别能力。LLM推导出的知识随后回传至本地设备，以实现高效的端侧推理。在公开数据集与自采集数据集上的评估表明，EgoLog在活动识别与场景识别任务中均实现了有效的多模态融合，其性能分别超越基线方法12%与15%。