Traditional human-in-the-loop-based annotation for time-series data like inertial data often requires access to alternate modalities like video or audio from the environment. These alternate sources provide the necessary information to the human annotator, as the raw numeric data is often too obfuscated even for an expert. However, this traditional approach has many concerns surrounding overall cost, efficiency, storage of additional modalities, time, scalability, and privacy. Interestingly, recent large language models (LLMs) are also trained with vast amounts of publicly available alphanumeric data, which allows them to comprehend and perform well on tasks beyond natural language processing. Naturally, this opens up a potential avenue to explore LLMs as virtual annotators where the LLMs will be directly provided the raw sensor data for annotation instead of relying on any alternate modality. Naturally, this could mitigate the problems of the traditional human-in-the-loop approach. Motivated by this observation, we perform a detailed study in this paper to assess whether the state-of-the-art (SOTA) LLMs can be used as virtual annotators for labeling time-series physical sensing data. To perform this in a principled manner, we segregate the study into two major phases. In the first phase, we investigate the challenges an LLM like GPT-4 faces in comprehending raw sensor data. Considering the observations from phase 1, in the next phase, we investigate the possibility of encoding the raw sensor data using SOTA SSL approaches and utilizing the projected time-series data to get annotations from the LLM. Detailed evaluation with four benchmark HAR datasets shows that SSL-based encoding and metric-based guidance allow the LLM to make more reasonable decisions and provide accurate annotations without requiring computationally expensive fine-tuning or sophisticated prompt engineering.

翻译：传统基于人机协同的时间序列数据（如惯性数据）标注方法通常需要依赖环境中的视频或音频等替代模态。这些替代源为人工标注员提供必要信息，因为原始数值数据即使对专家而言也往往难以解读。然而，这种传统方法在总体成本、效率、额外模态数据的存储、时间消耗、可扩展性及隐私保护等方面存在诸多问题。值得注意的是，当前大语言模型（LLMs）通过海量公开的字母数字数据训练，使其具备理解并执行自然语言处理以外任务的能力。这自然开辟了探索将LLMs作为虚拟标注器的潜在途径——直接向LLMs提供原始传感器数据进行标注，而无需依赖任何替代模态。此举有望缓解传统人机协同方法的诸多问题。受此启发，本文开展系统性研究，评估当前最优（SOTA）LLMs能否作为虚拟标注器用于时间序列物理传感数据的标注。为进行规范研究，我们将研究分为两个主要阶段：第一阶段，探究GPT-4等LLMs在理解原始传感器数据时面临的挑战；第二阶段，基于第一阶段发现，研究采用SOTA自监督学习（SSL）方法编码原始传感器数据，并利用投影后的时间序列数据从LLMs获取标注的可行性。对四个标准HAR基准数据集的详细评估表明，基于SSL的编码和度量引导策略能使LLMs做出更合理决策，在不需计算密集型微调或复杂提示工程的情况下提供精确标注。