We introduce UbiPhysio, a milestone framework that delivers fine-grained action description and feedback in natural language to support people's daily functioning, fitness, and rehabilitation activities. This expert-like capability assists users in properly executing actions and maintaining engagement in remote fitness and rehabilitation programs. Specifically, the proposed UbiPhysio framework comprises a fine-grained action descriptor and a knowledge retrieval-enhanced feedback module. The action descriptor translates action data, represented by a set of biomechanical movement features we designed based on clinical priors, into textual descriptions of action types and potential movement patterns. Building on physiotherapeutic domain knowledge, the feedback module provides clear and engaging expert feedback. We evaluated UbiPhysio's performance through extensive experiments with data from 104 diverse participants, collected in a home-like setting during 25 types of everyday activities and exercises. We assessed the quality of the language output under different tuning strategies using standard benchmarks. We conducted a user study to gather insights from clinical physiotherapists and potential users about our framework. Our initial tests show promise for deploying UbiPhysio in real-life settings without specialized devices.

翻译：我们提出了UbiPhysio，一个里程碑式的框架，能够以自然语言提供精细的动作描述与反馈，以支持人们的日常功能、健身和康复活动。这一类似专家级别的能力有助于用户正确执行动作，并保持对远程健身与康复项目的参与。具体而言，所提出的UbiPhysio框架包含一个精细动作描述器和一个基于知识检索增强的反馈模块。动作描述器将由我们基于临床先验设计的一组生物力学运动特征所表示的动作数据，转化为动作类型与潜在运动模式的文本描述。基于物理治疗领域知识，反馈模块提供清晰且引人入胜的专家级反馈。我们通过涉及104名不同参与者的数据进行了广泛实验评估UbiPhysio的性能，数据收集于类似家庭的环境中，涵盖25种日常活动与锻炼。我们使用标准基准评估了在不同调优策略下语言输出的质量。还开展了一项用户研究，以收集临床物理治疗师及潜在用户对框架的见解。初步测试表明，在无需专用设备的情况下，UbiPhysio可在现实环境中进行部署应用。