This demo presents a novel end-to-end framework that combines on-device large language models (LLMs) with smartphone sensing technologies to achieve context-aware and personalized services. The framework addresses critical limitations of current personalization solutions via cloud-based LLMs, such as privacy concerns, latency and cost, and limited personal sensor data. To achieve this, we innovatively proposed deploying LLMs on smartphones with multimodal sensor data and customized prompt engineering, ensuring privacy and enhancing personalization performance through context-aware sensing. A case study involving a university student demonstrated the proposed framework's capability to provide tailored recommendations. In addition, we show that the proposed framework achieves the best trade-off in privacy, performance, latency, cost, battery and energy consumption between on-device and cloud LLMs. Future work aims to integrate more diverse sensor data and conduct large-scale user studies to further refine the personalization. We envision the proposed framework could significantly improve user experiences in various domains such as healthcare, productivity, and entertainment by providing secure, context-aware, and efficient interactions directly on users' devices.

翻译：本演示提出了一种新颖的端到端框架，该框架将设备端大型语言模型与智能手机感知技术相结合，以实现情境感知和个性化服务。该框架解决了当前基于云端LLM的个性化解决方案的关键局限性，例如隐私问题、延迟与成本，以及有限的个人传感器数据。为此，我们创新性地提出在智能手机上部署LLM，并利用多模态传感器数据和定制化的提示工程，通过情境感知确保隐私并提升个性化性能。一项涉及大学生的案例研究展示了所提框架提供定制化推荐的能力。此外，我们表明所提框架在设备端与云端LLM之间，就隐私、性能、延迟、成本、电池及能耗方面实现了最佳权衡。未来的工作旨在整合更多样化的传感器数据，并进行大规模用户研究，以进一步完善个性化。我们设想，所提框架能够通过在用户设备上直接提供安全、情境感知且高效的交互，显著改善医疗保健、生产力和娱乐等多个领域的用户体验。