Foundation large language models (LLMs) have shown an impressive ability to solve tasks across a wide range of fields including health. To effectively solve personalized health tasks, LLMs need the ability to ingest a diversity of data modalities that are relevant to an individual's health status. In this paper, we take a step towards creating multimodal LLMs for health that are grounded in individual-specific data by developing a framework (HeLM: Health Large Language Model for Multimodal Understanding) that enables LLMs to use high-dimensional clinical modalities to estimate underlying disease risk. HeLM encodes complex data modalities by learning an encoder that maps them into the LLM's token embedding space and for simple modalities like tabular data by serializing the data into text. Using data from the UK Biobank, we show that HeLM can effectively use demographic and clinical features in addition to high-dimensional time-series data to estimate disease risk. For example, HeLM achieves an AUROC of 0.75 for asthma prediction when combining tabular and spirogram data modalities compared with 0.49 when only using tabular data. Overall, we find that HeLM outperforms or performs at parity with classical machine learning approaches across a selection of eight binary traits. Furthermore, we investigate the downstream uses of this model such as its generalizability to out-of-distribution traits and its ability to power conversations around individual health and wellness.

翻译：基础大语言模型（LLMs）在包括健康在内的广泛领域展现出解决任务的卓越能力。为有效解决个性化健康任务，大语言模型需要具备处理与个体健康状况相关的多样化数据模态的能力。本文通过开发一个框架（HeLM：健康多模态理解大语言模型），使大语言模型能够利用高维临床模态估算潜在疾病风险，从而向构建面向个体特异性数据的健康多模态大语言模型迈出关键一步。HeLM通过学习编码器将复杂数据模态映射到大语言模型的词元嵌入空间，同时针对表格数据等简单模态采用序列化文本处理方式。基于英国生物银行数据，我们证明HeLM能有效结合人口统计学特征、临床特征及高维时序数据来评估疾病风险。例如，在哮喘预测中，HeLM融合表格数据和肺活量测定数据模态时AUROC达到0.75，而仅使用表格数据时仅为0.49。总体而言，在八项二元特征选择任务中，HeLM的表现优于或等同于传统机器学习方法。此外，我们探究了该模型的下游应用，包括其对分布外特征的泛化能力以及赋能个体健康与福祉对话的潜力。