Artificial intelligence holds promise to fundamentally enhance healthcare. Developing an integrated many-to-many framework leveraging multimodal data for multiple tasks is essential to unifying modern medicine. We introduce M3H, an explainable Multimodal Multitask Machine Learning for Healthcare framework that consolidates learning from tabular, time-series, language, and vision data for supervised binary/multiclass classification, regression, and unsupervised clustering. M3H encompasses an unprecedented range of medical tasks and problem domains and consistently outperforms traditional single-task models by on average 11.6% across 40 disease diagnoses from 16 medical departments, three hospital operation forecasts, and one patient phenotyping task. It features a novel attention mechanism balancing self-exploitation (learning source-task), and cross-exploration (learning cross-tasks), and offers explainability through a proposed TIM score, shedding light on the dynamics of task learning interdependencies. Its adaptable architecture supports easy customization and integration of new data modalities and tasks, establishing it as a robust, scalable solution for advancing AI-driven healthcare systems.

翻译：人工智能有望从根本上提升医疗水平。构建一个利用多模态数据执行多任务的一体化多对多框架，对于统一现代医学体系至关重要。我们提出M3H——一种可解释的多模态多任务机器学习医疗框架，能够整合表格数据、时序数据、语言数据和视觉数据的学习，支持监督式二分类/多分类、回归以及无监督聚类任务。M3H覆盖了前所未有的医疗任务与问题领域范围，在来自16个医疗科室的40种疾病诊断、三项医院运营预测及一项患者表型分析任务中，其性能平均超越传统单任务模型11.6%。该框架具有创新的注意力机制，能够平衡自我开发（学习源任务）与交叉探索（学习跨任务），并通过所提出的TIM评分提供可解释性，揭示任务学习相互依赖的动态过程。其可适配架构支持便捷地定制及集成新数据模态与任务，使其成为推动AI驱动医疗系统的稳健可扩展解决方案。