Deep neural networks are often applied to medical images to automate the problem of medical diagnosis. However, a more clinically relevant question that practitioners usually face is how to predict the future trajectory of a disease. Current methods for prognosis or disease trajectory forecasting often require domain knowledge and are complicated to apply. In this paper, we formulate the prognosis prediction problem as a one-to-many prediction problem. Inspired by a clinical decision-making process with two agents -- a radiologist and a general practitioner -- we predict prognosis with two transformer-based components that share information with each other. The first transformer in this framework aims to analyze the imaging data, and the second one leverages its internal states as inputs, also fusing them with auxiliary clinical data. The temporal nature of the problem is modeled within the transformer states, allowing us to treat the forecasting problem as a multi-task classification, for which we propose a novel loss. We show the effectiveness of our approach in predicting the development of structural knee osteoarthritis changes and forecasting Alzheimer's disease clinical status directly from raw multi-modal data. The proposed method outperforms multiple state-of-the-art baselines with respect to performance and calibration, both of which are needed for real-world applications. An open-source implementation of our method is made publicly available at \url{https://github.com/Oulu-IMEDS/CLIMATv2}.

翻译：深度神经网络常被应用于医学图像以实现医学诊断自动化。然而，临床实践中更常见的挑战是如何预测疾病的未来轨迹。当前的预后或疾病轨迹预测方法通常需要领域知识且实施复杂。本文将预后预测问题建模为一对多预测问题。受临床决策过程中两个智能体（放射科医师与全科医师）的协作启发，我们采用两个共享信息的Transformer组件进行预后预测。框架中的第一个Transformer专注于分析影像数据，第二个则利用其内部状态作为输入，并将其与辅助临床数据融合。问题的时间特性被建模于Transformer状态中，使我们能够将预测任务转化为多任务分类问题，并为此提出一种新型损失函数。我们验证了该方法在预测结构性膝骨关节炎进展以及直接从原始多模态数据预测阿尔茨海默病临床状态方面的有效性。所提方法在性能和校准度上均优于多个最先进基线模型，这两项指标对于实际应用至关重要。本方法的开源实现已公开于\url{https://github.com/Oulu-IMEDS/CLIMATv2}。