Large Multimodal Models (LMMs) achieve state-of-the-art performance in high-stakes domains like healthcare, yet their reasoning remains opaque. Current interpretability methods, such as attention mechanisms or post-hoc saliency, often fail to faithfully represent the model's decision-making process, particularly when integrating heterogeneous modalities like time-series and text. We introduce Tree-of-Evidence (ToE), an inference-time search algorithm that frames interpretability as a discrete optimization problem. Rather than relying on soft attention weights, ToE employs lightweight Evidence Bottlenecks that score coarse groups or units of data (e.g., vital-sign windows, report sentences) and performs a beam search to identify the compact evidence set required to reproduce the model's prediction. We evaluate ToE across six tasks spanning three datasets and two domains: four clinical prediction tasks on MIMIC-IV, cross-center validation on eICU, and non-clinical fault detection on LEMMA-RCA. ToE produces auditable evidence traces while maintaining predictive performance, retaining over 0.98 of full-model AUROC with as few as five evidence units across all settings. Under sparse evidence budgets, ToE achieves higher decision agreement and lower probability fidelity error than other approaches. Qualitative analyses show that ToE adapts its search strategy: it often resolves straightforward cases using only vitals, while selectively incorporating text when physiological signals are ambiguous. ToE therefore provides a practical mechanism for auditing multimodal models by revealing which discrete evidence units support each prediction.

翻译：大型多模态模型（LMMs）在医疗保健等高关键领域取得了最先进的性能，但其推理过程仍不透明。当前的可解释性方法（如注意力机制或事后显著性分析）通常无法忠实反映模型的决策过程，尤其是在融合时间序列与文本等异质模态时。我们提出“证据之树”（ToE），一种将可解释性建模为离散优化问题的推理时搜索算法。ToE不依赖软注意力权重，而是采用轻量级证据瓶颈（Evidence Bottlenecks）对数据的粗略组或单元（例如生命体征窗口、报告句子）进行评分，并通过束搜索识别重现模型预测所需的精简证据集。我们在涵盖三个数据集、两个领域的六项任务上评估ToE：MIMIC-IV上的四项临床预测任务、eICU上的跨中心验证以及LEMMA-RCA上的非临床故障检测。ToE在保持预测性能的同时生成可审计的证据轨迹，在所有场景下仅需五个证据单元即可保留完整模型AUROC的0.98以上。在稀疏证据预算下，ToE相比其他方法实现了更高的决策一致性和更低的概率保真度误差。定性分析显示，ToE能自适应调整搜索策略：对于简单案例常仅使用生命体征完成推理，而在生理信号模糊时选择性纳入文本信息。因此，ToE通过揭示支撑每个预测的离散证据单元，为审计多模态模型提供了实用机制。