Explainability for Deep Learning Models is especially important for clinical applications, where decisions of automated systems have far-reaching consequences. While various post-hoc explainable methods, such as attention visualization and saliency maps, already exist for common data modalities, including natural language and images, little work has been done to adapt them to the modality of Flow CytoMetry (FCM) data. In this work, we evaluate the usage of a transformer architecture called ReluFormer that ease attention visualization as well as we propose a gradient- and an attention-based visualization technique tailored for FCM. We qualitatively evaluate the visualization techniques for cell classification and polygon regression on pediatric Acute Lymphoblastic Leukemia (ALL) FCM samples. The results outline the model's decision process and demonstrate how to utilize the proposed techniques to inspect the trained model. The gradient-based visualization not only identifies cells that are most significant for a particular prediction but also indicates the directions in the FCM feature space in which changes have the most impact on the prediction. The attention visualization provides insights on the transformer's decision process when handling FCM data. We show that different attention heads specialize by attending to different biologically meaningful sub-populations in the data, even though the model retrieved solely supervised binary classification signals during training.

翻译：深度学习模型的可解释性在临床应用中尤为重要，因为自动化系统的决策可能产生深远影响。尽管针对自然语言和图像等常见数据模态已存在多种后验解释方法（如注意力可视化与显著性图），但针对流式细胞术（FCM）数据模态的适配研究仍十分有限。本研究评估了一种名为ReluFormer的Transformer架构的可用性，该架构可简化注意力可视化，同时我们提出了两种针对FCM数据定制的可视化技术：基于梯度的方法与基于注意力的方法。我们以定性方式评估了这些可视化技术在儿科急性淋巴细胞白血病（ALL）FCM样本的细胞分类与多边形回归任务中的表现。结果揭示了模型的决策过程，并展示了如何利用所提技术对训练后的模型进行检验。基于梯度的可视化方法不仅能识别对特定预测最重要的细胞，还能指示FCM特征空间中变化对预测影响最大的方向。基于注意力的可视化则揭示了Transformer在处理FCM数据时的决策机制。研究表明，即使模型在训练过程中仅接收有监督的二分类信号，不同的注意力头仍会专注于数据中具有生物学意义的亚群。