Analysis of 3D segmentation models, especially in the context of medical imaging, is often limited to segmentation performance metrics that overlook the crucial aspect of explainability and bias. Currently, effectively explaining these models with saliency maps is challenging due to the high dimensions of input images multiplied by the ever-growing number of segmented class labels. To this end, we introduce Agg^2Exp, a methodology for aggregating fine-grained voxel attributions of the segmentation model's predictions. Unlike classical explanation methods that primarily focus on the local feature attribution, Agg^2Exp enables a more comprehensive global view on the importance of predicted segments in 3D images. Our benchmarking experiments show that gradient-based voxel attributions are more faithful to the model's predictions than perturbation-based explanations. As a concrete use-case, we apply Agg^2Exp to discover knowledge acquired by the Swin UNEt TRansformer model trained on the TotalSegmentator v2 dataset for segmenting anatomical structures in computed tomography medical images. Agg^2Exp facilitates the explanatory analysis of large segmentation models beyond their predictive performance. The source code is publicly available at https://github.com/mi2datalab/agg2exp.

翻译：对三维分割模型的分析，尤其是在医学影像领域，通常局限于分割性能指标，忽视了可解释性与偏差这一关键方面。当前，由于输入图像的高维度与不断增长的待分割类别标签数量相乘，利用显著性图有效解释这些模型具有挑战性。为此，我们提出了Agg^2Exp方法，用于聚合分割模型预测的细粒度体素归因。与主要关注局部特征归因的经典解释方法不同，Agg^2Exp能够对三维图像中预测分割区域的重要性提供更全面的全局视角。我们的基准实验表明，基于梯度的体素归因比基于扰动的解释更能忠实反映模型的预测。作为一个具体用例，我们将Agg^2Exp应用于分析在TotalSegmentator v2数据集上训练的Swin UNEt TRansformer模型，该模型用于分割计算机断层扫描医学图像中的解剖结构。Agg^2Exp促进了超越预测性能的大型分割模型的解释性分析。源代码公开于 https://github.com/mi2datalab/agg2exp。