While emerging 3D medical foundation models are envisioned as versatile tools with offer general-purpose capabilities, their validation remains largely confined to regional and structural imaging, leaving a significant modality discrepancy unexplored. To provide a rigorous and objective assessment, we curate the UMD dataset comprising 490 whole-body PET/CT and 464 whole-body PET/MRI scans ($\sim$675k 2D images, $\sim$12k 3D organ annotations) and conduct a thorough and comprehensive evaluation of representative 3D segmentation foundation models. Through intra-subject controlled comparisons of paired scans, we isolate imaging modality as the primary independent variable to evaluate model robustness in real-world applications. Our evaluation reveals a stark discrepancy between literature-reported benchmarks and real-world efficacy, particularly when transitioning from structural to functional domains. Such systemic failures underscore that current 3D foundation models are far from achieving truly general-purpose status, necessitating a paradigm shift toward multi-modal training and evaluation to bridge the gap between idealized benchmarking and comprehensive clinical utility. This dataset and analysis establish a foundational cornerstone for future research to develop truly modality-agnostic medical foundation models.

翻译：尽管新兴的三维医学基础模型被设想为具备通用能力的多功能工具，但其验证工作仍主要局限于区域性和结构性成像，尚未探索显著的模态差异。为提供严谨客观的评估，我们构建了UMD数据集，包含490例全身PET/CT扫描和464例全身PET/MRI扫描（约67.5万张二维图像，约1.2万个三维器官标注），并对代表性的三维分割基础模型进行了全面系统的评估。通过对配对扫描进行受试者内对照比较，我们将成像模态作为主要自变量，以评估模型在真实世界应用中的鲁棒性。我们的评估揭示了文献报道的基准与真实世界效能之间的显著差异，尤其是在从结构域转向功能域时。此类系统性失效表明，当前的三维基础模型远未达到真正的通用水平，亟需向多模态训练与评估的范式转变，以弥合理想化基准测试与全面临床效用之间的差距。本数据集与分析为未来开发真正模态无关的医学基础模型奠定了基石。