Although deep learning have revolutionized abdominal multi-organ segmentation, models often struggle with generalization due to training on small, specific datasets. With the recent emergence of large-scale datasets, some important questions arise: \textbf{Can models trained on these datasets generalize well on different ones? If yes/no, how to further improve their generalizability?} To address these questions, we introduce A-Eval, a benchmark for the cross-dataset Evaluation ('Eval') of Abdominal ('A') multi-organ segmentation. We employ training sets from four large-scale public datasets: FLARE22, AMOS, WORD, and TotalSegmentator, each providing extensive labels for abdominal multi-organ segmentation. For evaluation, we incorporate the validation sets from these datasets along with the training set from the BTCV dataset, forming a robust benchmark comprising five distinct datasets. We evaluate the generalizability of various models using the A-Eval benchmark, with a focus on diverse data usage scenarios: training on individual datasets independently, utilizing unlabeled data via pseudo-labeling, mixing different modalities, and joint training across all available datasets. Additionally, we explore the impact of model sizes on cross-dataset generalizability. Through these analyses, we underline the importance of effective data usage in enhancing models' generalization capabilities, offering valuable insights for assembling large-scale datasets and improving training strategies. The code and pre-trained models are available at \href{https://github.com/uni-medical/A-Eval}{https://github.com/uni-medical/A-Eval}.

翻译：尽管深度学习已革新了腹部多器官分割，但模型常因在小规模特定数据集上训练而难以实现泛化。随着大规模数据集的近期涌现，一些重要问题随之产生：**基于这些数据集训练的模型能否在不同数据集上实现良好泛化？若能/否，如何进一步提升其泛化能力？** 为解答上述问题，我们提出了A-Eval——一个用于腹部（Abdominal）多器官分割跨数据集评估（Evaluation）的基准（Benchmark）。我们采用了四个大规模公开数据集的训练集：FLARE22、AMOS、WORD和TotalSegmentator，每个数据集均提供丰富的腹部多器官分割标签。评估环节中，我们整合了上述数据集的验证集以及BTCV数据集的训练集，构建了一个包含五个独立数据集的稳健基准。利用A-Eval基准，我们重点评估了各类模型在不同数据使用场景下的泛化能力，包括：独立使用单个数据集训练、通过伪标签利用无标注数据、混合不同模态数据，以及联合使用所有可用数据集进行训练。此外，我们还探讨了模型规模对跨数据集泛化能力的影响。通过上述分析，我们强调了有效数据利用对提升模型泛化能力的重要性，为构建大规模数据集及改进训练策略提供了宝贵见解。相关代码与预训练模型已开源至 \href{https://github.com/uni-medical/A-Eval}{https://github.com/uni-medical/A-Eval}。