Deep Learning (DL) and specifically CNN models have become a de facto method for a wide range of vision tasks, outperforming traditional machine learning (ML) methods. Consequently, they drew a lot of attention in the neuroimaging field in particular for phenotype prediction or computer-aided diagnosis. However, most of the current studies often deal with small single-site cohorts, along with a specific pre-processing pipeline and custom CNN architectures, which make them difficult to compare to. We propose an extensive benchmark of recent state-of-the-art (SOTA) 3D CNN, evaluating also the benefits of data augmentation and deep ensemble learning, on both Voxel-Based Morphometry (VBM) pre-processing and quasi-raw images. Experiments were conducted on a large multi-site 3D brain anatomical MRI data-set comprising N=10k scans on 3 challenging tasks: age prediction, sex classification, and schizophrenia diagnosis. We found that all models provide significantly better predictions with VBM images than quasi-raw data. This finding evolved as the training set approaches 10k samples where quasi-raw data almost reach the performance of VBM. Moreover, we showed that linear models perform comparably with SOTA CNN on VBM data. We also demonstrated that DenseNet and tiny-DenseNet, a lighter version that we proposed, provide a good compromise in terms of performance in all data regime. Therefore, we suggest to employ them as the architectures by default. Critically, we also showed that current CNN are still very biased towards the acquisition site, even when trained with N=10k multi-site images. In this context, VBM pre-processing provides an efficient way to limit this site effect. Surprisingly, we did not find any clear benefit from data augmentation techniques. Finally, we proved that deep ensemble learning is well suited to re-calibrate big CNN models without sacrificing performance.

翻译：深度学习（DL）特别是CNN模型已成为各类视觉任务的事实标准方法，其性能显著超越传统机器学习（ML）方法。因此，这类模型在神经影像领域（特别是表型预测或计算机辅助诊断中）引起了广泛关注。然而，现有研究多基于小型单中心数据集，并结合特定预处理流程与定制化CNN架构，导致结果难以横向比较。我们提出了一个针对最新（SOTA）3D CNN的全面基准测试，系统评估了数据增强与深度集成学习在基于体素的形态学（VBM）预处理图像与准原始图像上的效果。实验采用包含N=10,000次扫描的大规模多中心三维脑部解剖MRI数据集，在年龄预测、性别分类与精神分裂症诊断三项挑战性任务上进行验证。研究发现：所有模型在VBM图像上的预测性能均显著优于准原始数据。当训练样本接近10,000例时，准原始数据性能趋近VBM。此外，线性模型在VBM数据上的表现可媲美SOTA CNN模型。我们提出的DenseNet及其轻量版本tiny-DenseNet在所有数据规模下均展现出良好的性能权衡，建议将其作为默认架构。关键发现是：即便使用N=10,000例多中心图像训练，当前CNN仍存在严重的采集站点偏差，而VBM预处理能有效抑制该偏差。令人意外的是，数据增强技术并未带来显著性能提升。最后，我们证实深度集成学习可在不牺牲性能的前提下有效校准大型CNN模型。