The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.

翻译：国际机器学习（ML）研究与实践中，基准测试竞赛的数量正稳步增长。然而，目前关于社区在解决研究问题时面临的常见实践与瓶颈知之甚少。为了揭示生物医学图像分析这一特定领域中算法开发的现状，我们设计了一项国际调查，面向在IEEE ISBI 2021和MICCAI 2021会议期间举办的所有竞赛（共80项）的参与者。调查内容涵盖参与者的专业知识与工作环境、所选策略以及算法特征。平均72%的竞赛参与者参与了调查。结果显示，知识交流是参与的主要动机（70%），而获得奖金仅起次要作用（16%）。尽管方法开发的中位投入时间为80小时，但大部分参与者表示没有足够的时间进行方法开发（32%）。25%的参与者认为基础设施是瓶颈。总体而言，94%的解决方案基于深度学习，其中84%采用了标准架构。43%的受访者报告数据样本（如图像）过大，无法一次性处理。最常用的应对方式是基于块（patch）的训练（69%）、降采样（37%）以及将3D分析任务分解为一系列2D任务。仅37%的参与者在训练集上执行K折交叉验证，仅50%的参与者进行了集成学习，其中基于多个相同模型（61%）或异质模型（39%）。48%的受访者应用了后处理步骤。