Drawing on ideas from continuous integration, we present concepts of an automated benchmarking pipeline for high performance applications. Customization and collaboration have been key design goals owing to the requirements of research-software development as a continuous community effort. We have extended our previous conceptual work on systematic benchmarking workflows with the functionality of user-agnostic operations as well as continuous benchmarking. This fosters reproducibility and re-use of benchmarking results to ensure sustainable technological progress. We provide software-engineering solutions to keep pace with the rapid evolution of both large-scale models and high-performance computing systems with a view towards the scientific domains of neuroscience and artificial intelligence.

翻译：基于持续集成的理念，我们提出了面向高性能应用的自动化基准测试流水线概念。考虑到研究型软件开发作为持续社区协作的需求，可定制化与协同性已成为关键设计目标。我们扩展了先前关于系统性基准测试工作流程的概念研究，新增了用户无关操作及持续基准测试功能。这促进了基准测试结果的可复现性与复用性，从而确保可持续的技术进步。我们提供了软件工程解决方案，以跟上大规模模型与高性能计算系统快速演进的步伐，重点关注神经科学和人工智能科学领域。