Drawing on ideas from continuous integration, we present concepts of an automated benchmarking pipeline for high performance applications. Customization and collaboration have been key design goals owing to the requirements of research-software development as a continuous community effort. We have extended our previous conceptual work on systematic benchmarking workflows with the functionality of user-agnostic operations as well as continuous benchmarking. This fosters reproducibility and re-use of benchmarking results to ensure sustainable technological progress. We provide software-engineering solutions to keep pace with the rapid evolution of both large-scale models and high-performance computing systems with a view towards the scientific domains of neuroscience and artificial intelligence.

翻译：借鉴持续集成的理念，我们提出了面向高性能应用的自动化基准测试流程的概念。考虑到研究型软件开发作为持续的社区协作努力的需求，定制化与协作一直是关键的设计目标。我们在先前关于系统性基准测试工作流的理论基础上，扩展了用户无关操作以及持续基准测试的功能。这有助于提升基准测试结果的可复现性与重用性，从而确保可持续的技术进步。我们提供软件工程解决方案，以跟上大规模模型与高性能计算系统的快速演进，尤其聚焦于神经科学与人工智能的科学领域。