The race for the most efficient, accurate, and universal algorithm in scientific computing drives innovation. At the same time, this healthy competition is only beneficial if the research output is actually comparable to prior results. Fairly comparing algorithms can be a complex endeavor, as the implementation, configuration, compute environment, and test problems need to be well-defined. Due to the increase in computer-based experiments, new infrastructure for facilitating the exchange and comparison of new algorithms is also needed. To this end, we propose a benchmark framework, as a set of generic specifications for comparing implementations of algorithms using test cases native to a community. Its value lies in its ability to fairly compare and validate existing methods for new applications, as well as compare newly developed methods with existing ones. As a prototype for a more general framework, we have begun building a benchmark tool for the model order reduction (MOR) community. The data basis of the tool is the collection of the Model Order Reduction Wiki (MORWiki). The wiki features three main categories: benchmarks, methods, and software. An editorial board curates submissions and patrols edited entries. Data sets for linear and parametric-linear models are already well represented in the existing collection. Data sets for non-linear or procedural models, for which only evaluation data, or codes / algorithmic descriptions, rather than equations, are available, are being added and extended. Properties and interesting characteristics used for benchmark selection and later assessments are recorded in the model metadata. Our tool, the Model Order Reduction Benchmarker (MORB) is under active development for linear time-invariant systems and solvers.

翻译：在科学计算领域，追求最高效、最精确且普适算法的竞赛推动了创新。然而，这种良性竞争仅在研究成果能与先前结果进行有效比较时才有意义。公平比较算法是一项复杂工程，因为实现方式、配置参数、计算环境及测试问题均需明确定义。随着基于计算机的实验不断增加，亟需构建新型基础设施以促进新算法的交换与比较。为此，本文提出一个基准框架——通过特定领域社区的测试用例，为算法实现比较提供通用规范。该框架的价值在于能够公平比较和验证现有方法在新应用场景中的表现，同时实现新方法与传统方法的对比。作为通用框架的原型，我们已着手为模型降阶(MOR)社区构建基准工具。该工具的数据基础源于模型降阶维基(MORWiki)。该维基包含三大核心类别：基准、方法与软件。编委会负责审核提交内容并维护编辑条目。现有数据库中已充分收录线性与参数线性模型的数据集。对于非线性或程序化模型——仅有评估数据、代码或算法描述而无方程形式——的数据集正在持续扩充。模型元数据记录了用于基准选择及后续评估的特性参数与关键属性。我们的工具——模型降阶基准测试器(MORB)正针对线性时不变系统及求解器进行积极开发。