We discuss guidelines for evaluating the performance of parameterized stochastic solvers for optimization problems, with particular attention to systems that employ novel hardware, such as digital quantum processors running variational algorithms, analog processors performing quantum annealing, or coherent Ising Machines. We illustrate through an example a benchmarking procedure grounded in the statistical analysis of the expectation of a given performance metric measured in a test environment. In particular, we discuss the necessity and cost of setting parameters that affect the algorithm's performance. The optimal value of these parameters could vary significantly between instances of the same target problem. We present an open-source software package that facilitates the design, evaluation, and visualization of practical parameter tuning strategies for complex use of the heterogeneous components of the solver. We examine in detail an example using parallel tempering and a simulator of a photonic Coherent Ising Machine computing and display the scoring of an illustrative baseline family of parameter-setting strategies that feature an exploration-exploitation trade-off.

翻译：我们讨论了评估参数化随机求解器在优化问题上性能的指南，重点关注采用新型硬件的系统，例如运行变分算法的数字量子处理器、执行量子退火的模拟处理器或相干伊辛机。我们通过一个示例说明了基于测试环境中测量的给定性能指标期望值进行统计分析的标准测试流程。特别是，我们探讨了设置影响算法性能参数的必要性及成本。这些参数的最优值在同一目标问题的不同实例间可能存在显著差异。我们提供了一个开源软件包，用于促进针对求解器异构组件复杂使用场景的实际参数调优策略的设计、评估与可视化。我们详细研究了一个示例，该示例使用了并行回火算法和光子相干伊辛机模拟器进行计算，并展示了一个体现探索-利用权衡的基线参数设置策略系列的评分结果。