Machine learning is a prevalent approach to tame the complexity of design space exploration for domain-specific architectures. Using ML for design space exploration poses challenges. First, it's not straightforward to identify the suitable algorithm from an increasing pool of ML methods. Second, assessing the trade-offs between performance and sample efficiency across these methods is inconclusive. Finally, lack of a holistic framework for fair, reproducible, and objective comparison across these methods hinders progress of adopting ML-aided architecture design space exploration and impedes creating repeatable artifacts. To mitigate these challenges, we introduce ArchGym, an open-source gym and easy-to-extend framework that connects diverse search algorithms to architecture simulators. To demonstrate utility, we evaluate ArchGym across multiple vanilla and domain-specific search algorithms in designing custom memory controller, deep neural network accelerators, and custom SoC for AR/VR workloads, encompassing over 21K experiments. Results suggest that with unlimited samples, ML algorithms are equally favorable to meet user-defined target specification if hyperparameters are tuned; no solution is necessarily better than another (e.g., reinforcement learning vs. Bayesian methods). We coin the term hyperparameter lottery to describe the chance for a search algorithm to find an optimal design provided meticulously selected hyperparameters. The ease of data collection and aggregation in ArchGym facilitates research in ML-aided architecture design space exploration. As a case study, we show this advantage by developing a proxy cost model with an RMSE of 0.61% that offers a 2,000-fold reduction in simulation time. Code and data for ArchGym is available at https://bit.ly/ArchGym.

翻译：机器学习已成为应对领域专用架构设计空间探索复杂性的主流方法。然而，将机器学习应用于设计空间探索仍面临挑战：首先，从日益丰富的机器学习方法中识别合适算法并非易事；其次，评估不同方法在性能与样本效率之间的权衡尚无定论；最后，缺乏用于公平、可重复及客观比较这些方法的整体框架，阻碍了机器学习辅助架构设计空间探索的进展，并妨碍了可复现成果的生成。为应对这些挑战，我们提出ArchGym——一个开源且易于扩展的平台，可连接多样化搜索算法与架构模拟器。为展示其效用，我们基于ArchGym评估了多种基础及领域专用搜索算法在定制存储控制器、深度神经网络加速器及AR/VR定制片上系统设计中的应用，共涉及超过21,000次实验。结果表明：在无限样本条件下，若超参数经调优，各类机器学习算法在满足用户定义目标规格方面表现相当，无必然优劣之分（如强化学习与贝叶斯方法对比）。我们提出“超参数彩票”这一术语，用以描述搜索算法在精心选择超参数前提下发现最优设计的概率。ArchGym便捷的数据采集与聚合能力可促进机器学习辅助架构设计空间探索研究。通过案例研究，我们开发了一个均方根误差仅0.61%的代理成本模型，将仿真时间降低约2000倍。ArchGym代码与数据已发布于https://bit.ly/ArchGym。