As one of the emerging challenges in Automated Machine Learning, the Hardware-aware Neural Architecture Search (HW-NAS) tasks can be treated as black-box multi-objective optimization problems (MOPs). An important application of HW-NAS is real-time semantic segmentation, which plays a pivotal role in autonomous driving scenarios. The HW-NAS for real-time semantic segmentation inherently needs to balance multiple optimization objectives, including model accuracy, inference speed, and hardware-specific considerations. Despite its importance, benchmarks have yet to be developed to frame such a challenging task as multi-objective optimization. To bridge the gap, we introduce a tailored streamline to transform the task of HW-NAS for real-time semantic segmentation into standard MOPs. Building upon the streamline, we present a benchmark test suite, CitySeg/MOP, comprising fifteen MOPs derived from the Cityscapes dataset. The CitySeg/MOP test suite is integrated into the EvoXBench platform to provide seamless interfaces with various programming languages (e.g., Python and MATLAB) for instant fitness evaluations. We comprehensively assessed the CitySeg/MOP test suite on various multi-objective evolutionary algorithms, showcasing its versatility and practicality. Source codes are available at https://github.com/EMI-Group/evoxbench.

翻译：作为自动化机器学习中的新兴挑战之一，硬件感知神经架构搜索（HW-NAS）任务可被视作黑箱多目标优化问题（MOPs）。HW-NAS在实时语义分割中具有重要应用，该技术在自动驾驶场景中扮演关键角色。面向实时语义分割的HW-NAS本质上需要权衡多个优化目标，包括模型精度、推理速度及硬件特定约束。尽管其重要性显著，但目前尚无基准测试将该类挑战性任务构建为多目标优化问题。为此，我们提出一套定制化流程，将面向实时语义分割的HW-NAS任务转化为标准MOPs。基于该流程，我们发布了基准测试套件CitySeg/MOP，其包含源自Cityscapes数据集的十五个MOPs。该测试套件已集成至EvoXBench平台，支持通过多种编程语言（如Python和MATLAB）实现即时适应度评估的无缝接口。我们利用多种多目标进化算法对CitySeg/MOP测试套件进行了全面评估，验证了其通用性与实用性。源代码获取地址：https://github.com/EMI-Group/evoxbench。