The discovery of neural architectures from simple building blocks is a long-standing goal of Neural Architecture Search (NAS). Hierarchical search spaces are a promising step towards this goal but lack a unifying search space design framework and typically only search over some limited aspect of architectures. In this work, we introduce a unifying search space design framework based on context-free grammars that can naturally and compactly generate expressive hierarchical search spaces that are 100s of orders of magnitude larger than common spaces from the literature. By enhancing and using their properties, we effectively enable search over the complete architecture and can foster regularity. Further, we propose an efficient hierarchical kernel design for a Bayesian Optimization search strategy to efficiently search over such huge spaces. We demonstrate the versatility of our search space design framework and show that our search strategy can be superior to existing NAS approaches. Code is available at https://github.com/automl/hierarchical_nas_construction.

翻译：从简单构建块出发发现神经架构是神经架构搜索（NAS）的长期目标。层级搜索空间是实现这一目标的有前景方向，但缺乏统一的搜索空间设计框架，且通常仅搜索架构的有限方面。本文引入一种基于上下文无关文法的统一搜索空间设计框架，该框架能够自然且紧凑地生成富有表现力的层级搜索空间，其规模比文献中常见空间大数百个数量级。通过增强并利用其特性，我们有效实现了对完整架构的搜索，并能促进规则性。此外，我们为贝叶斯优化搜索策略提出了一种高效的层级核设计，以高效搜索如此巨大的空间。我们展示了搜索空间设计框架的多功能性，表明我们的搜索策略可优于现有NAS方法。代码见https://github.com/automl/hierarchical_nas_construction。