With the wide and deep adoption of deep learning models in real applications, there is an increasing need to model and learn the representations of the neural networks themselves. These models can be used to estimate attributes of different neural network architectures such as the accuracy and latency, without running the actual training or inference tasks. In this paper, we propose a neural architecture representation model that can be used to estimate these attributes holistically. Specifically, we first propose a simple and effective tokenizer to encode both the operation and topology information of a neural network into a single sequence. Then, we design a multi-stage fusion transformer to build a compact vector representation from the converted sequence. For efficient model training, we further propose an information flow consistency augmentation and correspondingly design an architecture consistency loss, which brings more benefits with less augmentation samples compared with previous random augmentation strategies. Experiment results on NAS-Bench-101, NAS-Bench-201, DARTS search space and NNLQP show that our proposed framework can be used to predict the aforementioned latency and accuracy attributes of both cell architectures and whole deep neural networks, and achieves promising performance. Code is available at https://github.com/yuny220/NAR-Former.

翻译：随着深度学习模型在实际应用中的广泛与深入应用，对神经网络自身进行建模和表示学习的需求日益增长。此类模型可用于评估不同神经网络架构的属性（如准确率和延迟），而无需执行实际的训练或推理任务。本文提出一种神经架构表示模型，能够整体性地估计这些属性。具体而言，我们首先设计一种简单有效的分词器，将神经网络的操作和拓扑信息编码为单一序列；然后，设计一种多阶段融合Transformer，从转换后的序列中构建紧凑的向量表示。为实现高效模型训练，我们进一步提出信息流一致性增强方法，并相应设计架构一致性损失函数，与以往的随机增强策略相比，该方法能用更少的增强样本带来更多收益。在NAS-Bench-101、NAS-Bench-201、DARTS搜索空间及NNLQP上的实验结果表明，本文提出的框架能够预测单元架构和整体深度神经网络的延迟与准确率属性，并取得了优异性能。代码开源地址：https://github.com/yuny220/NAR-Former。