As an essential resource management problem in network virtualization, virtual network embedding (VNE) aims to allocate the finite resources of physical network to sequentially arriving virtual network requests (VNRs) with different resource demands. Since this is an NP-hard combinatorial optimization problem, many efforts have been made to provide viable solutions. However, most existing approaches have either ignored the admission control of VNRs, which has a potential impact on long-term performances, or not fully exploited the temporal and topological features of the physical network and VNRs. In this paper, we propose a deep Hierarchical Reinforcement Learning approach to learn a joint Admission Control and Resource Allocation policy for VNE, named HRL-ACRA. Specifically, the whole VNE process is decomposed into an upper-level policy for deciding whether to admit the arriving VNR or not and a lower-level policy for allocating resources of the physical network to meet the requirement of VNR through the HRL approach. Considering the proximal policy optimization as the basic training algorithm, we also adopt the average reward method to address the infinite horizon problem of the upper-level agent and design a customized multi-objective intrinsic reward to alleviate the sparse reward issue of the lower-level agent. Moreover, we develop a deep feature-aware graph neural network to capture the features of VNR and physical network and exploit a sequence-to-sequence model to generate embedding actions iteratively. Finally, extensive experiments are conducted in various settings, and show that HRL-ACRA outperforms state-of-the-art baselines in terms of both the acceptance ratio and long-term average revenue. Our code is available at \url{https://github.com/GeminiLight/hrl-acra}.

翻译：作为网络虚拟化中的核心资源管理问题，虚拟网络嵌入旨在将物理网络的有限资源分配给按序到达且具有不同资源需求的虚拟网络请求。由于这是一个NP难组合优化问题，已有大量研究致力于提供可行的解决方案。然而，现有方法大多忽略了虚拟网络请求的准入控制（这对长期性能具有潜在影响），或未能充分利用物理网络与虚拟网络请求的时空特征与拓扑特征。本文提出一种分层深度强化学习方法，用于学习虚拟网络嵌入的联合准入控制与资源分配策略，命名为HRL-ACRA。具体而言，通过分层强化学习方法将完整的虚拟网络嵌入过程分解为：上层策略用于决定是否接纳到达的虚拟网络请求，下层策略负责分配物理网络资源以满足虚拟网络请求的需求。以近端策略优化算法作为基础训练框架，我们采用平均奖励方法处理上层智能体的无限时域问题，并设计定制化的多目标内在奖励以缓解下层智能体的稀疏奖励问题。此外，我们开发了深度特征感知图神经网络以捕获虚拟网络请求与物理网络的特征，并利用序列到序列模型迭代生成嵌入动作。最后，通过多种场景下的广泛实验表明，HRL-ACRA在请求接受率与长期平均收益指标上均优于现有先进基线方法。代码已开源：\url{https://github.com/GeminiLight/hrl-acra}。