This paper addresses the escalating challenge of redundant data transmission in networks. The surge in traffic has strained backhaul links and backbone networks, prompting the exploration of caching solutions at the edge router. Existing work primarily relies on Markov Decision Processes (MDP) for caching issues, assuming fixed-time interval decisions; however, real-world scenarios involve random request arrivals, and despite the critical role of various file characteristics in determining an optimal caching policy, none of the related existing work considers all these file characteristics in forming a caching policy. In this paper, first, we formulate the caching problem using a semi-Markov Decision Process (SMDP) to accommodate the continuous-time nature of real-world scenarios allowing for caching decisions at random times upon file requests. Then, we propose a double deep Q-learning-based caching approach that comprehensively accounts for file features such as lifetime, size, and importance. Simulation results demonstrate the superior performance of our approach compared to a recent Deep Reinforcement Learning-based method. Furthermore, we extend our work to include a Transfer Learning (TL) approach to account for changes in file request rates in the SMDP framework. The proposed TL approach exhibits fast convergence, even in scenarios with increased differences in request rates between source and target domains, presenting a promising solution to the dynamic challenges of caching in real-world environments.

翻译：本文针对网络中日益严重的数据冗余传输问题展开研究。流量激增导致回程链路与骨干网络承受巨大压力，促使业界探索边缘路由器端的缓存解决方案。现有工作主要采用马尔可夫决策过程处理缓存问题，其前提是固定时间间隔决策；然而实际场景中请求到达具有随机性，且文件各类特征对最优缓存策略的制定至关重要，但现有相关研究均未在构建缓存策略时综合考虑所有文件特征。本文首先采用半马尔可夫决策过程对缓存问题进行建模，以适应实际场景的连续时间特性，允许在文件请求到达的随机时刻进行缓存决策。随后提出基于双深度Q学习的缓存方法，全面考虑文件生存期、大小及重要性等特征。仿真结果表明，与近期基于深度强化学习的方法相比，本方法具有更优性能。进一步地，我们引入迁移学习方法，以应对半马尔可夫决策过程框架中文件请求率的变化。所提迁移学习方法在源域与目标域请求率差异增大的场景下仍能快速收敛，为实际环境中缓存的动态挑战提供了有前景的解决方案。