Delayed and partially observable state information poses significant challenges for reinforcement learning (RL)-based control in real-world autonomous driving. In highway on-ramp merging, a roadside unit (RSU) can sense nearby traffic, perform edge perception, and transmit state estimates to the ego vehicle over vehicle-to-infrastructure (V2I) links. With recent advancements in intelligent transportation infrastructure and edge computing, such RSU-assisted perception is increasingly realistic and already deployed in modern connected roadway systems. However, edge processing time and wireless transmission can introduce stochastic V2I communication delays, violating the Markov assumption and substantially degrading control performance. In this work, we propose DAROM, a Delay-Aware Reinforcement Learning framework for On-ramp Merging that is robust to stochastic delays. We model the problem as a random delay Markov decision process (RDMDP) and develop a unified RL agent for joint longitudinal and lateral control. To recover a Markovian representation under delayed observations, we introduce a Delay-Aware Encoder that conditions on delayed observations, masked action histories, and observed delay magnitude to infer the current latent state. We further integrate a physics-based safety controller to reduce collision risk during merging. Experiments in the Simulation of Urban MObility (SUMO) simulator using real-world traffic data from the Next Generation Simulation (NGSIM) dataset demonstrate that DAROM consistently outperforms standard RL baselines across traffic densities. In particular, the gated recurrent unit (GRU)-based encoder achieves over 99% success in high-density traffic with random V2I delays of up to 2.0 seconds.

翻译：延迟和部分可观测的状态信息给基于强化学习的真实世界自动驾驶控制带来了重大挑战。在高速公路匝道汇入场景中，路侧单元可以感知附近交通，执行边缘感知，并通过车联网链路将状态估计传输给自车。随着智能交通基础设施和边缘计算的最新进展，这种路侧单元辅助感知变得越来越现实，并且已经部署在现代联网道路系统中。然而，边缘处理时间和无线传输可能引入随机的车联网通信延迟，这违反了马尔可夫假设，并显著降低了控制性能。在本工作中，我们提出了DAROM，一种对随机延迟具有鲁棒性的延迟感知强化学习框架，用于匝道汇入。我们将该问题建模为随机延迟马尔可夫决策过程，并开发了一个统一的强化学习智能体，用于联合纵向和横向控制。为了在延迟观测下恢复马尔可夫表示，我们引入了一个延迟感知编码器，该编码器基于延迟观测、掩蔽动作历史和观测到的延迟幅度来推断当前潜在状态。我们进一步集成了一个基于物理学的安全控制器，以降低汇入过程中的碰撞风险。在基于下一代仿真数据集真实世界交通数据的城市交通仿真器中的实验表明，DAROM在各种交通密度下始终优于标准强化学习基线。特别地，基于门控循环单元的编码器在高达2.0秒的随机车联网延迟下的高密度交通中实现了超过99%的成功率。