Large discrete action spaces (LDAS) remain a central challenge in reinforcement learning. Existing solution approaches can handle unstructured LDAS with up to a few million actions. However, many real-world applications in logistics, production, and transportation systems have combinatorial action spaces, whose size grows well beyond millions of actions, even on small instances. Fortunately, such action spaces exhibit structure, e.g., equally spaced discrete resource units. With this work, we focus on handling structured LDAS (SLDAS) with sizes that cannot be handled by current benchmarks: we propose Dynamic Neighborhood Construction (DNC), a novel exploitation paradigm for SLDAS. We present a scalable neighborhood exploration heuristic that utilizes this paradigm and efficiently explores the discrete neighborhood around the continuous proxy action in structured action spaces with up to $10^{73}$ actions. We demonstrate the performance of our method by benchmarking it against three state-of-the-art approaches designed for large discrete action spaces across two distinct environments. Our results show that DNC matches or outperforms state-of-the-art approaches while being computationally more efficient. Furthermore, our method scales to action spaces that so far remained computationally intractable for existing methodologies.

翻译：大离散动作空间（LDAS）仍是强化学习中的核心挑战。现有方法可处理高达数百万动作的无结构LDAS，但物流、生产与运输系统中的实际应用常具有组合动作空间，即便在小型实例中，其规模也远超数百万动作。所幸此类动作空间存在结构特性（如等间距的离散资源单位）。本研究聚焦于处理结构化大离散动作空间（SLDAS），其规模超出当前基准测试的承载力：我们提出动态邻域构建（DNC），一种面向SLDAS的新型开发范式。我们给出可扩展的邻域探索启发式算法，利用该范式高效探索结构化动作空间中连续代理动作周围的离散邻域，动作数可达$10^{73}$量级。通过在两个不同环境中与三种面向大离散动作空间的先进方法进行基准对比，验证了本方法的性能。结果表明，DNC在匹配或超越现有先进方法的同时，计算效率更高。此外，本方法可扩展至现有方法仍计算棘手的动作空间规模。