Controller synthesis is in essence a case of model-based planning for non-deterministic environments in which plans (actually ''strategies'') are meant to preserve system goals indefinitely. In the case of supervisory control environments are specified as the parallel composition of state machines and valid strategies are required to be ''non-blocking'' (i.e., always enabling the environment to reach certain marked states) in addition to safe (i.e., keep the system within a safe zone). Recently, On-the-fly Directed Controller Synthesis techniques were proposed to avoid the exploration of the entire -and exponentially large-environment space, at the cost of non-maximal permissiveness, to either find a strategy or conclude that there is none. The incremental exploration of the plant is currently guided by a domain-independent human-designed heuristic. In this work, we propose a new method for obtaining heuristics based on Reinforcement Learning (RL). The synthesis algorithm is thus framed as an RL task with an unbounded action space and a modified version of DQN is used. With a simple and general set of features that abstracts both states and actions, we show that it is possible to learn heuristics on small versions of a problem that generalize to the larger instances, effectively doing zero-shot policy transfer. Our agents learn from scratch in a highly partially observable RL task and outperform the existing heuristic overall, in instances unseen during training.

翻译：控制器综合本质上是一种基于模型的非确定性环境规划方法，其中的规划（实际上是“策略”）旨在无限期地维护系统目标。在监督控制环境下，环境被建模为状态机的并行组合，有效的策略除了需要满足安全性（即保持系统处于安全区域内）之外，还必须具备“无阻塞性”（即始终允许环境到达某些标记状态）。最近，人们提出了在线定向控制器综合技术，以非最大容许性为代价，避免探索整个且呈指数级增长的环境空间，从而找到一种策略或判定不存在这样的策略。当前，对受控对象的增量式探索由一种领域无关的人工设计启发式方法引导。在本工作中，我们提出了一种基于强化学习获取启发式方法的新方案。由此，综合算法被构建为一个具有无界动作空间的强化学习任务，并采用了改进版DQN。通过使用一组既抽象状态又抽象动作的简单且通用的特征，我们证明可以在问题的小规模版本上学习启发式方法，并泛化到更大的实例，从而有效实现零样本策略迁移。我们的智能体在高度部分可观测的强化学习任务中从零开始学习，并且在训练过程中未见过的实例中，其整体性能优于现有启发式方法。