Regular decision processes (RDPs) are a subclass of non-Markovian decision processes where the transition and reward functions are guarded by some regular property of the past (a lookback). While RDPs enable intuitive and succinct representation of non-Markovian decision processes, their expressive power coincides with finite-state Markov decision processes (MDPs). We introduce omega-regular decision processes (ODPs) where the non-Markovian aspect of the transition and reward functions are extended to an omega-regular lookahead over the system evolution. Semantically, these lookaheads can be considered as promises made by the decision maker or the learning agent about her future behavior. In particular, we assume that, if the promised lookaheads are not met, then the payoff to the decision maker is $\bot$ (least desirable payoff), overriding any rewards collected by the decision maker. We enable optimization and learning for ODPs under the discounted-reward objective by reducing them to lexicographic optimization and learning over finite MDPs. We present experimental results demonstrating the effectiveness of the proposed reduction.

翻译：正则决策过程（RDPs）是非马尔可夫决策过程的一个子类，其中转移函数和奖励函数受限于过去的某种正则性质（回溯）。尽管RDPs能够直观且简洁地表示非马尔可夫决策过程，但其表达能力与有限状态马尔可夫决策过程（MDPs）等价。我们引入了Omega-正则决策过程（ODPs），其中转移函数和奖励函数的非马尔可夫特性被扩展为对系统演化的Omega-正则前视。从语义上看，这些前视可被视为决策者或学习主体对其未来行为的承诺。具体而言，我们假设若承诺的前视未被满足，决策者获得的收益为$\bot$（最低期望收益），并覆盖决策者收集的任何奖励。我们通过将ODPs的折扣奖励目标约简为有限MDPs上的词典序优化与学习，实现了其优化与学习过程。实验结果表明该约简方法的有效性。