We study countably infinite Markov decision processes (MDPs) with real-valued transition rewards. Every infinite run induces the following sequences of payoffs: 1. Point payoff (the sequence of directly seen transition rewards), 2. Mean payoff (the sequence of the sums of all rewards so far, divided by the number of steps), and 3. Total payoff (the sequence of the sums of all rewards so far). For each payoff type, the objective is to maximize the probability that the $\liminf$ is non-negative. We establish the complete picture of the strategy complexity of these objectives, i.e., how much memory is necessary and sufficient for $\varepsilon$-optimal (resp. optimal) strategies. Some cases can be won with memoryless deterministic strategies, while others require a step counter, a reward counter, or both.
翻译:我们研究具有实值转移奖励的可数无穷马尔可夫决策过程(MDPs)。每条无穷路径诱导出以下收益序列:1. 点收益(直接观察到的转移奖励序列),2. 平均收益(当前所有奖励之和除以步数的序列),3. 总收益(当前所有奖励之和的序列)。对于每种收益类型,目标是最大化$\liminf$非负的概率。我们建立了这些目标策略复杂度的完整图景,即实现$\varepsilon$-最优(或最优)策略所需必要且充分的内存大小。某些情形可通过无记忆确定性策略获胜,而其他情形则需要步计数器、奖励计数器或两者兼备。