Partially observable Markov decision processes (POMDPs) provide a flexible representation for real-world decision and control problems. However, POMDPs are notoriously difficult to solve, especially when the state and observation spaces are continuous or hybrid, which is often the case for physical systems. While recent online sampling-based POMDP algorithms that plan with observation likelihood weighting have shown practical effectiveness, a general theory characterizing the approximation error of the particle filtering techniques that these algorithms use has not previously been proposed. Our main contribution is bounding the error between any POMDP and its corresponding finite sample particle belief MDP (PB-MDP) approximation. This fundamental bridge between PB-MDPs and POMDPs allows us to adapt any sampling-based MDP algorithm to a POMDP by solving the corresponding particle belief MDP, thereby extending the convergence guarantees of the MDP algorithm to the POMDP. Practically, this is implemented by using the particle filter belief transition model as the generative model for the MDP solver. While this requires access to the observation density model from the POMDP, it only increases the transition sampling complexity of the MDP solver by a factor of $\mathcal{O}(C)$, where $C$ is the number of particles. Thus, when combined with sparse sampling MDP algorithms, this approach can yield algorithms for POMDPs that have no direct theoretical dependence on the size of the state and observation spaces. In addition to our theoretical contribution, we perform five numerical experiments on benchmark POMDPs to demonstrate that a simple MDP algorithm adapted using PB-MDP approximation, Sparse-PFT, achieves performance competitive with other leading continuous observation POMDP solvers.

翻译：部分可观测马尔可夫决策过程（POMDPs）为现实决策与控制问题提供了灵活的表示方法。然而，POMDPs求解极为困难，尤其是当状态和观测空间为连续或混合型时（在物理系统中常有此情况）。尽管近期基于在线采样、利用观测似然加权进行规划的POMDP算法展示了实际有效性，但此前尚未有通用理论刻画这些算法所采用的粒子滤波技术的近似误差。我们的主要贡献是界定了任意POMDP与其对应的有限样本粒子信念MDP（PB-MDP）近似之间的误差。这一PB-MDP与POMDP之间的基础桥梁使我们能够通过求解对应粒子信念MDP，将任何基于采样的MDP算法适配到POMDP，从而将MDP算法的收敛保证扩展到POMDP。在实际实现中，这通过将粒子滤波信念转移模型作为MDP求解器的生成模型来完成。虽然这需要访问POMDP中的观测密度模型，但仅将MDP求解器的转移采样复杂度增加了因子$\mathcal{O}(C)$，其中$C$为粒子数。因此，当与稀疏采样MDP算法结合时，该方法能够产生对状态和观测空间规模无直接理论依赖的POMDP算法。除理论贡献外，我们在基准POMDP上进行了五项数值实验，证明采用PB-MDP近似适配的简单MDP算法——Sparse-PFT，其性能与其他领先的连续观测POMDP求解器相当。