Partially observable Markov decision processes (POMDPs) provide a flexible representation for real-world decision and control problems. However, POMDPs are notoriously difficult to solve, especially when the state and observation spaces are continuous or hybrid, which is often the case for physical systems. While recent online sampling-based POMDP algorithms that plan with observation likelihood weighting have shown practical effectiveness, a general theory characterizing the approximation error of the particle filtering techniques that these algorithms use has not previously been proposed. Our main contribution is bounding the error between any POMDP and its corresponding finite sample particle belief MDP (PB-MDP) approximation. This fundamental bridge between PB-MDPs and POMDPs allows us to adapt any sampling-based MDP algorithm to a POMDP by solving the corresponding particle belief MDP, thereby extending the convergence guarantees of the MDP algorithm to the POMDP. Practically, this is implemented by using the particle filter belief transition model as the generative model for the MDP solver. While this requires access to the observation density model from the POMDP, it only increases the transition sampling complexity of the MDP solver by a factor of $\mathcal{O}(C)$, where $C$ is the number of particles. Thus, when combined with sparse sampling MDP algorithms, this approach can yield algorithms for POMDPs that have no direct theoretical dependence on the size of the state and observation spaces. In addition to our theoretical contribution, we perform five numerical experiments on benchmark POMDPs to demonstrate that a simple MDP algorithm adapted using PB-MDP approximation, Sparse-PFT, achieves performance competitive with other leading continuous observation POMDP solvers.

翻译：部分可观测马尔可夫决策过程（POMDPs）为现实世界的决策与控制问题提供了灵活的表述框架。然而，POMDPs的求解极具挑战性，尤其在状态与观测空间为连续或混合类型时（这在物理系统中极为常见）。尽管近年来基于在线采样并采用观测似然加权的POMDP算法展现出实际有效性，但尚未有通用理论刻画这些算法所采用的粒子滤波技术的近似误差。本文的主要贡献是界定了任意POMDP与其对应有限样本粒子信念MDP（PB-MDP）近似之间的误差。这一PB-MDP与POMDP之间的基础桥梁使得我们能够通过求解对应的粒子信念MDP，将任意基于采样的MDP算法适配至POMDP，从而将MDP算法的收敛性保证推广至POMDP。在实际实现中，该方法通过将粒子滤波信念转移模型作为MDP求解器的生成模型来完成。尽管这需要访问POMDP中的观测密度模型，但它仅将MDP求解器的转移采样复杂度增加因子$\mathcal{O}(C)$，其中$C$为粒子数量。因此，当与稀疏采样MDP算法结合时，该方案可衍生出对状态与观测空间规模无直接理论依赖性的POMDP算法。除理论贡献外，我们在基准POMDP上进行了五项数值实验，证明通过PB-MDP近似适配的简单MDP算法Sparse-PFT，其性能可与其他前沿连续观测POMDP求解器相媲美。